Wenn Nachrichten warten, sichtbar werden lassen.

im4hc Monitor – Comprehensive Technical Documentation

Document ID: DOC-IM4HC-001
Version: 1.1.0
Last Updated: February 2026
Maintained by: DigitalON Technology Services
Support Portal: support.digitalon.co.za

---

Table of Contents

1. Executive Summary
2. System Architecture
3. Component Documentation
4. Data Flow & Workflows
5. Installation & Deployment
6. Configuration Reference (Abschnitt in der Quelle nicht enthalten)
7. Operational Runbook (Abschnitt in der Quelle nicht enthalten)
8. Troubleshooting Playbook (Abschnitt in der Quelle nicht enthalten)
9. Alert Reference (Abschnitt in der Quelle nicht enthalten)
10. API & Integration Reference (Abschnitt in der Quelle nicht enthalten)
11. Appendix (Abschnitt in der Quelle nicht enthalten)

---

1. Executive Summary

1.1 Purpose

The im4hc Monitor is a production-grade Python monitoring solution designed to detect and alert on stale HL7 messages in the im4hc healthcare integration platform. It provides near real-time visibility into message queue health across 148+ queue tables and 59+ healthcare providers, and integrates directly into CheckMK for service monitoring.

1.2 Key Capabilities

1.3 Current Deployment Status

1.4 Business Impact

• Early Detection: identifies stuck messages before they impact clinical operations and patient workflows.
• Reduced MTTR: alerts include historical context and provider hints to accelerate investigation.
• No Alert Fatigue: deduplication prevents repeated alerts for the same incident.
• Provider Visibility: problems are grouped by provider and classified by severity.
• Mobile-Friendly: compact format is readable in WhatsApp/Matrix.
• Audit Trail: discovery snapshots + incident lifecycle improve traceability.

---

2. System Architecture

2.1 High-Level Architecture

At a glance:

• CheckMK agent runs local checks (every 900s and 1800s).
• Checks run the im4hc_monitor Python package to discover queues and detect stale messages.
• Incidents are tracked persistently in a JSON state file.
• Alerts are sent via Telegram and Matrix (bridged to WhatsApp).
• Dashboards are accessible via CheckMK UI and Grafana.

Key runtime components:

• Monitoring Host: ncdcncogrd03 (CheckMK agent + Python package)
• Database: MySQL @ 10.8.7.225:3306 (schema: im4hc)
• Notifications: Telegram bot, Matrix homeserver + WhatsApp bridge
• Dashboards: CheckMK service status + Grafana deep link

2.2 Package Module Structure

2.3 Component Responsibilities

---

3. Component Documentation

3.1 Discovery Engine

The discovery engine automatically identifies HL7 queue tables without hardcoding table names. Queue tables are detected using regex patterns and explicit exclusions.

3.1.1 Table Detection Patterns

# Queue Table Detection Patterns (from discovery.py)
QUEUE_TABLE_PATTERNS = [
 "^out_\\w+", # out_hims_orm, out_lancet_orm_12900
 "^in_\\w+", # in_ampath_orm, in_sanbs_oru
 "^\\w+_orm_in$", # beningfield_orm_in, labos_orm_in
 "^\\w+_oru_in$", # labos_oru_in, chiro_oru_in
 "^\\w+_mdm_in$", # skylims_mdm_in, valabjee_mdm_in
]

EXCLUDE_TABLES = {"queuedetail", "orm_queue_ack"}
DEFAULT_EXCLUDE_SUFFIXES = ("_keytable", "_stat")

3.1.2 Provider Extraction Logic

Providers are extracted from queue table names to allow grouping and history tracking:

Examples:
 out_hims_orm -> hims
 out_lancet_orm_12900 -> lancet
 in_ampath_orm -> ampath
 in_sanbs_oru -> sanbs
 beningfield_orm_in -> beningfield
 skylims_mdm_in -> skylims

3.2 State Management

The state module provides persistent tracking of active incidents, alert cooldowns, discovery history, and provider MTTR. State is stored in a JSON file (default: /var/tmp/im4hc_monitor_state.json).

3.2.1 State File Structure

{
 "active_incidents": {
 "out_hims_orm": {
 "first_detected": 1738742400.0,
 "last_seen": 1738743300.0,
 "message_count": 15,
 "peak_count": 25,
 "oldest_message": "2026-02-05 08:45:00",
 "incident_id": "INC-20260205-001",
 "provider": "hims",
 "alert_sent": true,
 "alert_sent_at": 1738742500.0
 }
 },
 "alerts": {
 "queue_scan": {"last_sent": 1738743000.0, "message_hash": "abc123"},
 "matrix_alert": {"last_sent": 1738742400.0}
 },
 "incident_counter": {"date": "20260205", "count": 3},
 "incident_history": { "... per provider ...": {} },
 "discovered_tables": { "... per table ...": {} },
 "discovery_snapshots": [ "... history ..." ]
}

3.2.2 State Sections

3.3 Notification System

3.3.1 Dual-Channel Design

• Both Telegram and Matrix are supported.
• Each channel has an independent cooldown to prevent flooding.
• Each channel has an alert time window (default 04:00–23:00 local time).
• Matrix messages are formatted compactly and may appear in WhatsApp via a bridge.

3.3.2 Severity Classification

Legend:

• >>> / > = outbound queue (to provider)
• <<<<[NEW][INCREASING][STABLE][DECREASING] = partial recovery
• INC-YYYYMMDD-NNN = incident reference ID

3.4 Preflight Validation

Before monitoring checks run (especially in automated service contexts), the system supports a 6-point preflight. This prevents false alerts caused by missing config, broken DB connectivity, or invalid tokens.

Preflight checks:
 [config] Required configuration fields are present
 [state_file] State file path exists / is writable
 [mysql] MySQL connectivity and schema access
 [discovery] Queue discovery returns tables
 [telegram] Telegram token validated via getMe
 [matrix] Matrix token validated via whoami

Example preflight output:

=== im4hc_monitor Preflight Report ===

 [PASS] config: All required fields present
 [PASS] state_file: Writable: /var/tmp/im4hc_monitor_state.json
 [PASS] mysql: Connected and schema accessible
 [PASS] discovery: Found 148 queue tables
 [PASS] telegram: Token valid (getMe OK)
 [PASS] matrix: Connected as @digitalon:alldigital-on.com

Result: 6/6 checks passed

---

4. Data Flow & Workflows

4.1 Queue Scan Check Flow (900s)

Goal: detect stale messages across all queue tables and trigger alerts with deduplication.

1. Load config + state.
2. Discover queue tables.
3. Record discovery snapshot history.
4. Scan tables for stale messages older than queue_age_seconds (default 880s).
5. If no problems: output OK to CheckMK.
6. If problems: classify WARN/CRIT based on affected-table count and format alert.
7. Send alerts (Telegram + Matrix) if time window + cooldown allow.
8. Save updated state and exit.

4.2 Matrix Check Flow (1800s) with Incident Lifecycle

Goal: manage incident lifecycle state and send recovery notifications when queues clear.

1. Load config + state.
2. Discover and scan tables.
3. Compare current problems vs active incidents in state.
4. Resolved incidents: close and send recovery notifications.
5. Still-active incidents: update age, counts, trends, and keep open.
6. New incidents: create incident IDs and mark NEW vs ONGOING based on age.
7. Output CheckMK status and save state.

4.3 Incident Lifecycle State Machine

• Detected → queue has stale messages
• New → age < 5 minutes (CheckMK CRIT: “NEW integration problem”)
• Ongoing → age ≥ 5 minutes (CheckMK WARN: “Ongoing integration issue”)
• Resolved → queue clears; recovery is sent once

---

5. Installation & Deployment

5.1 Prerequisites

6.2 Configuration Sections Explained

Security note: Do not publish token values, internal IPs, or credentials in a public-facing portal. Keep this article internal or redact sensitive values.

---