Drift-Aware Medical AI

Medical imaging models degrade silently. A classifier that performs well at release can drift as scanners, populations, and acquisition protocols change — and in healthcare, silent failure is the most dangerous kind.

This is built and evaluated as a monitoring pipeline for an ML operations team, not a clinical decision-support tool. It treats drift as a first-class engineering problem: detect it early without new labels, trace it to a likely cause, explain it in plain English, and recommend a response.

A chest X-ray classifier is trained once on CheXpert, then monitored against a cross-source drift stream from NIH ChestX-ray14 using nine unsupervised two-sample drift tests — MMD, energy distance, Wasserstein, PSI, BBSD, C2ST, entropy, trainable MMD, and UMAP+JSD — run on pixels, embeddings, and predictions, with a streaming ensemble that votes across a rolling window.

`orchestrator.py` is the always-on agent: for each incoming batch it chains Detect → Analyze → Explain → Recommend end to end without a human driving each step. The Explanation Agent produces a rule-based and an LLM-generated (Phi-3) report; the Recommendation Agent picks from a fixed catalogue of responses — recalibrate, update a threshold, or flag a slice for retraining or targeted triage.

Root-cause attribution ranks metadata and image-level candidates by effect size to identify what actually drove a shift, validated live by injecting a known corruption and confirming the agent recovers it as the top driver.

The best single detector (Wasserstein / C2ST) reaches AUROC ≈0.91; the 4-detector ensemble does not beat the best single detector in any of 16 tested conditions — an honest negative result, reported as such rather than hidden.

Streaming detection holds AUROC 0.989 at a mean latency of 6.6ms per batch (p95 11.1ms) under genuinely incremental arrival, not offline batching. Cross-dataset generalisation drops from 0.877 (CheXpert) to 0.841 (NIH) macro AUROC on five shared diseases.

Per-disease temperature scaling meaningfully outperforms a single global temperature, which barely moves calibration error (0.1094 → 0.1063) — the 14 diseases don't share one correction direction or magnitude.

The ensemble not beating the best single detector was initially disappointing, then became the most useful result in the write-up — an honest negative finding, with the failure mode documented in an ethics/risk log, is worth more to an ML ops team than a cherry-picked win.

Reliability work is mostly about what happens after the model ships. Building the monitoring and orchestration layer taught me more about production ML than training the classifier did.