Intelligence Breaks the System It Controls

Technical Summary: "Intelligence Breaks the System It Controls"

Problem Statement
The paper addresses a fundamental structural failure in the design of dual-intelligence clinical systems (clinician + AI). The prevailing assumption is that safe medical AI can be achieved through a governance model where AI handles high-dimensional pattern recognition and clinicians apply contextual judgment. The authors argue this assumption fails because effective treatment itself deforms the patient's physiological trajectory, creating a "shared blind spot." In this state, neither the clinician nor the AI possesses sufficient information to predict the subsequent patient trajectory, rendering cooperation under current information structures ineffective. The core problem is not a lack of cooperation, but a lack of instrumentation: neither agent measures how treatment has altered the patient's trajectory relative to its own history.

Methodology
The study utilizes a "treatment-path concordance framework" to audit whether treatment decisions (by clinicians or AI models) match the subsequent patient trajectory. The research employs three large-scale longitudinal datasets:

1. MIMIC-IV (Acute Care): 6,243 ICU admissions (73,141 encounters) focusing on vasopressor titration (norepinephrine) in sepsis. This represents a low-information setting with brief clinician-patient contact.
2. ADNI (Longitudinal Care): 2,262 participants with serial cognitive assessments (Alzheimer's Disease), representing a high-information setting where clinicians accumulate years of private observation.
3. OhioT1DM: A supplemental dataset of 6 subjects with 87,091 CGM readings to verify target-zone ambiguity in glucose trajectories.

The authors trained a "13-family computational model zoo" (including linear, tree-based, ensemble, boosting, and neural architectures) and introduced several novel metrics:

• PBR (Probability of Branch Reversal): Measures the ambiguity of future direction (0.5 = maximal ambiguity).
• CS (Curvature Shift) & DR (Deformation Radius): Quantify how much a treatment changes the trajectory geometry relative to pre-event continuation.
• CRA (Cooperation Recovery Audit): A logistic regression-based metric measuring the extent to which pooling clinician and AI information resolves directional state change beyond majority-class prediction.
• Branch Balance: A derived metric (2 × PBR) where 1.0 indicates maximal ambiguity.

Key Results

1. Target-Zone Ambiguity: Clinical targets (e.g., MAP 65 mmHg in sepsis) are zones of maximal branch ambiguity. At MAP 65, PBR approaches 0.5, meaning the future direction is effectively a coin flip.
2. Treatment-Induced Deformation: Effective treatment (e.g., norepinephrine onset) significantly deforms trajectory geometry. Treated trajectories show a median Curvature Shift of 1.143 versus 0.566 for stable comparators. This deformation injects non-identifiability; repeated adjustments drive branch balance from 0.816 (first adjustment) to 0.994 (after nine+ adjustments), collapsing directional identifiability.
3. The Shared Blind Spot (Acute Care): In MIMIC-IV, both clinicians and AI models failed to match the subsequent trajectory in 53% of treatment events.
   • Cooperation Surplus: Pooling clinician and AI information yielded negligible gains (< 0.003).
   • Model Failure: AI models defaulted to population median actions (e.g., "maintain" dose) because they could not see the specific branches. Dose recommenders consistently over-dosed relative to clinicians (12.5% to 104.2% excess at MAP 65).
   • Geometry as Solution: Direct measurement of serial trajectory deformation (how far the profile drifted from its initial shape) recovered 10–19 percentage points of directional resolution where action exchange added nothing.
4. The Reversal (Longitudinal Care): In ADNI, the landscape reversed. The shared blind spot shrank to 11%.
   • Clinician decisions added up to 25 percentage points of directional information (reflecting accumulated private knowledge), while trajectory geometry added negligible value.
   • Genuine cooperation surplus (+0.020) was observed only when pooling the clinician's private knowledge with the trajectory's deformation history.
5. Architecture Limitations: Increasing model complexity (from linear to deep neural networks) did not recover missing branch history. The obstruction is informational, not a matter of model capacity.

Key Contributions

• Structural Diagnosis: The paper identifies that the obstacle to safe clinical AI is not poor governance or lack of cooperation, but a fundamental "non-identifiability" caused by the treatment itself altering the system's law of motion.
• Deformation-Aware Estimation: The authors propose that the solution is not better prediction of the next state, but "instrumentation"—directly measuring the deformation of the patient's trajectory relative to its own history.
• Context-Dependent Information Architecture: The study demonstrates that the source of critical information is domain-specific: acute care requires better trajectory instrumentation, while longitudinal care requires the encoding of clinician private knowledge into the model.
• CRA Metric: Introduction of the Cooperation Recovery Audit to empirically test whether information pooling resolves directional ambiguity.

Significance and Claims
The paper claims that the current paradigm of "governance" (assigning roles between AI and clinicians) is premature because it assumes both agents have access to the necessary information to make optimal decisions. The authors argue that in acute care, neither agent currently measures the right thing.

The central thesis is that "intelligence breaks the system it controls" because the act of intervention (treatment) reshapes the patient, creating a feedback loop where the input to the next decision is a distorted version of the previous state. The authors conclude that safe clinical AI requires a shift from predictive modeling on fixed surfaces to deformation-aware estimation, where architectures infer the latent operative law from action-response history. Until the system measures how treatment has changed the patient's trajectory relative to its own history, autonomous control is not justified, and cooperation will yield no surplus.