🤖 machine learning

Counterfactual Evaluation Reveals Hidden Capability Profiles in Clinical LLMs and Agents

This paper introduces the Causal Sensitivity Score (CSS), an interventional metric that reveals significant discrepancies between clinical AI models' performance on standard coverage-based benchmarks and their actual responsiveness to critical changes in patient data, exposing hidden capability profiles and universal safety blind spots that traditional evaluation methods miss.

Original authors: Matt Turk

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Matt Turk

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a team of medical advisors (AI models) to help decide the best treatment for cancer patients. Traditionally, we've tested these advisors by giving them a patient's file and checking if their final recommendation matches what a panel of human experts would say. If the advice looks right, they get a good grade.

This paper argues that this traditional grading system is like judging a chef only by the taste of the final dish, without ever asking: "Did you actually taste the ingredients, or did you just memorize the recipe and serve the same dish no matter what?"

Here is the breakdown of the paper's findings using simple analogies:

1. The Problem: The "Look-Alike" Trap

The authors found that two AI systems can get nearly identical scores on standard tests, yet behave completely differently when the facts change.

The Scenario: Imagine a patient has a specific marker on their cancer cells. The AI suggests a drug.
The Twist: Now, imagine we secretly change the file to say the patient doesn't have that marker anymore.
The Result: A "smart" AI should change its drug recommendation. A "dumb" (but lucky) AI might just ignore the change and suggest the same drug anyway.
The Flaw: Standard tests (called CMS in the paper) only check if the final answer matches the experts. They don't catch the AI that is just "stuck" on one answer. It's like a student who memorized the answer key; they get an 'A' on the test, but if you change the question slightly, they fail because they don't understand the logic.

2. The Solution: The "Causal Sensitivity Score" (CSS)

To fix this, the authors created a new test called the Causal Sensitivity Score (CSS).

The Analogy: Think of this as a "spot the difference" game for medical files. The researchers take a patient file and make small, pre-planned changes (like flipping a switch from "surgery done" to "surgery not done," or removing a specific drug history).
The Test: They ask the AI: "Now that I changed this fact, did you change your recommendation?"
The Score:
- 1.0: You changed your mind correctly (Great!).
- 0.5: You noticed the change but didn't change your mind (Okay).
- 0.0: You ignored the change completely (Bad).

3. The Big Surprise: The Rankings Flipped

The authors tested six of the smartest AI models available. When they used the old test (CMS), the models were ranked in a certain order. When they used the new test (CSS), the rankings completely flipped.

The model that was ranked last by the old test became the winner by the new test.
The model that was ranked first by the old test dropped to fourth place.
The Takeaway: The "best" AI according to old standards was actually the worst at reacting to new information.

4. The Universal Blind Spot: The "Surgery" Glitch

The paper discovered a specific type of change that every single AI model failed, regardless of how smart they were.

The Scenario: Changing whether a patient had surgery or not.
The Failure: When the file said "Patient had surgery," the AI suggested one treatment. When the file was changed to "Patient did not have surgery," the AI often suggested the exact same treatment.
Why it matters: In real medicine, whether a patient had surgery is a huge deal. If an AI ignores this, it's a safety risk. The old test (CMS) never caught this because the AI's answer still "looked" like a valid medical opinion, even though it was ignoring a critical fact.

5. The "Tool-Using" Experiment

The researchers also tested these AIs when they were allowed to use tools (like a search engine to look up patient records) instead of just reading a static file.

The Result: For most models, using tools helped them get better scores. They could find the new info and update their advice.
The Odd One Out: One specific model (gpt-5.4) used the tools just as well as the others—it found the right information—but still didn't change its recommendation.
The Metaphor: It's like a detective who finds the smoking gun (the evidence) but still insists the suspect is innocent. This suggests the problem isn't that the AI can't find the info; it's that its brain is structurally unable to update its conclusion based on that info.

Summary

This paper introduces a new way to test medical AIs that checks if they are actually thinking or just repeating.

Old Way: Did you give the right answer? (Yes/No)
New Way (CSS): If I change the facts, do you change your answer? (Yes/No)

The study shows that the models we thought were the best might actually be the most rigid, and that all current top-tier models have a dangerous blind spot when it comes to surgery status. The authors suggest we need this new "responsiveness" test to ensure AI agents are truly safe and reliable before we let them help doctors.

Technical Summary: Counterfactual Evaluation Reveals Hidden Capability Profiles in Clinical LLMs and Agents

Problem Statement

Current evaluation paradigms for clinical Large Language Models (LLMs) and agents rely heavily on coverage-based metrics, such as string similarity or LLM-as-judge rubrics that compare outputs against reference behaviors. These metrics ask, "Does the output look right?" but fail to address, "Is the model updating its output for the right reasons?"

This limitation is critical in clinical settings where two systems may score identically on coverage metrics yet behave radically differently under changing patient inputs. For instance, an oncology AI might propose a standard treatment (e.g., FOLFIRINOX) for a pancreatic case regardless of whether the patient is treatment-naive or has failed prior therapies. While coverage metrics (like the Consensus Match Score, CMS) might reward this consistency, they cannot detect a lack of causal responsiveness. This issue is exacerbated in tool-using agents, which can retrieve correct information via tool calls but still generate recommendations that ignore that information.

Methodology

The authors introduce the Causal Sensitivity Score (CSS), a pre-registered interventional metric designed to measure a model's behavioral responsiveness to clinically meaningful changes in input data.

1. Pre-registered Intervention Catalog

The study utilizes a catalog of 12 interventions across five clinically motivated families, specified in a YAML format prior to evaluation to prevent post-hoc selection bias:

Family A (Biomarker Flip): Replacing status for HER2, ER, or PD-L1.
Family B (Prior Treatment): Injecting a history of progression on a prior-line therapy.
Family C (Biomarker Strip): Deleting mentions of specific biomarkers.
Family D (Surgery Status): Toggling resection history (e.g., resected vs. unresected).
Family E (Stage Perturbation): Changing the disease stage.

2. Scoring Mechanism

For each model, case, and intervention tuple, the system generates a baseline recommendation and an intervened recommendation. A judge LLM (defaulting to gpt-5.4, with claude-opus-4-7 used when gpt-5.4 is the subject to avoid self-preference bias) evaluates whether the recommendation updates in the pre-registered correct direction.

Score: {0.0, 0.5, 1.0} corresponding to (no change / acknowledged but unchanged / updated correctly).
Metric: CSS is the mean score across all tuples, calculated both in aggregate and per intervention family.

3. Experimental Setup

Cohort: 224 expert-annotated oncology tumor-board cases with ground-truth treatment recommendations.
Models: Six frontier models from three labs (OpenAI, Anthropic, xAI) evaluated in two settings:
1. Single-shot LLM inference: Direct prompt-response.
2. Tool-using Agent: A ReAct-style agent using a read_chart_section tool to retrieve information, where interventions mutate the underlying data retrieved by the tool.
Baseline: The Consensus Match Score (CMS), a weighted recall metric measuring overlap with expert consensus.

Key Results

1. Rank Reversal Between Metrics

The study found a near-opposite ranking of models between CMS (coverage) and CSS (responsiveness).

Correlation: Spearman $\rho = -0.49$ (though underpowered due to $n=6$ ).
Rank Swaps: All six models changed rank. The model ranked last on CMS (grok-4.20-reasoning) ranked first on CSS. Conversely, the CMS-best model (gpt-5) dropped to fourth on CSS.
Specific Failure: gpt-5.4 ranked 4th on CMS but dead last (6th) on CSS.

2. Universal Safety Blind Spot (Family D)

A critical finding was a universal failure across all frontier models regarding surgery status interventions (Family D).

Performance: All models scored $\le 17.2\%$ on CSS for this family. The best performer (grok-4.20) scored 17.2%, while the worst (gpt-5.4) scored 3.9%.
Implication: None of the models reliably updated treatment recommendations when the patient's resection status changed, a clinically critical safety signal that CMS failed to expose.

3. Agent Transfer and Structural Responsiveness

When evaluated as tool-using agents on a subset of Family D cases:

Improvement: Tool use increased CSS for five of the six models (gains of +2.5 to +20.3 percentage points).
The gpt-5.4 Anomaly: gpt-5.4 remained unchanged (0.050 $\to$ 0.050) despite retrieving the same chart sections as the responsive models. This suggests a structural-responsiveness deficit rather than an information-access failure; the model retrieves the data but fails to integrate it into the reasoning process.

4. Validation

Cross-Judge Replication: Using claude-opus-4-7 as a uniform judge preserved the rank order ( $\rho = +1.00$ ), confirming the findings were not artifacts of judge bias.
Human Validation: Three medical professionals validated a subset of 100 tuples. While per-case agreement was moderate, the aggregate failure rates for Family D aligned closely between LLM judges (0.10) and humans (0.09).

Significance and Claims

The paper claims that CSS serves as a necessary complement to coverage-based evaluation for clinical AI. Its primary contributions are:

Revealing Hidden Profiles: CSS uncovers capability profiles (e.g., sensitivity to surgery status vs. biomarker flips) that aggregate coverage metrics obscure.
Safety Detection: It identifies universal safety blind spots (specifically regarding surgery status) that standard metrics miss because they do not test for behavioral change under counterfactuals.
Agent Evaluation: CSS transfers directly to tool-using agents, distinguishing between models that fail due to lack of information access versus those with a structural inability to update reasoning based on retrieved data.
Future Utility: The authors propose CSS as a candidate "dense reward" signal for future agentic Reinforcement Learning (RL), moving beyond static output matching to dynamic responsiveness.

The authors remain modest regarding the generalizability of the specific findings to all medical domains, noting that the results are specific to oncology tumor-board cases and that the small sample size ( $n=6$ models) limits the statistical power of rank correlation claims. However, they assert that the methodology of pre-registered interventional metrics is robust and essential for evaluating clinical AI agents.