💬 NLP

Compositional Reasoning Depth Predicts Clinical AI Failure: Empirical Evidence Consistent with Transformer Compositionality Limits in Electronic Health Record Question Answering

This paper demonstrates that the number of inferential steps (hop count) required to answer clinical questions from electronic health records is a robust, theory-motivated predictor of large language model failure, showing a consistent monotonic decline in accuracy across multiple architectures as reasoning depth increases, a trend that persists even with extended thinking and is not attributable to context truncation.

Original authors: Sanjay Basu

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sanjay Basu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a very smart, well-read intern to help you read a patient's medical file (a stack of papers called an Electronic Health Record, or EHR) and answer questions about it.

This paper is a report card on how that intern performs. The researchers found a surprising pattern: The more steps the intern has to take to find the answer, the more likely they are to fail.

Here is the breakdown of their findings using simple analogies:

1. The "Hop" Count: How Many Steps?

The researchers created a way to measure how hard a question is by counting the "hops" (steps) needed to answer it:

Hop 1 (The Easy Fetch): The question asks for one specific fact, like "What is the patient's blood pressure?" The intern just looks at one line and reads it.
Hop 2 (The Simple Comparison): The question asks, "Was the blood pressure high when they arrived?" The intern has to find the number and compare it to a standard.
Hop 3 (The Detective Work): The question asks, "Based on their age and family history, should they get a mammogram?" The intern has to find the age, find the family history, look up the medical rules, and combine them.
Hop 4 (The Master Chef): The question asks to summarize all specialist visits and flag any unresolved issues. The intern has to read many different sections, remember details from weeks ago, and synthesize a whole new story.

2. The Main Discovery: The "Step-Ladder" of Failure

The paper tested three different "super-interns" (AI models from companies like Anthropic and OpenAI). They found a consistent rule: As the number of steps increases, the accuracy drops like a stone.

Hop 1: The interns got about 30–38% of the answers right. (Not perfect, but decent).
Hop 4: The interns dropped to about 15–23% accuracy.

It's like a student who can easily solve a single math problem but completely falls apart when asked to solve a word problem that requires five different math concepts chained together. The more complex the chain, the more the AI gets lost.

3. The "Thinking Time" Experiment

The researchers wondered: What if we tell the intern to "think out loud" or give them more time to brainstorm before answering? This is called "Chain of Thought" or "Extended Thinking."

The Result: It didn't really help. Even when the AI was forced to write down its reasoning steps or given a massive budget of "thinking tokens" (mental energy), the accuracy still dropped as the questions got harder.
The Metaphor: Imagine giving a confused student a whiteboard to write down their thoughts. If the problem is too complex for their brain structure, writing it down doesn't magically make the solution appear. They still get stuck.

4. Why Does This Happen? (The "Brain" Limit)

The paper suggests this isn't because the AI is lazy or didn't read the file. It's a fundamental limit of how these AI "brains" (Transformers) are built.

The Analogy: Think of the AI's attention span like a flashlight. It can shine very brightly on one spot (Hop 1). But if it has to connect four different spots in a dark room to see a pattern (Hop 4), the flashlight isn't designed to hold all those beams in focus at once. The "light" gets too scattered to connect the dots.

5. The "Safety" Twist

Interestingly, when the AI got stuck on hard questions, it changed its behavior:

Before: It would sometimes guess wildly and make up facts (Hallucination).
After (with "thinking"): It stopped guessing. Instead, it often just said, "I don't know" or refused to answer (Omission).
Why this matters: In a hospital, it is safer for a system to say "I don't know" (so a human doctor can step in) than to confidently make up a fake diagnosis. The "thinking" mode made the AI more honest about its confusion, even if it didn't make it smarter.

6. What Didn't Cause the Failure?

The researchers checked to make sure the AI wasn't failing because the medical file was too long or cut off.

The Test: They checked if the answers were actually hidden in the parts of the file the AI could see. They found that the answers were there, even for the hardest questions.
The Conclusion: The problem wasn't missing information; it was the AI's inability to connect the information it had.

Summary

This paper tells us that current AI tools are great at finding single facts but struggle significantly when they have to connect multiple dots to solve a complex medical puzzle. Giving them more time or asking them to "think harder" doesn't fix this structural weakness.

The Takeaway: If you are using AI to read medical records, you need to know that it is much more likely to make mistakes on complex, multi-step questions than on simple ones. You can't just look at an "average" score; you have to look at how hard the questions were.

Technical Summary: Compositional Reasoning Depth Predicts Clinical AI Failure

Problem Statement

While large language models (LLMs) demonstrate improving performance on standardized medical examinations, their application to real-world electronic health record (EHR) question answering (QA) is characterized by inconsistent and potentially dangerous failures. A critical limitation of current evaluation frameworks is the reliance on aggregate accuracy metrics, which conceal the structural nature of model failures. Specifically, researchers and clinicians lack a principled understanding of when and why models fail on EHR tasks.

This paper posits that compositional reasoning depth—defined as the number of distinct inferential steps (hops) required to synthesize an answer from an EHR—is a systematic predictor of failure. This hypothesis is grounded in a converging body of theoretical literature on transformer compositionality limits (Hahn, 2020; Merrill et al., 2022; Sanford et al., 2023; Dziri et al., 2023; Peng et al., 2024), which suggests that fixed-architecture transformers struggle to represent complex compositional functions in a single forward pass. The study investigates whether accuracy degrades monotonically with hop count and whether test-time reasoning mechanisms (e.g., Chain-of-Thought) can mitigate this degradation.

Methodology

Dataset and Taxonomy

The study utilizes MedAlign, a dataset of clinician-generated instructions paired with EHR XML records and clinician reference answers.

Sample: 313 unique clinician-generated EHR question-answer pairs were annotated and evaluated (301 with complete model responses).
Hop-Count Taxonomy: A pre-specified four-level taxonomy was applied to annotate the reasoning depth required for each question:
- Hop 1: Direct retrieval of a single named fact.
- Hop 2: Two-fact composition, threshold application, or binary temporal comparison.
- Hop 3: Three-step reasoning chains involving cross-domain lookup, temporal arithmetic, or guideline application.
- Hop 4: Multi-component synthesis requiring integration of $\ge$ 4 EHR sources or time spans.
Annotation Reliability: Annotations were performed by LLMs and validated by three board-certified internal medicine physicians. Clinician majority-vote agreement with the primary LLM annotator was substantial ( $\kappa = 0.916$ ).

Experimental Design

The study evaluated three models across two providers (Anthropic, OpenAI) and two OpenAI generations:

Claude Sonnet 4-6 (Anthropic): Evaluated in two conditions:
- Zero-Shot (ZS): Direct answer without explicit reasoning instructions.
- Extended Thinking (ET): Enabled internal chain-of-thought with a 3,000-token budget.
GPT-4o (OpenAI): Evaluated in Zero-Shot condition.
GPT-5.4-2026-03-05 (OpenAI): Evaluated in Zero-Shot condition (flagship model at the time of evaluation).

Controls and Ablations:

Context Sufficiency Audit: Verified that EHR truncation (limited to 8,000 characters) did not differentially disadvantage higher-hop questions.
Reasoning Conditions: Additional ablations included a structured four-step explicit CoT prompt and an Extended Thinking condition with a 5 $\times$ larger budget (16,000 tokens).
Retrieval Augmentation: Tested BM25 and dense semantic RAG baselines to determine if retrieval alone could resolve the reasoning bottleneck.
Judge Validation: Responses were scored by an automated judge (GPT-4o-mini) validated against strict clinician adjudication ( $\kappa = 0.957$ ). Cross-provider validation was performed using Claude Haiku to rule out provider-family bias.

Key Results

1. Monotone Accuracy Decline (Prediction P1)

All three evaluated models exhibited a statistically significant, monotone decline in accuracy as compositional depth increased.

Claude Sonnet (ZS): Accuracy dropped from 30.6% (Hop 1) to 17.6% (Hop 4) ( $p=0.011$ ). Odds Ratio (OR) per hop: 0.72.
GPT-4o (ZS): Accuracy dropped from 37.8% to 14.7% ( $p<0.001$ ). OR per hop: 0.58.
GPT-5.4 (ZS): Accuracy dropped from 37.8% to 23.5% ( $p=0.027$ ). OR per hop: 0.80.
Robustness: This trend persisted after adjusting for question type, EHR length, and when using a 4 $\times$ larger context window (32,000 characters), confirming the degradation is due to reasoning difficulty rather than context availability.

2. Failure of Test-Time Reasoning to Flatten the Curve (Prediction P2)

Contrary to the hypothesis that Chain-of-Thought (CoT) reasoning would flatten the accuracy-versus-depth curve, extended thinking did not significantly mitigate the decline.

Extended Thinking (Claude): Showed a similar slope to Zero-Shot (34.2% to 19.6%), with no significant interaction effect between hop count and reasoning condition ( $p=0.72$ ).
Explicit CoT & Larger Budgets: Neither a structured four-step prompt nor a 5 $\times$ larger thinking budget (16,000 tokens) significantly flattened the curve.
Mechanism: While models allocated proportionally more computation to harder tasks (thinking tokens scaled with hop count, $r=0.31$ ), this increased compute did not translate to higher accuracy. In fact, correct answers required fewer thinking tokens than incorrect ones at matched hop levels, suggesting the model struggles to apply compute effectively to difficult compositions.

3. Error Type Shifts

The nature of errors changed systematically with depth:

Omission vs. Hallucination: As hop count increased, models shifted from hallucination to omission. At Hop 4, omission errors dominated (72.5% in ZS, 71.6% in ET), while hallucination rates declined.
Clinical Implication: Structured decomposition (Explicit CoT) further reduced hallucinations (0.3%) but increased omissions (76.1%). The authors note this is a favorable safety trade-off, as omission triggers clinician escalation, whereas confident hallucinations may not.

4. Retrieval Augmentation Limits

Both lexical (BM25) and dense semantic (RAG) retrieval baselines failed to significantly flatten the hop-accuracy slope. While retrieval improved absolute accuracy in some cases, the monotone decline persisted, indicating that multi-hop integration remains a reasoning bottleneck beyond simple fact retrieval.

Significance and Claims

The paper establishes hop count as a theory-motivated, cross-architecture predictor of LLM failure in clinical EHR QA. The findings suggest that the "compositional reasoning ceiling" is a qualitative limitation of current transformer architectures that is not resolved by simply increasing context windows, expanding thinking budgets, or applying standard retrieval augmentation.

Primary Contributions:

Taxonomy: A pre-specified hop-count taxonomy for stratifying clinical EHR QA tasks.
Empirical Evidence: Cross-architecture validation (Anthropic and OpenAI models) demonstrating that accuracy degrades monotonically with reasoning depth.
Mechanistic Insight: Evidence that test-time reasoning (CoT) does not currently overcome the compositional limits of transformers in this domain, despite increased token usage.
Safety Framework: A proposal to use hop-count stratification as a risk stratification tool for clinical AI deployment, moving beyond aggregate accuracy metrics to identify high-stakes tasks where models are most likely to fail.

The authors conclude that while aggregate accuracy metrics may suggest high performance, they conceal a systematic vulnerability in multi-hop reasoning. This necessitates a shift in evaluation frameworks to account for compositional depth, particularly for clinical applications where multi-step inference is critical.