🤖 AI

EarlyDx: An Admission-Anchored Benchmark for Open-Ended Generation of Evidence-Supported ED-Encounter Diagnoses

This paper introduces EarlyDx, a large-scale benchmark for open-ended, evidence-supported emergency diagnosis using admission-only data from MIMIC-IV, which reveals that current large language models struggle to reliably infer diagnoses from limited initial evidence despite post-training improvements.

Original authors: Jiahui Li, Ruili Fang, Zishuai Liu, Yutong Guo, Nan Yang, Wenzhan Song, Jin Lu, Fei Dou

Published 2026-08-03

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiahui Li, Ruili Fang, Zishuai Liu, Yutong Guo, Nan Yang, Wenzhan Song, Jin Lu, Fei Dou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but you are only allowed to look at the crime scene for the first five minutes. You have a notebook full of clues: a muddy footprint, a broken window, and a witness who is still shaking. Your job is to guess what happened right now, before the police arrive, before the lab tests come back, and before the suspect confesses. This is the daily reality for doctors in an Emergency Room (ER). They must make a "working diagnosis"—a best guess about what is wrong with a patient—using only the limited information available the moment the patient walks through the door.

For a long time, scientists have tried to teach computers to do this job using Artificial Intelligence (AI). They built "benchmarks," which are like test scores for AI, to see how smart these machines are. But most of these tests were rigged. They gave the AI the whole story of the patient's hospital stay (including the final diagnosis made days later) and asked it to predict the beginning. It's like giving a detective the solution to the mystery and asking them to guess the clues. Furthermore, these tests forced the AI to pick from a short, fixed list of answers, like a multiple-choice quiz, rather than letting them write their own answer in plain language. This paper, called EarlyDx, decides to fix the game. It builds a new, fairer test that forces AI to act like a real ER doctor: looking only at the evidence available at the exact moment of admission and writing out a diagnosis in free text, just like a human would.

The New Game: EarlyDx

The researchers created a massive new playground called EarlyDx, built from over 154,000 real emergency room visits. Imagine a giant library of patient stories. In this library, they carefully cut off every story right at the moment the patient was admitted to the hospital. They threw away any notes written later, any lab results that came back the next day, and any final diagnoses made after the patient was fully treated. They kept only the "admission-time" evidence: the patient's age, their chief complaint (what hurts), their vital signs, and any immediate tests like X-rays or heart monitors that were done right then.

But here is the tricky part: the "answers" in this library aren't just the final diagnoses. The researchers used a special AI "auditor" to check every single diagnosis recorded in the patient's file. They asked: "Can this diagnosis be proven only by the evidence we have at the admission moment?"

Supported: The evidence is right there (e.g., an X-ray shows a broken bone).
Partially Supported: The evidence is a hint, but not a slam dunk (e.g., the patient has a history of diabetes, but we don't have a blood test yet).
Unsupported: The diagnosis is true, but we couldn't have known it at admission (e.g., a blood culture that takes three days to grow).

The AI models are only graded on the "Supported" and "Partially Supported" answers. If the AI guesses a diagnosis that requires information we didn't have yet, it gets no credit. This ensures the test measures true reasoning, not just lucky guessing or memorizing the final answer key.

The Big Surprise: AI Relies on Explicit Text

When the researchers ran their tests, they found something surprising. They tested the world's smartest AI models, including huge general-purpose ones and specialized medical ones. The results showed that most of these "zero-shot" models (models that haven't been specifically trained on this new test) are actually terrible at inferring a diagnosis.

Instead of thinking like a detective, they act like a photocopier. About 43% of the time, the correct diagnosis was written verbatim in the patient's notes (e.g., the note said "suspected pneumonia," and the AI just copied "pneumonia"). When the AI had to infer a diagnosis—connecting dots that weren't explicitly stated—these models collapsed. They only managed to find 3% to 31% of the hidden diagnoses. It's like a student who can copy the answer if it's written on the board but fails completely if they have to solve the math problem themselves.

Even when the researchers "fine-tuned" (retrained) a smaller AI model on this specific task, it got much better, but it still didn't reach the level of a human doctor. The fine-tuned model could find about 56% of the hidden diagnoses, which is a big improvement, but it still left a lot of critical cases behind.

The Human Factor and the "Time-Critical" Gap

The study also compared these AI models to a real human doctor who looked at the same admission notes. The human doctor was much better at finding the hidden clues, catching about 68% of the supported diagnoses. However, the human doctor also listed many more possibilities than the AI, creating a "differential diagnosis" (a list of things it could be). The AI models tended to either list too few things (missing the danger) or too many random things (being too vague).

The most worrying finding came when they looked at time-critical conditions—life-or-death emergencies like heart attacks, strokes, or sepsis. In these cases, missing a diagnosis is far worse than making a false alarm. The human doctor strikes a careful balance: they flag a lot of potential dangers to be safe. The AI models, however, struggled to find this balance. Some were so cautious they missed the danger; others were so reckless they flagged everything. No AI system tested so far could match the human doctor's ability to say, "This looks dangerous, let's check it out," with the right level of confidence.

The Verdict

The paper concludes that while AI is getting better at reading medical notes, it is still not ready to replace the human detective work required in an emergency room. The current models are mostly good at extraction (finding what is already written) but bad at inference (figuring out what is implied). The researchers suggest that for AI to be truly useful, it needs to learn how to reason with uncertainty, admitting when it's not sure and flagging potential dangers, rather than just trying to guess the final answer. Until then, the "early diagnosis" game remains a human specialty, with AI acting as a helpful but imperfect assistant.

Technical Summary: EarlyDx

Problem Statement

Clinical diagnosis at hospital admission requires synthesizing limited, incomplete evidence (vitals, labs, imaging, history) to form a working diagnosis under uncertainty. Existing benchmarks for medical diagnosis are ill-suited for evaluating modern Large Language Models (LLMs) in this specific setting due to three primary limitations:

Closed-Set Labels: Diagnoses are mapped to fixed ICD code sets, discarding clinical nuance and information lost during text-to-code conversion.
Restricted Modalities: Benchmarks often exclude rich free-text signals (chief complaints, radiology findings) in favor of tabular features.
Temporal Inconsistency: Models receive admission-time inputs but are supervised with discharge diagnoses that incorporate the full inpatient course, including information unavailable at the time of admission.

Consequently, current benchmarks fail to distinguish between a model's ability to extract explicit information from a record and its ability to infer diagnoses from partially observed signals.

Methodology

The authors introduce EarlyDx, a large-scale benchmark ( $N=154,834$ encounters) derived from MIMIC-IV, designed to evaluate open-ended early diagnosis anchored strictly to the admission time ( $t_0$ ).

Data Construction

Cohort: Includes all emergency department (ED) encounters resulting in hospital admission.
Input: A serialized text prompt containing only records available at or before $t_0$ (admission time). This includes demographics, chief complaint, triage vitals, serial ED vitals, home medications, baseline measurements, lab results, ECG interpretations, radiology findings, and prior history. Imaging and physiological signals are rendered as report findings, not raw data.
Labels: Reference diagnoses are the open-vocabulary ICD titles from the ED-encounter diagnosis table, rather than closed-set codes or discharge diagnoses.

Evidence-Grounded Verification

A critical innovation is the use of an LLM auditor (MiniMax M3) to verify every free-text label against the admission-time evidence. Labels are categorized into:

Supported: Direct evidence in the input (labs, imaging, vitals) substantiates the diagnosis.
Partially Supported: Evidence is indirect or circumstantial (e.g., past history, home medications) but not definitive at admission.
Unsupported: The diagnosis depends on information unavailable at admission (e.g., microbiology cultures, post-admission workup).

The primary evaluation track scores models only on fully supported labels (32.7% of total labels), ensuring the benchmark measures diagnostic inference rather than retrieval of future information.

Evaluation Protocol

Semantic Matching: Exact string matching is unreliable for free text. The authors employ an LLM-as-judge protocol (Qwen3.5-27B) to compute one-to-one matching between predicted and reference diagnoses, handling synonyms and abbreviations.
Metrics: Micro-averaged Precision, Recall, and F1 are reported.
Baselines: The study evaluates zero-shot general LLMs (e.g., GPT-5.5, Claude Opus), medical-specialized LLMs (e.g., MedGemma, HuatuoGPT), and a post-trained model (Qwen3.5-4B fine-tuned with gold-conditioned Chain-of-Thought).

Key Contributions

Admission-Anchored Benchmark: A large-scale dataset where supervision is strictly limited to evidence available at admission, using open-ended free-text labels rather than closed codes.
Evidence Verification: A pipeline that filters labels based on whether they are supported by the input evidence, separating "extraction" tasks from "inference" tasks.
Empirical Analysis of LLM Capabilities: A systematic evaluation showing that current LLMs struggle with early diagnosis inference, often relying on text extraction rather than reasoning over uncertainty.

Results

Performance Gap: Zero-shot models (both general and medical) generally score below 0.40 F1 on the primary track. Post-training yields significant gains, with the fine-tuned Qwen3.5-4B achieving an F1 of 0.51 on supported labels, outperforming the strongest zero-shot baseline by +0.11.
Extraction vs. Inference:
- 43% of supported diagnoses appear verbatim in the input (explicit). Zero-shot models recover 37–65% of these.
- 57% of supported diagnoses are implicit (must be inferred). Zero-shot models recover only 3–31% of these, with medical-specialized models recovering as little as 3–6% and general models recovering 9–31%.
- Post-training improves implicit recall to 56% for the fine-tuned model, but a significant gap remains compared to human performance.
Time-Critical Conditions: On six time-critical diagnoses (e.g., myocardial infarction, sepsis), no evaluated system achieves a clinician's balance of sensitivity and precision. Zero-shot models tend to have high sensitivity but very low precision (25–28%), while the post-trained model achieves high precision (78%) but lower sensitivity (54%).
Human Reference: A human clinician evaluated on the same inputs achieved the highest supported recall (0.68) but a comparable F1 to top zero-shot models, largely due to over-prediction (listing a broad differential), suggesting that even experts struggle to narrow the differential without the full clinical encounter.

Significance and Claims

The paper claims that EarlyDx reveals a fundamental limitation in current LLMs: they are effective at extracting diagnoses explicitly stated in records but fail to synthesize evidence to infer diagnoses under uncertainty.

Negative but Useful Finding: The benchmark demonstrates that "frontier" general and medical LLMs do not reliably synthesize admission-time evidence. The bottleneck is not medical knowledge, but the capacity to translate that knowledge into a calibrated diagnosis under uncertainty.
Evaluation Validity: By filtering for evidence-supported labels, the benchmark avoids the "temporal leakage" of using discharge diagnoses, providing a more rigorous test of early diagnostic capability.
Limitations: The authors note that the reference labels are administrative billing codes (not adjudicated clinical diagnoses) and that the input lacks physical examination findings (e.g., patient appearance), which limits the achievable ceiling for both models and humans. They suggest that future work should explore graded predictions (e.g., "suspected" diagnoses) to better align with clinical practice where unconfirmed dangers must be raised.

The paper concludes that while post-training narrows the gap, no current system reaches a clinician's operating point on time-critical conditions, highlighting the need for further research in evidence-grounded reasoning under uncertainty.