🤖 AI

CheXthought: A global multimodal dataset of clinical chain-of-thought reasoning and visual attention for chest X-ray interpretation

This paper introduces CheXthought, a large-scale global multimodal dataset comprising over 100,000 chain-of-thought reasoning traces and 6.6 million visual attention annotations from 501 radiologists across 71 countries, which demonstrates significant improvements in factual accuracy, hallucination reduction, and model interpretability for chest X-ray interpretation compared to existing vision-language models.

Original authors: Sonali Sharma, Jin Long, George Shih, Sarah Eid, Christian Bluethgen, Francine L. Jacobson, Emily B. Tsai, Global Radiology Consortium, Ahmed M. Alaa, Curtis P. Langlotz

Published 2026-04-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sonali Sharma, Jin Long, George Shih, Sarah Eid, Christian Bluethgen, Francine L. Jacobson, Emily B. Tsai, Global Radiology Consortium, Ahmed M. Alaa, Curtis P. Langlotz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to read a chest X-ray. For years, scientists have fed the robot thousands of pictures paired with the final "answer key" (the radiologist's report). It's like showing a student a math problem and only the final number, without showing the work. The robot learns to guess the answer, but it doesn't actually understand how to solve the problem, and it often makes up facts to sound smart.

CheXthought is a new, massive dataset designed to fix this by teaching the robot the entire thought process, not just the answer.

Here is a breakdown of what the paper actually found, using simple analogies:

1. The "Global Study Group"

Instead of a few experts in one lab, the researchers built a "global study group" of 501 radiologists from 71 different countries.

The Analogy: Imagine a massive classroom where students from every continent are solving the same math problems together.
What they did: These doctors didn't just write down the final diagnosis. They spoke out loud their entire thought process ("I see a shadow here, let me check the heart size, is this old or new?") while simultaneously drawing a line on the screen showing exactly where they were looking.
The Result: They created a library of 103,000 thought traces and 6.6 million visual "dots" showing exactly where human eyes went and what they were thinking at that exact moment.

2. The "Human vs. Robot" Showdown

The researchers tested the best AI models (like GPT-5 and others) against these human thought processes.

The Analogy: It's like comparing a student who memorized the textbook answers to a student who actually understands the logic.
The Findings: The AI models were good at guessing the final answer, but they were terrible at explaining why.
- Hallucinations: When the AI couldn't see a problem, it often made one up to sound confident. The human doctors, however, would say, "I'm not sure," or "The image is blurry."
- Spatial Grounding: If you covered up the part of the X-ray with the disease, the AI often still claimed the disease was there (like a student guessing the answer without reading the question). The human-trained model was much better at realizing, "Hey, I can't see that part, so I can't diagnose it."

3. The "Search Pattern" Secret

The researchers discovered that human doctors don't just look randomly; they have specific "search patterns."

The Analogy: Think of looking for a lost set of keys.
- The "Broad" Searcher: Looks everywhere, from the ceiling to the floor. (This group was the most accurate).
- The "Central" Searcher: Only looks in the middle of the room.
- The "Narrow" Searcher: Only looks at the one spot they think the keys are.
The Finding: The "Broad" searchers found the most problems. The researchers then taught the AI to use these "Broad" search patterns as a hint.
The Result: When the AI was given a "hint" on where to look (based on human patterns), it stopped missing obvious problems and stopped making up fake ones. It was like giving the robot a map instead of letting it wander blindly.

4. Predicting "Confusion"

One of the most unique things this dataset allows is predicting when a case is tricky.

The Analogy: Imagine a teacher grading a test. If every student gets the same answer, the teacher knows the question was easy. If half the students get it right and half get it wrong, the teacher knows the question is confusing or hard.
The Finding: Because the dataset has 2 to 36 different doctors looking at the same image, the researchers could train the AI to predict: "This image is confusing even for humans," or "This image is easy, but the AI is likely to get it wrong."
The Benefit: This helps the AI know when to say, "I'm not sure, a human should check this," rather than confidently giving a wrong answer.

5. The "Time Travel" Test

Radiologists often look at a patient's X-ray from today and compare it to one from last year to see if things are getting better or worse.

The Analogy: It's like looking at a photo of a garden today and comparing it to a photo from last month to see if the flowers are blooming or dying.
The Finding: Most AI models get confused if you show them the photos in the wrong order (like showing the "dead flowers" photo before the "blooming" one). The model trained on CheXthought, however, learned to look at the actual visual changes, regardless of the order they were shown in.

Summary

CheXthought is a massive collection of "thinking aloud" and "looking here" data from doctors around the world. The paper proves that when you teach AI to mimic this human process—watching where they look, hearing how they reason, and acknowledging when they are unsure—the AI becomes:

More accurate at finding real problems.
Less likely to make up fake problems.
Better at knowing when it is confused.

The paper concludes that to build trustworthy medical AI, we need to stop just teaching them the answers and start teaching them the process of how humans actually think and look.

1. Problem Statement

Current Vision-Language Models (VLMs) for chest X-ray (CXR) interpretation are primarily trained on paired images and final radiology reports. This approach suffers from several critical limitations:

Lack of Cognitive Process: Models learn the output (the report) but not the underlying reasoning process (how experts arrive at conclusions).
Unfaithful Reasoning: State-of-the-art Chain-of-Thought (CoT) models often generate plausible-sounding rationales that are disconnected from their actual internal computations or the visual evidence (hallucinations).
Absence of Visual Attention: Existing datasets lack synchronized data linking verbalized reasoning to specific visual search patterns. Eye-tracking datasets exist but are limited by small sample sizes, single-institution bias, and the inability to capture the full step-by-step verbal reasoning process.
Uncertainty and Variability: Current models struggle to express diagnostic uncertainty or account for inter-reader variability, leading to "confident inaccuracy" which is dangerous in clinical settings.

2. Methodology

A. Dataset Construction: CheXthought

The authors created CheXthought, the first large-scale, global, multimodal dataset for CXR interpretation.

Scale & Diversity: Contains 50,312 multi-read chest X-rays (sourced from CheXpert Plus) annotated by 501 radiologists (residents, fellows, and attending staff) across 71 countries.
Annotation Process:
- Platform: A custom, browser-based DICOM viewer allowing global participation without specialized hardware.
- Dual-Modal Input: Annotators simultaneously dictated their clinical reasoning (CoT) and traced their visual attention (cursor movement) on the image.
- Data Volume: 103,592 reasoning traces and 6.6 million synchronized visual attention coordinates.
- Content: CoTs include hypotheses, visual observations, deliberations, differential diagnoses, and explicit uncertainty statements. Visual attention is recorded as image-relative coordinates synchronized with text segments.
Preprocessing: Non-English dictations were translated and verified by native speakers. Data was structured to capture temporal comparisons (serial images) and uncertainty.

B. Experimental Design

The study utilized CheXthought for three main purposes:

Human Reasoning Analysis: Characterizing expert search strategies, inter-reader agreement, and uncertainty expression.
Model Benchmarking: Blinded evaluation of human CoTs against state-of-the-art VLMs (GPT 5.2, Claude Opus 4.5, MedGemma 1.5).
Model Training & Evaluation: Fine-tuning a base model (Qwen3-VL-8B-Thinking) with different supervision signals to test the impact of CoT and visual attention.
- Models Trained:
  - CheXthought-VLM: Trained on Human CoT + Visual Attention.
  - CheXthought-CoT: Trained on Human CoT only.
  - Synthetic-CoT: Trained on AI-generated CoT.
  - CheXpert-Report: Trained on standard Image-Report pairs.
- Evaluation Metrics: Pathology classification accuracy, visual faithfulness (occlusion, image quality, temporal reasoning), and hallucination rates.

3. Key Contributions

Global Multimodal Dataset: A unique resource linking step-by-step verbal reasoning with synchronized visual attention from a globally diverse cohort of 501 radiologists.
Discovery of Search Trajectories: Identification of three distinct visual search strategies (Broad, Central, Narrow) via K-means clustering. The Broad strategy (comprehensive coverage) was found to be significantly more accurate (78.2%) than Narrow (68.3%).
Inference-Time Hinting: Demonstration that using expert search trajectories as "hints" during inference (without retraining) significantly recovers missed findings and reduces hallucinations in existing models.
Disagreement Prediction: A novel capability to predict Human-Human and AI-Human disagreement directly from an image, serving as a proxy for case complexity and model reliability.
Training Signal Analysis: Evidence that combining CoT supervision with visual attention data yields superior pathology classification and visual faithfulness compared to CoT alone or report-only training.

4. Key Results

A. Human Reasoning Patterns

Context Integration: Referencing clinical context improved accuracy for specific pathologies (e.g., Pneumothorax, Cardiomegaly) but had variable effects overall.
Visual Search: Experts typically start in the paramediastinal/hilar regions. Early detection of a finding did not guarantee higher accuracy; in fact, for pathologies like Pneumothorax, later attention (after a broader survey) correlated with higher accuracy.
Uncertainty: Uncertainty language increased with image suboptimality and diagnostic complexity. Confident CoTs were more likely to disagree with ground truth for visually salient findings (e.g., Pneumothorax), whereas hedged CoTs were more accurate for integrative diagnoses (e.g., Pneumonia).
Inter-Reader Agreement: Fleiss' $\kappa$ varied by pathology (0.92 for Pneumothorax vs. 0.23 for Pleural Other), highlighting that definitional ambiguity, not just rarity, drives disagreement.

B. Model Performance

Human vs. AI: Human-authored CoTs significantly outperformed all tested VLMs in Factuality (4.88 vs. ~4.1) and Spatial Localization (4.88 vs. ~3.5–4.3).
Training Impact:
- CheXthought-VLM achieved the highest overall pathology classification accuracy (62.7%) and F1 score (59.3%), outperforming report-only and synthetic-CoT baselines.
- Visual Faithfulness: CheXthought-VLM showed the lowest hallucination rates when relevant image regions were occluded (4% persistence vs. 47.4% for MedGemma 1.5) and best recognized degraded image quality.
- Temporal Reasoning: CheXthought-VLM correctly tracked serial image progressions in 95% of cases, significantly outperforming MedGemma 1.5 (59%).
Visual Attention Alignment: Surprisingly, training with visual attention coordinates did not align the model's internal Grad-CAM maps with human eye-tracking patterns (correlation decreased slightly). However, it still improved performance, suggesting the benefit comes from the reasoning linked to the attention, not the attention map itself.

C. Inference-Time Hints

Applying expert search trajectories (specifically the Narrow strategy) as text prompts during inference recovered 70–76% of missed pathologies in failing models and reduced hallucinations by ~2.0 findings per image.

D. Disagreement Prediction

The model successfully predicted human-human and AI-human disagreement rates with high calibration (deviation < 5% for most pathologies), enabling the system to flag cases where the model is likely to deviate from expert consensus.

5. Significance and Impact

Advancing Clinical AI: CheXthought moves beyond simple pattern recognition to model the cognitive process of radiology. This is crucial for developing AI that is transparent, interpretable, and safe.
Reducing Hallucinations: By grounding reasoning in visual attention and explicit uncertainty, the resulting models are less prone to "confident inaccuracy," a major barrier to clinical adoption.
Global Representation: The dataset's diversity (71 countries) mitigates the geographic and demographic biases prevalent in current medical imaging datasets, leading to more robust models for global deployment.
New Paradigm for Training: The study proves that visual attention data serves as a powerful, complementary supervision signal to text-based CoT, improving model reliability even if the internal attention maps do not perfectly mimic human gaze.
Actionable Uncertainty: The ability to predict disagreement allows for dynamic clinical workflows (e.g., flagging high-disagreement cases for senior review), directly addressing patient safety and regulatory needs.

In summary, CheXthought provides the first comprehensive resource to bridge the gap between visual evidence and clinical reasoning, demonstrating that training on human-verified, visually-grounded chain-of-thought reasoning significantly enhances the reliability, transparency, and clinical utility of medical AI.