💬 NLP

Checkup2Action: A Multimodal Clinical Check-up Report Dataset for Patient-Oriented Action Card Generation

This paper introduces Checkup2Action, a multimodal dataset and benchmark comprising 2,000 de-identified clinical check-up reports designed to evaluate and improve large language models' ability to generate safe, structured, and patient-oriented action cards from complex medical evidence.

Original authors: Sike Xiang, Shuang Chen, Kevin Qinghong Lin, Jialin Yu, Yijia Sun, Philip Torr, Amir Atapour-Abarghouei

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sike Xiang, Shuang Chen, Kevin Qinghong Lin, Jialin Yu, Yijia Sun, Philip Torr, Amir Atapour-Abarghouei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you just finished a full-body health check-up. You walk away with a thick, multi-page report. It's a chaotic mix of numbers, charts, arrows pointing up and down, blurry X-ray images, and dense medical jargon. For a regular person, trying to figure out "What does this mean for me?" and "What do I do next?" is like trying to read a map written in a language you don't speak, where the landmarks are scattered across different pages.

This paper introduces a new tool called Checkup2Action to solve that problem. Think of it as a "translator" that turns that confusing medical report into a simple, prioritized "To-Do List" for the patient.

Here is a breakdown of what the paper does, using simple analogies:

1. The Problem: The "Medical Maze"

When you get a check-up, the results are multimodal. That's a fancy way of saying they come in many different formats:

Tables of numbers (like blood pressure).
Images (like X-rays).
Flags (red arrows indicating something is "abnormal").
Text from the doctor.

Currently, if you look at this, you might see a red arrow next to your cholesterol and panic, or you might miss a subtle note about your heart because it was buried on page 12. The paper calls this the "interpretability gap." The information is there, but it's not organized into a clear plan of action.

2. The Solution: The "Action Card"

The researchers built a dataset and a system called Checkup2Action. Instead of giving you a 20-page report back, this system generates Action Cards.

Think of an Action Card like a travel itinerary for your health. Instead of a wall of text, you get a series of clear, prioritized cards. Each card focuses on one specific issue and answers four simple questions:

What is the issue? (e.g., "Your cholesterol is a bit high.")
How urgent is it? (e.g., "High," "Medium," or "Just keep an eye on it.")
Who should you see? (e.g., "Go to a Cardiologist" or "See your GP.")
When should you go? (e.g., "Within a week" or "Next year's check-up.")
What questions should you ask? (e.g., "Ask if I need to change my diet.")

Crucially, the system is designed not to act like a doctor. It doesn't say, "You have heart disease." It says, "Here is a finding that needs a professional's attention." It's a guide, not a diagnostician.

3. The Dataset: The "Training Gym"

To teach computers how to do this, the researchers created a massive library called C2A.

It contains 2,000 real, anonymous check-up reports.
These reports are like the "textbooks" the AI studies.
They also have "answer keys" created by real doctors, showing what the correct Action Cards should look like for each report.

4. The Experiment: Who is the Best Translator?

The researchers tested several famous AI models (like GPT-5, Gemini, and Claude) to see which one could turn the messy reports into good Action Cards.

They found a few interesting things:

The "Over-achiever" vs. The "Safe" Bot: Some AIs were great at finding every possible problem (high recall) but sometimes got the priority wrong or suggested too many actions. Others were very safe and conservative, missing some issues but giving very accurate advice on the ones they did find.
General vs. Medical AIs: Surprisingly, a general-purpose AI (trained on all kinds of internet data) often did a better job at creating patient-friendly plans than a specialized medical AI. The specialized AI was great at diagnosing diseases but sometimes forgot to keep the language simple and safe for a regular person.
The Safety Filter: The most important finding was about safety. When they told the AI, "Do not diagnose diseases, just suggest next steps," the AI became much better at its job. If they removed this rule, the AI started guessing diseases (like "You might have cancer"), which is dangerous and not what the patient needs.

5. The Bottom Line

This paper isn't about replacing doctors. It's about building a bridge between a confusing medical report and a patient's next steps.

Before: Patient gets a 20-page PDF, feels confused, and doesn't know what to do.
After: Patient gets a stack of 3 or 4 clear "Action Cards" telling them exactly what to prioritize, who to call, and when.

The researchers proved that with the right rules (safety constraints) and the right data, AI can help turn a scary, complex medical document into a clear, manageable plan for everyday people.

Technical Summary: Checkup2Action

Problem Statement

Routine clinical check-ups generate complex, multimodal reports that integrate visual layouts, tabular laboratory results, numerical biomarkers, abnormality flags, imaging findings, and free-text physician comments. While these documents contain critical health information, laypersons often struggle to interpret them due to the "interpretability gap" between heterogeneous clinical evidence and concrete patient actions. Patients frequently misinterpret isolated cues (e.g., arrows or abnormal labels) or fail to synthesize distributed findings into prioritized next steps. Existing Large Language Models (LLMs) and medical AI agents have shown promise in summarization and triage but generally produce free-form explanations or acuity labels rather than structured, patient-oriented action plans. Furthermore, current benchmarks lack the ability to evaluate whether systems can consistently identify clinically relevant issues, rank them by priority, recommend appropriate departments, and provide safe, actionable guidance without overstepping into diagnostic or prescriptive roles.

Methodology

The paper introduces Checkup2Action (C2A), a multimodal dataset and benchmark designed to evaluate the generation of structured "Action Cards" from clinical check-up reports.

1. The C2A Dataset

The dataset comprises 2,000 de-identified, real-world check-up reports in PDF format. Each report is a multi-page document containing:

Five major categories: Basic Information (demographics/vitals), General Physical Examination, Laboratory Tests, Imaging Examinations, and Cardiovascular Tests.
Multimodal complexity: Reports include visual layouts, structured tables, numerical measurements, abnormality flags, embedded images, and physician summaries.
Annotations: Reference issues and action-card attributes are derived from physician-written report summaries. The average report contains 7.2 issues, with a priority distribution of 7.4% High, 20.8% Medium, and 71.8% Low.

2. The Checkup2Action Workflow

The authors instantiate a constrained baseline workflow (Figure 3) that transforms a raw report into an ordered sequence of JSON-formatted Action Cards. The pipeline consists of three steps:

Step 1: Report Ingestion: The input PDF is parsed and linearized to extract text blocks, tables, and image objects while removing layout noise.
Step 2: Multimodal Segmentation: The linearized content is organized into structured sections (e.g., lab tests, imaging) using a hybrid strategy of tool-based parsing, rule-based localization, and LLM verification.
Step 3: Schema-Constrained Generation: The model generates Action Cards under strict behavioral constraints. Each card is a 6-tuple containing: problem, priority, why (explanation), department, time_suggestion, and questions.
- Critical Constraint: The system is explicitly prohibited from making definitive diagnostic statements or prescribing treatments. It acts as an "action translator," reorganizing existing findings into follow-up guidance.

3. Evaluation Protocol

The benchmark employs a dual-component evaluation framework:

Structured Consistency: Measures alignment with reference annotations using:
- Problem Recall/Precision: Based on semantic similarity (cosine similarity $\tau=0.8$ ) between generated and reference issues.
- Attribute Accuracy: Priority, Department, and Time Accuracy (matching reference categories).
- Action Complexity Bias: Measures deviation in the number of generated cards vs. reference cards.
Subjective Quality: Human-rated dimensions (1–5 scale) covering Problem Relevance, Safety, Usefulness, Clarity, and Tone.

Key Results

Experiments were conducted on various general-purpose and medical-domain LLMs (including GPT-5.1, Gemini variants, Claude variants, and MedGemma-27B).

Trade-offs in Performance: No single model dominated all metrics. Gemini-3-pro-preview achieved the highest Problem Recall (0.602) but lagged in priority accuracy and action complexity control. GPT-5.1 demonstrated the most balanced performance, achieving high accuracy in priority (0.719), department (0.844), and time recommendations (0.915), alongside strong subjective scores.
General vs. Medical Models: Surprisingly, the general-purpose model GPT-5.1 outperformed the medical-domain backbone MedGemma-27B on downstream decision metrics (priority, department, and time accuracy). The authors attribute this to the training objectives of general models (RLHF alignment for instruction following and layperson communication) being better suited for the "no diagnosis, only action guidance" constraint than models trained primarily on professional diagnostic reasoning.
Safety Ablation: Removing safety constraints significantly increased Problem Recall (from 0.527 to 0.825) but drastically reduced Safety and Problem Relevance scores. Unconstrained models tended to produce diagnostic statements (e.g., "early-stage lung cancer"), whereas constrained models correctly limited outputs to objective findings and follow-up suggestions.
Subjective Quality: Models with higher recall did not necessarily yield better subjective scores. GPT-5.1 received the highest overall subjective ratings, while models with conservative behaviors (like Claude-haiku-4.5) showed high safety but lower usefulness due to under-generation.

Significance and Contributions

The paper claims three primary contributions:

C2A Dataset and Benchmark: The introduction of a real-world, multimodal dataset of 2,000 check-up reports with expert annotations specifically for patient-oriented Action Card generation.
Structured Task Formulation: The definition of checkup-to-action generation as a constrained structured generation task, accompanied by a comprehensive evaluation protocol that jointly measures issue coverage, prioritization, department/time recommendation, and safety compliance.
Constrained Baseline Workflow: The instantiation of Checkup2Action as a workflow that successfully converts multi-section evidence into prioritized, patient-facing cards while strictly avoiding diagnostic or treatment-prescriptive outputs.

The authors position Checkup2Action as a necessary step toward bridging the gap between complex clinical evidence and patient action. They emphasize that the benchmark reveals clear trade-offs between issue coverage, action correctness, conciseness, and safety alignment, suggesting that future research should focus on developing dedicated models for multimodal check-up-to-action generation rather than relying solely on general-purpose or diagnostic-focused medical models.