💬 NLP

CARE: A Conformal Safety Layer for Medical Summarization

The paper introduces CARE, a post-hoc, model-agnostic safety layer that utilizes conformal risk control to provide formal, finite-sample guarantees for bounding both hallucinations and medically important omissions in LLM-generated medical summaries, thereby significantly reducing clinician review burden while maintaining rigorous safety standards.

Original authors: Suhana Bedi, Bridget Lin, Anson Y. Zhou, Chloe O. Stanwyck, Jenelle A. Jindal, Sanmi Koyejo, David Stutz, Nigam H. Shah

Published 2026-06-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Suhana Bedi, Bridget Lin, Anson Y. Zhou, Chloe O. Stanwyck, Jenelle A. Jindal, Sanmi Koyejo, David Stutz, Nigam H. Shah

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a highly skilled but occasionally careless medical scribe (an AI) who listens to a doctor's conversation with a patient and writes a summary note. This scribe is fast and smart, but it makes two specific types of mistakes:

The "Ghost" Mistake (Hallucination): It invents facts that never happened, like saying the patient has a fever when they don't.
The "Missing Page" Mistake (Omission): It forgets to write down crucial details the doctor mentioned, like a specific allergy or a new symptom.

In the past, if you wanted to catch these mistakes, you had to either read the entire note yourself (which is slow and boring) or use a "stop sign" system that would reject the whole note if it looked even slightly suspicious. But doctors don't want to throw away a whole note just because of one small error; they just want to know exactly where to look.

Enter CARE (Conformal Assessment for Risk Evaluation). Think of CARE as a smart, calibrated highlighter pen that runs over the AI's draft before a doctor sees it.

How the "Highlighter" Works

CARE doesn't rewrite the AI's work or retrain the AI. Instead, it acts as a safety layer that adds two types of colored flags to the text:

Red Flags: "Hey, this sentence looks like a 'Ghost' (hallucination). It might be made up."
Blue Flags: "Hey, this sentence from the original conversation is important, but the AI forgot to write it in the summary. You should check the original source for this."

The Secret Sauce: The "Risk Budget" Knob

The most clever part of CARE is how it decides how many flags to put on the page. It uses a concept called a "Risk Budget" (represented by the Greek letter alpha, $\alpha$ ).

Imagine you are the doctor in charge of the hospital. You have a Risk Budget of 15%. This means you are willing to accept that, on average, 15% of the dangerous mistakes might slip through without a flag.

If you set the budget low (very strict), the highlighter goes crazy, flagging almost everything. This is safe, but the doctor gets overwhelmed by too many flags.
If you set the budget high (very loose), the highlighter is lazy, flagging almost nothing. This is efficient, but dangerous.

CARE finds the perfect middle ground. It calculates the exact number of flags needed to stay within your 15% risk budget while flagging as few sentences as possible. It's like a smart security guard who knows exactly how many people to stop at the door to keep the building safe without causing a traffic jam.

Why This is Different (The "Two-Dimensional" Puzzle)

Most previous tools treated "missing information" as a simple yes/no question. But CARE realized that missing information is actually a two-dimensional puzzle:

Is the missing piece important? (Did the doctor mention a life-saving drug?)
Is it actually missing? (Did the AI forget to write it down?)

If you only check one of these, you either miss real dangers or flag too many unimportant things. CARE solves this by checking both at the same time. It's like a security scanner that checks for both "Is this a weapon?" and "Is this a weapon that matters right now?" simultaneously.

By doing this, CARE found that it could catch the same amount of errors as other methods but required doctors to review up to 5 times fewer sentences. It's the difference between asking a doctor to read 100 pages of notes versus just 20 pages of flagged highlights.

The Proof in the Pudding

The researchers tested this "highlighter" on five different types of medical notes (from radiology reports to discharge summaries).

The Result: In 100 different test runs, CARE successfully kept the "slip-through" rate at or below the 15% target.
The Human Test: They asked three real doctors to review notes with and without the CARE flags. With the flags, the doctors found 28.6% more missing information than they did without them. They also spent slightly less time reviewing the notes.

The Bottom Line

CARE is a tool that lets AI write medical notes quickly but adds a layer of mathematically guaranteed safety. It doesn't try to be perfect; instead, it gives doctors a tunable knob to decide how much risk they are willing to take in exchange for how much time they want to spend reviewing. It turns a chaotic, error-prone process into a targeted, efficient one, ensuring that when a doctor looks at a note, they know exactly where the potential traps are.

Technical Summary: CARE – A Conformal Safety Layer for Medical Summarization

1. Problem Statement

Large language models (LLMs) are increasingly deployed for medical summarization tasks, such as drafting clinical notes, converting radiology findings, and generating discharge summaries. However, these systems exhibit two critical failure modes that threaten clinical safety:

Omissions: Medically important content from the source text is absent from the summary. Studies report omission rates ranging from 3–4% in controlled settings to 47% in encounter-level evaluations.
Hallucinations: The summary contains claims unsupported by the source text. Rates range from 1–2% in controlled settings to 42% in open-ended encounters.

Current detection methods rely on heuristic scoring or "LLM-as-a-judge" evaluations that produce uncalibrated scores. These approaches lack a principled mechanism to select decision thresholds or formally control the rate of missed errors. Consequently, deployment faces a trade-off: permissive thresholds allow unsafe errors to reach clinicians, while conservative thresholds overwhelm clinicians with excessive flags, eroding the efficiency gains of AI-assisted documentation. Furthermore, existing conformal prediction methods for text generation typically operate at the response level (abstaining from or rewriting entire outputs), which is misaligned with clinical workflows that require targeted, sentence-level guidance rather than wholesale rejection of a draft.

2. Methodology: CARE Framework

The authors propose CARE (Conformal Assessment for Risk Evaluation), a post-hoc, model-agnostic safety layer that overlays calibrated omission and hallucination flags onto LLM-generated summaries without retraining the underlying summarizer. CARE utilizes Conformal Risk Control (CRC) to provide finite-sample, distribution-free guarantees.

2.1 Pipeline Overview

The system operates in four phases:

Oracle Labeling (Offline): A high-capacity oracle LLM (GPT-5) generates reference labels for a calibration set. These labels define:
- Support: Whether a summary sentence is supported by the source ( $Y_{sup}$ ).
- Importance: Whether a source sentence is medically important ( $Y_{imp}$ ), inferred from a clinician-written reference summary.
- Coverage: Whether a source sentence is represented in the summary ( $Y_{cov}$ ).
  A "true omission" is defined as an important source sentence not covered by the summary.
Scoring by Judge Model (Online/Offline): A cheaper "judge" model (GPT-5-mini) scores sentences at deployment. Instead of binary labels, it uses a three-tier rubric (Supported/Essential, Partial/Relevant, Unsupported/Omitted) mapped to scores $\{0, 0.5, 1\}$ . To reduce stochasticity, the judge provides $m=5$ independent replicates per sentence, which are averaged.
Conformal Risk Control (Offline): The system selects thresholds that guarantee risks stay below user-specified levels ( $\alpha$ $α$ ).
- Hallucination Controller: Flags summary sentences with support scores $\hat{p}_{sup} \le \lambda$ . It bounds the probability that a document contains at least one unflagged hallucinated sentence.
- Omission Controller: Surfaces source sentences where importance $\hat{p}_{imp} \ge \tau$ AND non-coverage $\hat{p}_{ncov} \ge \gamma$ . It bounds the expected fraction of true omissions not surfaced for review.
Deployment: CARE applies the calibrated thresholds $(\lambda^*, \tau^*, \gamma^*)$ to new summaries, returning the original draft augmented with red flags (hallucinations) and blue flags (important omissions).

2.2 Key Technical Innovation: Joint 2D Calibration

A central insight of the paper is that omission control is inherently a two-dimensional calibration problem over importance ( $\tau$ ) and non-coverage ( $\gamma$ ).

The Challenge: Calibrating only one dimension (e.g., fixing $\gamma=0$ ) or using marginal decompositions (e.g., splitting the risk budget $\alpha/2$ per dimension via union bound) can either violate the target risk bound or be overly conservative, surfacing too many sentences.
The Solution: CARE employs the Learn-Then-Test Fixed-Sequence (LTT-FST) procedure. It defines a deterministic, data-independent ordering of the $(\tau, \gamma)$ grid (prioritizing lower workload) and selects the first threshold pair that satisfies the empirical risk bound. This approach preserves formal guarantees while minimizing the number of surfaced sentences.

3. Key Contributions

A Conformal Framework for Omission and Hallucination: CARE introduces the first framework providing post-hoc, model-agnostic, sentence-level risk guarantees for both failure modes in abstractive medical text generation.
Joint Two-Dimensional Calibration: The authors demonstrate that partial calibration violates risk bounds, while marginal decompositions are valid but inefficient. Joint calibration over the full $(\tau, \gamma)$ space preserves guarantees while surfacing up to 5× fewer sentences than alternative calibrated baselines.
Multi-Domain Validation: The framework was validated across five medical summarization tasks (dialogue-to-note, hospital-course synthesis, radiology, discharge summaries, and biomedical abstracts) using 2,123 documents.
Clinical Utility Evidence: A preliminary within-subjects study with three clinicians showed that CARE's flags improved omission detection by 28.6 percentage points (from 50.4% to 79.0%) while slightly reducing review time per sentence.

4. Experimental Results

The experiments were conducted across five datasets (ACI-Bench, MIMIC-BHC, MIMIC-CXR, Priv-DS, SumPubMed) with a target risk level of $\alpha = 0.15$ .

Safety Guarantees: Across 100 random calibration/test resplits, CARE satisfied the target risk bound for both controllers. Empirical mean violation rates remained at or below $\alpha$ (e.g., hallucination violations ranged from 5.7% to 12.8%; omission violations ranged from 9.3% to 14.0%).
Efficiency: CARE flagged 8–24% of summary sentences for hallucinations and surfaced 11–66% of source sentences for omissions (depending on dataset compression and length).
Comparison to Baselines:
- LTT-FST (CARE) significantly outperformed 1D and Union Bound baselines in reducing review workload while maintaining guarantees. For example, on ACI-Bench, LTT-FST surfaced 18.9% of sentences compared to 48.9% for 1D-Imp.
- Uncalibrated baselines (e.g., Max-F1, Fixed-0.5) often reduced workload but violated risk bounds significantly (e.g., violation rates up to 47.8%).
- Partial Calibration: Leaving the coverage dimension uncalibrated resulted in catastrophic failure, with violation rates reaching 49.8%.
Robustness: The system remained robust to variations in calibration size (requiring only ~100 labeled documents), prompt design, and summarizer choice (tested with Llama-3.3-70B, Gemini 2.5 Pro, and Claude Opus 4.6).

5. Significance and Claims

The paper claims that sentence-level safety guarantees are feasible for LLM-assisted medical summarization. By moving beyond response-level abstention, CARE offers a tunable mechanism (via the risk budget $\alpha$ ) for balancing residual risk against clinician review effort.

The authors emphasize that their guarantees are marginal (over the calibration distribution) and finite-sample. They acknowledge limitations, including the reliance on exchangeability between calibration and deployment data, the potential for oracle labeling errors (though validated against clinician annotations), and the binary nature of coverage labels. The preliminary clinician study is presented as "directional evidence" rather than definitive proof of downstream clinical impact, noting the need for larger, workflow-oriented studies to assess trust, alert burden, and override behavior.

Ultimately, CARE demonstrates that conformal risk control can be operationalized to provide targeted, clinician-actionable interventions, enabling safer deployment of LLMs in high-stakes medical documentation.