💻 computer science

ConfTriage: A Calibration-Aware LLM Triage Framework for Pulmonary Nodule Malignancy with Selective Specialist Deferral

ConfTriage is a calibration-aware framework that leverages a generalist LLM to predict pulmonary nodule malignancy from structured radiological descriptions, selectively deferring uncertain cases to a specialist deep learning model to achieve high diagnostic accuracy while minimizing reliance on image-trained models.

Original authors: Md Rabiul Islam, Samir Abdaljalil, Erchin Serpedin, Hasan Kurban

Published 2026-08-12

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Md Rabiul Islam, Samir Abdaljalil, Erchin Serpedin, Hasan Kurban

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of a crime scene, you are looking at a tiny, fuzzy cloud inside a person's lung. This is the world of pulmonary nodule detection, where doctors use special CT scans to find these little bumps. Most of these bumps are harmless, like harmless moles on a skin, but some are dangerous, like a ticking time bomb. The problem is that there are so many of these clouds that human doctors get overwhelmed, staring at thousands of images and trying to decide which ones need immediate attention and which ones can be ignored.

For a long time, scientists tried to build super-smart computer programs (called "specialist AI") that could look at the raw pictures of these clouds and tell the difference between safe and dangerous. These programs are like a master detective who has studied millions of crime scene photos; they are very good, but they are also very picky. They need to be trained specifically on pictures, they are hard to explain to humans, and they can't easily talk about why they think something is suspicious.

Enter the "Generalist AI," or Large Language Model (LLM). Think of these as the ultimate trivia masters. They have read almost everything written on the internet, including medical textbooks and doctor's notes. They are amazing at understanding words and stories, but they aren't usually trained to look at raw pixels of an X-ray. The big question scientists have been asking is: Can a word-smart AI, which has never seen a picture of a lung nodule, still solve the mystery just by reading a doctor's written description of it? And if it does, can we trust its confidence enough to let it make the first call, or do we need to double-check its work?

This paper introduces a clever new system called ConfTriage (short for Confidence Triage) to answer exactly those questions. The researchers wanted to see if a generalist AI could act as a smart "gatekeeper" for lung nodules, sorting the easy cases from the hard ones so that human doctors and specialized image-AIs only have to deal with the tricky stuff.

Here is how they did it and what they found:

The "Language-Only" Surprise
The team set up a massive experiment using five different top-tier AI models (from companies like OpenAI, Google, and others). They gave these models a lung nodule case in seven different ways:

Just a list of numbers (like "size: 5mm, shape: round").
Just a natural language story describing the nodule (like "a small, round nodule with jagged edges").
Just a few basic stats from the image (like "average brightness" or "texture patterns").
Various combinations of the above.

The results were a huge surprise. When the AI tried to guess the danger level using only the image statistics (the raw numbers from the picture), it was essentially guessing at random, performing no better than a coin flip. It was as if the AI was looking at a blurry photo and trying to guess the color of the sky by counting pixels.

However, when the AI was given a natural language description of the nodule—just the words a radiologist would write in a report—it became a star detective. The AI could identify dangerous nodules with incredible accuracy, reaching a score of 0.907 (on a scale where 1.0 is perfect). This suggests that the "secret sauce" of diagnosing these nodules isn't hidden in the raw pixels, but in the specific words doctors use to describe them. The AI, having read millions of medical texts, already knew that words like "spiculated" (jagged) or "lobulated" (bumpy) are red flags.

The "ConfTriage" Strategy
Knowing that the AI is good at reading descriptions, the team built a safety system called ConfTriage. They realized that even smart AIs can be overconfident. Sometimes an AI might say, "I'm 99% sure this is safe!" when it's actually wrong. To fix this, they added a "calibration" step.

Think of calibration like tuning a musical instrument. The AI's raw confidence is a bit out of tune. The researchers used a mathematical trick (called Platt scaling) to adjust the AI's confidence scores so they actually match reality. If the AI says it's 80% sure, it should be right 80% of the time.

Once the AI was tuned, they set up a rule:

If the AI is very confident (either very sure it's safe or very sure it's dangerous), it makes the call.
If the AI is unsure (hovering in the middle), it immediately passes the case to a "Specialist Backstop." This backstop is a traditional, image-trained AI (called Certain-Net) that is super good at looking at pictures but requires a lot of training.

The Results: A Perfect Team-Up
This team-up worked beautifully. The ConfTriage system was able to solve 76.5% of all cases using only the generalist AI reading the text. It didn't need to look at a single image for these cases. It only sent the remaining 23.5% of the "confusing" cases to the specialist image-AI.

The final system achieved an F1 score of 88.22% and an AUC of 0.92. This means it was highly accurate, but more importantly, it was efficient. It saved the heavy-duty image-processing power for the cases that really needed it.

What They Ruled Out
The paper is very clear about what doesn't work. They explicitly showed that feeding raw image statistics (like histograms of pixel brightness) into a generalist AI is useless for this task. The AI just couldn't make sense of it. They also showed that while some AI models are better than others, the "language-only" approach worked well across almost all of them, suggesting this isn't just a fluke of one specific company's model.

How Sure Are They?
The authors are quite confident in their findings because they didn't just guess; they proved it with math. They developed two mathematical theorems to back up their system.

One theorem proves that their combined system (AI + Specialist) has a guaranteed error rate that won't exceed a certain limit, even with a small amount of data.
The other theorem shows that if the AI's confidence is well-calibrated (tuned correctly), the system is almost as good as the theoretically perfect decision-maker.

They tested this on a public dataset of 955 lung nodules, using a rigorous method where they checked the AI's work against a consensus of human radiologists. They even ran "corruption tests," where they scrambled the words or changed the meaning of the descriptions, and the AI's performance dropped, proving it was actually understanding the medical meaning and not just memorizing random words.

The Bottom Line
ConfTriage suggests a new way to use AI in medicine. Instead of building a giant, expensive computer that tries to do everything, we can use a smart, language-savvy AI to handle the easy cases by reading doctor's notes. It acts like a skilled triage nurse, sorting the patients and only calling in the specialist surgeon (the image-AI) for the complex ones. This saves time, reduces costs, and keeps the system safe by admitting when it doesn't know the answer. While this study was done on public data and needs real-world testing in hospitals, it offers a promising roadmap for how general AI and specialist AI can work together to help doctors save lives.

Problem Statement

Pulmonary nodule malignancy prediction currently relies heavily on specialized deep learning (DL) models trained on 3D CT volumes. While these models achieve high accuracy (e.g., >91% on LIDC-IDRI), they require task-specific training, lack native linguistic interpretability for clinicians, and offer no standard interface for explanation. Conversely, generalist Large Language Models (LLMs) can process natural language and provide rationales but have not been rigorously evaluated for this specific task regarding: (1) the specific input modality that drives their predictions (structured text vs. image statistics), (2) the reliability of their confidence scores for operational use, and (3) the existence of a deferral policy with formal performance guarantees.

Methodology: ConfTriage Framework

The authors propose ConfTriage, a three-pillar methodological framework designed to use a generalist LLM as a calibrated triage layer, deferring uncertain cases to a specialist DL backstop.

1. Input Modality (Language as the Modality)
Instead of feeding raw CT pixels or low-level image statistics to the LLM, the framework converts structured radiological attributes into a deterministic natural-language description.

Data Source: The study uses the LIDC-IDRI dataset (955 nodules), extracting eight standard attributes (subtlety, internal structure, calcification, sphericity, margin, lobulation, spiculation, texture), nodule diameter, and anatomical location.
Generation: A fixed template ( $T$ ) converts these attributes into a sentence (e.g., "A lung nodule with a diameter of $d$ mm, located in [location], showing [attributes]...").
Ablation: The authors tested seven input regimes across five frontier LLMs (Gemini-3.1-Flash-Lite, Mistral-Large, Qwen-2.5-72B, Claude-3.5-Haiku, GPT-4o-mini) to determine which input channel carries the diagnostic signal.

2. Calibration as the Safety Mechanism
LLMs output verbalized confidences that are often uncalibrated. ConfTriage transforms these into a reliable probability:

Logit Transformation: Verbalized confidence ( $c_v$ ) is converted to logit space: $\ell_v = \log(c_v / (1 - c_v))$ .
Platt Scaling: A Platt scaling map ( $g_{Platt}$ ) is fitted on a held-out calibration fold to convert logits into a calibrated probability $\hat{p} = \sigma(A\ell_v + B)$ .
Routing: A threshold $\tau$ is selected. If $|\hat{p} - 0.5| \geq \tau$ , the LLM prediction is accepted. If $|\hat{p} - 0.5| < \tau$ , the case is deferred.

3. Selective Specialist Backstop
Cases where the LLM is uncertain are routed to a specialist DL classifier, Certain-Net (an uncertainty-aware ensemble of CNNs trained on 2D views of the CT volume). This creates a hybrid system where the LLM handles high-confidence cases and the DL model handles the remainder.

Theoretical Contributions

The paper provides two formal guarantees supporting the framework:

Theorem 1 (Finite-Sample Combined-Error Bound): Establishes that the population error of the combined system is bounded by the sum of empirical retained and deferred errors plus a Hoeffding correction term. This provides a "certificate" for the combined error rate at any given threshold.
Theorem 2 (Oracle Inequality): Proves that the excess risk of ConfTriage over a Bayes-optimal deferral classifier is controlled by the L1 calibration error of the LLM's probability. This formally links the act of calibration to triage optimality.

Key Results

1. Linguistic Sufficiency vs. Image Statistics

Language Dominance: Across all five LLMs, natural-language descriptions of structured attributes yielded the highest performance (e.g., Gemini-3.1-Flash-Lite achieved AUC 0.907 with text-only input).
Image Statistics Failure: Inputting low-level image statistics (histograms, GLCM features) resulted in near-random performance (AUC range 0.44–0.58), with several models defaulting to a single class.
Conclusion: The diagnostic signal in LIDC-IDRI is effectively captured by structured attributes rendered in language; low-level image statistics are diagnostically vacuous for generalist LLMs.

2. Performance Metrics

ConfTriage Performance: The hybrid framework achieved an F1 score of 88.22% and an AUC of 0.92.
Efficiency: At the optimal operating threshold ( $\tau = 0.28$ ), the LLM resolved 76.5% of cases using zero-shot inference alone. Only the remaining 23.5% (uncertain cases) were referred to the specialist DL backstop.
Calibration Impact: Platt scaling reduced the Expected Calibration Error (ECE) from 0.085 to 0.042 and increased LLM decision coverage from 65.45% to 76.54% while improving the F1 score.

3. Comparison to Human Readers
Using a Leave-One-Out (LOO) consensus protocol, ConfTriage achieved an AUC of 0.952, which is comparable to or slightly higher than the operating points of individual radiologists in the dataset (AUC range 0.874–0.922) and outperformed the standalone specialist backstop (Certain-Net, AUC 0.949).

Significance and Claims

The paper claims that ConfTriage offers a practical pathway for integrating generalist LLMs into medical decision-support systems without requiring task-specific image training. Its significance lies in:

Methodological Shift: Demonstrating that a generalist LLM, when fed a faithful language rendering of standard radiological attributes, can match the discriminative performance of specialized DL models for a significant portion of cases.
Operational Safety: Moving beyond raw accuracy metrics to provide a calibrated triage policy with formal error bounds (Theorem 1) and a theoretical link between calibration and optimality (Theorem 2).
Vendor Agnosticism: Showing that this approach works across multiple closed and open-source LLM vendors, reducing the risk of vendor lock-in.
Cost Efficiency: Enabling the resolution of ~76% of cases via zero-shot LLM inference, reserving expensive, task-specific image-trained models only for the most uncertain cases.

The authors explicitly note limitations, including the reliance on radiologist assessments (rather than pathology-confirmed ground truth) in LIDC-IDRI and the lack of prospective clinical validation, positioning the work as a methodological framework validated on public benchmarks.

Problem Statement

Methodology: ConfTriage Framework

Theoretical Contributions

Key Results

Significance and Claims

More like this