🤖 machine learning

RADIANT-PET: Reasoning-Augmented PET/CT Lesion Segmentation with Large Language Models and Reinforcement Learning

RADIANT-PET is a novel framework that enhances PET/CT lesion segmentation by combining a permissive voxel-level model with a reinforcement learning-optimized large language model to adjudicate candidates using structured textual descriptions and clinical reports, thereby significantly reducing false positives and aligning predictions with clinical interpretation.

Original authors: Jiasheng Wang, Tanun Jitwatcharakomol, Piyawadee Jongpradubgiat, Simeng Zhu

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiasheng Wang, Tanun Jitwatcharakomol, Piyawadee Jongpradubgiat, Simeng Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find specific, hidden treasures (cancer lesions) inside a giant, glowing, 3D map of a human body (a PET/CT scan). The problem is that the map is full of "noise"—natural body heat, inflammation from a cold, or even glitches in the camera—that looks exactly like the treasure you are hunting.

Traditional computer programs try to find these treasures by looking only at the picture. They are like a security guard who only checks the shape of an object. If it looks round and bright, they flag it as a treasure. But because the body is messy, this guard often mistakes a harmless bump for a treasure, leading to many false alarms.

RADIANT-PET is a new system that solves this by adding a "smart detective" to the team. Here is how it works, step-by-step:

1. The Over-enthusiastic Scout (High-Sensitivity Segmentation)

First, the system uses a specialized computer vision model (called HS-UNet) to scan the body. Think of this model as a scout who is told: "Find everything that might be a treasure. It's better to bring back 100 rocks thinking they are gold than to miss one real gold nugget."

The Result: This scout finds almost every real lesion, but it also brings back a huge pile of "false positives" (normal body parts that look suspicious).

2. The Translator (Structured Description)

The system then takes each suspicious spot the scout found and writes a detailed report card for it. Instead of just showing a blurry blob, it translates the image into a structured text description.

The Analogy: Imagine the scout points at a glowing spot and says, "It's a bright, round spot located right next to the left kidney, shaped like a flattened pancake, and it's glowing very brightly."
The system also adds context: "This spot is at the level of the 6th rib." It turns the visual data into a story.

3. The Smart Detective (The LLM)

This is where the magic happens. The system feeds these "report cards" into a Large Language Model (LLM)—a type of AI that is very good at reading and reasoning, similar to how a human doctor thinks.

The Job: The detective reads the report: "This spot is next to the kidney and looks like a flattened pancake." It then uses its medical knowledge to decide: "Ah, that sounds like normal kidney activity, not a tumor. Discard it." Or, "That sounds like a real tumor. Keep it."
The Bonus: If the patient has a written report from a human doctor (a radiology report), the detective can read that too! It can cross-reference the image description with the doctor's notes, like a detective checking a witness statement against the crime scene.

4. The Training Camp (Reinforcement Learning)

To make the detective even sharper, the authors trained it using a method called Reinforcement Learning (specifically GRPO).

The Analogy: Imagine the detective is a student taking a test. Every time it correctly identifies a real tumor or correctly dismisses a fake one, it gets a "gold star" (a reward). Every time it makes a mistake, it gets a "penalty."
Over thousands of practice tests, the detective learns exactly what patterns to look for and how to reason through tricky cases, eventually becoming much better at distinguishing real cancer from harmless body noise than the original image-only programs.

The Results

The paper tested this system on lymphoma patients.

Without the detective: The image-only programs were good at finding spots but bad at knowing which ones were real, leading to many errors.
With the detective (RADIANT-PET): The system successfully filtered out the "noise" (false alarms) while keeping the real treasures.
With the doctor's notes: When the detective was allowed to read the radiology reports, it performed the best of all, achieving the highest accuracy.

In short: RADIANT-PET doesn't just look at the picture; it reads the picture, describes it, and uses a smart AI detective to reason through the clues, ensuring that what is counted as a tumor is actually a tumor.

Technical Summary: RADIANT-PET

Problem Statement

Accurate lesion segmentation in PET/CT imaging is critical for oncology, particularly for lymphoma staging and prognostic modeling (e.g., post-CAR T-cell therapy). However, purely image-based segmentation models struggle to distinguish malignant uptake from physiologic tracer uptake, inflammation, infection, or imaging artifacts. Existing image-only approaches, including state-of-the-art nnUNet variants, have failed to achieve Dice Similarity Coefficient (DSC) values above 0.8, often resulting in significant false positives that distort metabolic tumor volume (MTV) measurements. The authors hypothesize that this limitation stems from the absence of explicit medical reasoning capabilities in current segmentation pipelines, which rely solely on visual patterns rather than integrating anatomical context and clinical knowledge.

Methodology

The paper proposes RADIANT-PET (Reasoning-Augmented Description–Inference Network for PET), a two-stage hybrid framework that decouples candidate generation from lesion-level adjudication.

1. High-Sensitivity Candidate Generation (HS-UNet)

The first stage prioritizes recall to ensure no true lesions are missed, accepting a higher rate of false positives.

Model Architecture: Based on nnUNet-v2, the winner of the AutoPET III FDG sub-track.
Training Objective: The loss function is modified to be deliberately permissive. It combines a Tversky loss (with $\alpha=1, \beta=0$ to reduce false positive penalties) and an Asymmetric Cross-Entropy (ACE) loss (emphasizing false negatives with $\lambda_{FN}=8.0$ ).
Inference: The decision threshold is lowered from 0.5 to 0.1 to maximize sensitivity.
Alternative: A fixed SUV thresholding approach (SUV $\ge$ 2.5) is also used as a clinical-standard baseline for candidate generation.
Post-Processing: A watershed splitting algorithm separates confluent uptake regions. Boundaries are validated via SUV-valley checks to ensure splits correspond to genuine metabolic separations rather than geometric artifacts.

2. Structured Description Generation

Each candidate region is converted into a structured textual description (JSON format) containing:

Quantitative Features: Voxel volume, 3D shape descriptors (elongation, sphericity, etc.), and PET intensity metrics ( $SUV_{max}$ , $SUV_{mean}$ ).
Local Anatomical Context: Overlap percentages with organs segmented by TotalSegmentator and relative position vectors.
Global Anatomical Context: Vertebral level assignment (e.g., T6) and whole-body localization relative to the image center.

3. LLM-Based Adjudication with Reinforcement Learning

A Large Language Model (LLM) evaluates the structured descriptions to classify candidates as "true lesion" or "physiologic uptake."

Base Model: A local 20-billion parameter model (gpt-oss-20b) is used to ensure privacy in clinical deployment.
Input: Structured JSON descriptions, optionally augmented with free-text radiology reports.
Optimization (GRPO): The model is fine-tuned using Group Relative Policy Optimization (GRPO), a reinforcement learning algorithm.
- Reward Function: A composite reward sums:
  1. Binary Correctness: +1 for correct class, -1 for mutually exclusive errors.
  2. Anatomical Localization: +2 for exact site match, +1.5 for match within an equivalence group.
  3. Reasoning Regularizer: -0.5 penalty if the reasoning trace fails to reference $SUV_{max}$ , ensuring the model utilizes imaging features.
Training Data: Models were trained on AutoPET and an internal OSU cohort (with and without radiology reports).

Key Contributions

Reasoning-Augmented Framework: A novel pipeline coupling conventional high-sensitivity segmentation with an LLM-based reasoning stage to adjudicate candidates, moving beyond pure pixel-level classification.
Reinforcement Learning for Medical Reasoning: Application of GRPO to optimize LLM lesion adjudication, directly targeting classification accuracy and anatomical plausibility.
Native Report Integration: Direct incorporation of radiology reports as context, allowing the LLM to leverage physician interpretations and clinical narratives during the reasoning process.

Results

The framework was evaluated on the AutoPET-Test and an internal OSU-Test cohort.

Candidate Generation: The modified HS-UNet achieved higher sensitivity (0.88) than the baseline nnUNet-v2 (0.63) on OSU-Test, with a significantly improved DSC (0.58 vs. 0.34) compared to simple thresholding.
LLM Performance: GRPO fine-tuning significantly improved the gpt-oss-20b model. Without reports, the RL-tuned model achieved an F1 of 0.68 (vs. 0.52 for the base model). With reports, it reached an F1 of 0.76, comparable to strong proprietary models (e.g., Gemini-Flash-3.0).
Segmentation Performance:
- Without Reports: RADIANT-PET (using RL-tuned LLM) outperformed the nnUNet-v2 baseline on both AutoPET-Test (DSC 0.82 vs. 0.78) and OSU-Test (DSC 0.77 vs. 0.72).
- With Reports: The best performance was achieved by combining HS-UNet candidates, RL-tuned LLM, and radiology reports, yielding a DSC of 0.84 and a Mean Absolute Percentage Error (MAPE) of 0.16 for MTV on OSU-Test. This represents a substantial improvement over the nnUNet-v2 baseline (DSC 0.72, MAPE 0.83).

Significance and Claims

The paper claims that RADIANT-PET demonstrates that LLM-based lesion-level reasoning serves as an effective post-segmentation adjudication layer. By adding a reasoning step that evaluates uptake intensity, morphology, and anatomical context, the framework successfully suppresses physiologic false positives while preserving sensitivity.

The authors emphasize that:

Explicit reasoning adds value beyond additional clinical context alone, as RL-tuned models matched or exceeded image-only baselines even without reports.
The integration of radiology reports provides a strong complementary signal, aligning image-derived findings with physician interpretation.
The use of a local LLM optimized via GRPO offers a privacy-preserving alternative to proprietary models, narrowing the performance gap in clinical deployment scenarios.

The study is limited to lymphoma FDG PET/CT and relies on upstream organ segmentation and handcrafted features, but it establishes a new paradigm for integrating medical reasoning into automated imaging workflows.