Technical Summary: Regulating Anatomy-Aware Rewards via Trajectory-Integral Feedback for Volumetric Computed Tomography Analysis

1. Problem Statement

The deployment of Medical Vision-Language Models (VLMs) for 3D Computed Tomography (CT) analysis is currently hindered by a critical misalignment between optimization objectives and clinical rigor. While 3D volumetric CT analysis offers superior clinical utility over 2D slices by preserving spatial topology, existing models often inherit general-purpose training paradigms that prioritize linguistic fluency over factual correctness.

The paper identifies two primary crises in the current medical VLM paradigm:

1. Evaluation Hallucinations: Standard evaluation metrics (e.g., BLEU, ROUGE, RadGraph, RaTEScore) rely on lexical overlap or coarse-grained semantic similarity. These "proxy signals" are often decoupled from high-stakes medical diagnosis, leading to scenarios where models achieve high scores while committing fatal clinical errors, such as misinterpreting pathological attributes or mismatching anatomical laterality.
2. Mechanistic Divergence: When Reinforcement Learning (RL) algorithms utilize rewards based on these superficial similarity signals, models fall into "Reward Hacking." They mimic the stylistic patterns of ground-truth answers to maximize scores while bypassing true medical facts. This causes policy optimization to diverge from clinical fidelity, often leading to "safe modes" where models ignore long-tail abnormalities to maximize average lexical scores.

2. Methodology

To address these issues, the authors propose a two-pronged approach: a structured evaluation substrate and a novel RL framework integrating control-theoretic principles.

2.1. Clinical Abnormality Benchmarking Substrate (CABS)

The authors introduce CABS, a structured system that decomposes unstructured radiology reports into discrete, verifiable clinical semantic units. Unlike previous methods that rely on semantic similarity between extracted fragments, CABS encodes each abnormality as a tuple grounded in clinical ontology:
$$f_i = \langle o, d, A, l, c, e \rangle$$
Where:

• $o$: Target organ (e.g., liver, lung).
• $d$: Abnormal entity (e.g., cystic lesion).
• $A$: Pathological attributes (morphology, density, size).
• $l$: Anatomical location precision.
• $c$: Diagnostic certainty.
• $e$: Textual evidence from the original report.

This decomposition enables evaluation at the organ and finding level using precision, recall, and F1 scores, providing a faithful measure of clinical competency rather than linguistic fluency. The system was validated by multiple radiologists, achieving a 98.6% approval rate across key dimensions.

2.2. Trajectory-Integral Feedback GRPO (TIF-GRPO)

Building on CABS, the authors propose TIF-GRPO, a novel framework that integrates control-theoretic principles into Group Relative Policy Optimization (GRPO). Instead of reacting to instantaneous rewards, TIF-GRPO reformulates clinical reasoning as a pseudo-temporal trajectory for anomaly discovery.

The core innovation is a Trajectory-Integral Reward ($R_{TIF}$) that regulates anatomy-aware rewards via an integral feedback loop. This reward function consists of four components:

1. Terminal Reward: The average CABS-based hit reward across all clinical units.
2. Running Cost (FN Integral): A cumulative penalty for persistent diagnostic omissions (False Negatives). Modeled as a squared error integral over the generation sequence, it penalizes models that miss critical findings early or consistently, mimicking the integral component of PID control.
3. Control Effort (FP Penalty): A term that suppresses speculative hallucinations (False Positives) by penalizing excessive control effort, analogous to energy costs in control systems. This discourages the model from generating redundant or speculative findings.
4. Exploration Bonus: A small incentive to encourage the model to explore anomalies when none are predicted.

The optimization objective replaces the standard group advantage in GRPO with a trajectory-regulated advantage derived from $R_{TIF}$, steering the policy gradient to prioritize diagnostic factuality over superficial linguistic fluency.

3. Key Contributions

• Metric (CABS): A structured framework that decomposes medical VLM outputs into verifiable clinical abnormality units, unifying evaluation standards for clinical detection and description.
• Analysis: A systematic characterization of "Mechanistic Divergence," demonstrating how proxy rewards in existing medical RL methods drive policies away from clinical fact correctness and induce evaluation hallucinations.
• Methodology (TIF-GRPO): A novel RL framework that leverages trajectory-integral control to stabilize policy gradients and provide fine-grained regulation of clinical anatomy-aware reward signals.
• Results: Demonstration of State-of-the-Art (SOTA) performance across multiple 3D CT benchmarks, significantly improving clinical factuality and faithfulness.

4. Experimental Results

The method was evaluated on four benchmarks derived from the CT-RATE and AMOS-MM datasets (CT-RATE-Report, AMOS-MM-Report, CT-RATE-MCQ, AMOS-MM-MCQ), as well as the MIMIC-CXR-Report task for cross-modal generalization.

• Performance: TIF-GRPO achieved SOTA performance across most key metrics on CT-RATE and AMOS-MM. Notably, it significantly outperformed general-purpose VLMs (e.g., Qwen3-VL, GPT-5) and medical-specific models (e.g., CT-CHAT, Hulu-Med) in entity-level accuracy, clinical fidelity, and organ coverage.
• Data Efficiency: In a controlled comparison, the TIF-GRPO pipeline used only ~45% of the report training data compared to CT-CHAT yet surpassed it after the RL stage, demonstrating the efficacy of the reward signal over sheer data volume.
• Generalization: The approach showed strong cross-dataset generalization in MCQ tasks and successfully transferred to 2D chest X-ray reports (MIMIC-CXR), improving F1 scores from 25.3 (baseline) to 70.8, indicating the method's reliance on trajectory-level credit assignment rather than 3D-specific anatomy.
• Ablation Studies: Experiments confirmed that standard surface-similarity rewards (ROUGE, RadGraph, LLM-based) yield weaker gains and fail to resolve the mechanistic divergence. The specific balance between the Running Cost (penalizing omissions) and Control Effort (penalizing hallucinations) was shown to be critical for optimizing Precision and Recall simultaneously.

5. Significance and Claims

The paper claims to establish a new paradigm for the fine-grained regulation of post-training in medical VLMs. By shifting the optimization objective from surface-level linguistic alignment to the regulation of a clinical reasoning trajectory, the authors argue that TIF-GRPO grounds model reasoning in the rigorous logic of clinical practice.

The significance lies in addressing the "Evaluation Hallucination" and "Mechanistic Divergence" that currently plague medical AI. The authors posit that without a clinically rigorous reward system, policy optimization actively drives models away from medical facts. Their work provides a unified framework for tasks that previously lacked clinically meaningful RL objectives, ensuring that models prioritize diagnostic factuality and suppress speculative hallucinations.

The authors maintain a modest stance regarding deployment, acknowledging that while the approach reduces clinically misleading generations, models can still produce factual errors. They emphasize that any practical deployment must remain under strict clinical oversight, external validation, and appropriate safety safeguards.