Calibrated Abstention and Clinical Deferral in Small Language Models via RLVR and GRPO

Technical Summary: Calibrated Abstention and Clinical Deferral in Small Language Models via RLVR and GRPO

Problem Statement: The "Dangerous Confidence" Trap
The paper addresses a critical safety failure in deploying Large Language Models (LLMs) and Small Language Models (SLMs) within clinical settings: "epistemic overconfidence." While base autoregressive models excel at linguistic fluency, they are trained to predict the next token probabilistically rather than to verify factual truth. In high-stakes medical domains, this manifests as "dangerous confidence," where models generate factually incorrect, highly plausible medical assertions with absolute certainty. The paper argues that standard evaluation metrics, such as raw accuracy on benchmarks like MedQA-USMLE, fail to capture this risk profile. A model that confidently guesses on ambiguous data poses a greater threat to patient safety than one that recognizes its uncertainty and abstains. Furthermore, deploying massive models is often impractical due to computational latency, infrastructure costs, and data privacy regulations (e.g., HIPAA, GDPR), necessitating a shift toward SLMs (specifically the 2.6B parameter Gemma 2B architecture) which, while efficient, suffer from limited parametric capacity and increased susceptibility to hallucination.

Methodology: RLVR and GRPO for Calibrated Abstention
To embed "Calibrated Abstention" into an SLM, the authors propose a novel alignment pipeline combining Reinforcement Learning from Verifiable Rewards (RLVR) and Group Relative Policy Optimization (GRPO).

• RLVR Architecture: Unlike Reinforcement Learning from Human Feedback (RLHF), which relies on subjective human preferences and risks optimizing for sycophancy or plausible hallucinations, this approach utilizes deterministic, programmatic verification. The reward function is anchored in objective clinical truths, specifically the Sequential Organ Failure Assessment (SOFA) scoring system.
• The Clinical Reasoning Protocol (SOFA & Cactus Signal): The model is forced to adhere to a strict "SOFA-First" reasoning protocol. Before generating a diagnosis, it must extract and verify six physiological parameters (Respiratory, Coagulation, Liver, Cardiovascular, CNS, Renal) into a structured matrix. If critical data is missing, the model must mark it as "N/P" (Not Provided). Following this reasoning, the model must emit a "Cactus Signal": either <|local_ok|> to proceed with a local answer or <|escalate|> to defer to a human expert.
• Reward Function Design: The aggregate reward is composed of four vectors:
  1. Correctness (50%): Exact match of the final diagnosis against ground truth.
  2. Clinical Logic (20%): Verification of the SOFA matrix against provided data, penalizing hallucinated metrics.
  3. Safety & Abstention (20%): A bonus for correctly identifying missing data and triggering the <|escalate|> signal; a penalty for "false confidence" (triggering <|local_ok|> on incomplete data).
  4. Format (10%): Validation of strict XML structural compliance.
• GRPO Implementation: To overcome the memory constraints of training SLMs on consumer-grade hardware (Dual NVIDIA T4 GPUs with 32GB VRAM), the authors employ GRPO. This value-function-free algorithm eliminates the need for a separate "critic" model. Instead, it samples a group of completions (G=2 initially, then G=4) for a single prompt and computes the advantage relative to the group's mean reward. This allows for stable training on edge devices where standard PPO would fail due to VRAM limitations.

Key Contributions

1. Operationalization of the "Cactus Protocol": The introduction of a deterministic routing mechanism that forces SLMs to explicitly signal epistemic confidence or uncertainty, converting abstract uncertainty into actionable routing commands.
2. Integration of SOFA as a Verifiable Reward: The successful embedding of a rigid clinical scoring matrix (SOFA) as a programmatic constraint within the GRPO framework, effectively penalizing hallucinations and enforcing logical coherence.
3. Empirical Quantification of the "Alignment Tax": The study provides data on the trade-off between raw accuracy and safety compliance, demonstrating that a reduction in benchmark performance is a necessary and justifiable sacrifice for patient safety.

Results & Analysis
The study evaluated the aligned model ("Gemma-Sync") against the unaligned base model on a strictly unseen subset of the MedQA-USMLE dataset (n=200).

• Alignment Tax: The aligned model experienced an 8.5% absolute decrease in raw accuracy (dropping from 44.0% to 35.5%). The authors attribute this to "Catastrophic Forgetting," where the model's limited attention heads were reallocated from probabilistic pattern matching to deterministic constraint satisfaction (SOFA verification).
• Safety Gain: Conversely, the Abstention Rate (frequency of <|escalate|> signals) increased by 129%, rising from 3.5% to 8.0%. This indicates the model is 2.3 times more likely to recognize its epistemic boundaries and defer to human experts rather than hallucinating.
• Efficiency and Latency: While the model achieved safety goals, inference latency increased from 138.9s to 153.3s. The authors identify a "Stopping Token Dilemma" where the model, over-optimized for generating extensive SOFA matrices, engages in "structured rambling" and fails to emit the End-of-Sequence token promptly, leading to token exhaustion.
• GRPO Resilience: The study demonstrates that RLVR can achieve robust convergence even with extremely small group sizes (G=2 to G=4) on constrained hardware, provided the reward signal is deterministic and verifiable.

Significance and Claims
The paper concludes that in high-stakes triage, prioritizing safety through conservative deferral yields a more reliable clinical assistant than maximizing raw benchmark performance. The authors argue that "Calibrated Abstention" is the operationalization of the Hippocratic oath ("First, do no harm") within AI architecture. By teaching SLMs to say "I do not know" or to escalate when data is ambiguous, the system acts as a trustworthy cognitive filter rather than a dangerous oracle.

The research posits that the "Alignment Tax" is not a failure but an "insurance premium"—a necessary trade-off to eradicate systemic hallucination risks. Furthermore, it suggests that RLVR offers a superior alignment pathway for deterministic domains like medicine compared to RLHF, as it bypasses the subjectivity and sycophancy inherent in human preference models. The study establishes a scalable blueprint for developing trustworthy, edge-deployable medical AI that possesses intellectual humility and metacognitive calibration.