🤖 AI

World Feedback for Clinical Agents: Diagnosing RL in FHIR Environments

This paper identifies that applying Reinforcement Learning to clinical agents in FHIR environments is currently hindered by silent-finish ceilings, capability gaps, and undiscoverable format knowledge, proposing a hybrid approach where Supervised Fine-Tuning injects necessary clinical codes while RL optimizes decision-making logic.

Original authors: Ananya Mantravadi, Harshit Rajgarhia, Prasanna Desikan, Abhishek Mukherji

Published 2026-07-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ananya Mantravadi, Harshit Rajgarhia, Prasanna Desikan, Abhishek Mukherji

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching a Robot Doctor's Assistant

Imagine you are trying to teach a robot assistant how to follow a hospital's strict rulebook. The rules are things like: "If a patient's blood sugar is too high, order a specific test. If it's normal, do nothing."

The researchers wanted to see if they could teach this robot using Reinforcement Learning (RL). In RL, the robot tries things, gets a "score" from a referee (a computer program that checks the rules), and learns from its mistakes. The goal was to see if the robot could learn to follow these medical protocols just by playing the game over and over, without a human teacher showing it the answers every time.

The Problem: The Game Was Rigged

When the researchers tried this on the old version of their test (MedAgentBench v1/v2), the robot failed miserably. But it wasn't because the robot was stupid; it was because the game was broken.

The "Do Nothing" Trap: In the old test, about 42% of the time, the correct answer was simply to "do nothing." Because the robot was rewarded for getting a high score, it quickly realized the easiest way to win was to just sit there and do nothing. It learned to be lazy because "doing nothing" was the dominant strategy.
The Fix: The researchers built a new, fairer test (MedAgentBench v3). They balanced the game so that "doing nothing" wasn't the easy way out. Now, the robot actually had to work to get a good score.

The Real Discovery: Two Walls the Robot Couldn't Climb

Even with the fair new test, the robot (a model called Qwen3-8B) still struggled. The researchers found two specific "walls" that stopped it from learning perfectly using only trial-and-error.

1. The "Blank Slate" Wall (Capability Ceiling)

Imagine asking a student to solve a math problem they have never seen before, and they don't even know what a "plus" sign is. No matter how many times you let them guess, they can't learn the concept because they lack the basic foundation.

In the paper: For about half of the tasks, the robot started with 0% ability. It didn't know how to look up a patient's ID or format a specific medical code. Because it failed every single time, it got no useful feedback. It was like trying to teach someone to drive a car when they don't even know how to turn the key.

2. The "Hidden Code" Wall (Format-Knowledge Barrier)

Imagine a video game where you have to type a secret password to open a door. If you type the wrong password, the game says "Wrong." But the game never tells you what the right password is. You could try 1,000 random passwords and get the same "Wrong" message every time. You can't learn the right code just by guessing.

In the paper: Some tasks required exact, specific medical codes (like a specific ID number for a drug). These codes aren't something you can figure out by looking at the patient's chart; they are just facts you have to memorize. The robot couldn't "discover" these codes through trial and error because the reward signal was flat (it just kept getting "Wrong" no matter what it guessed).

The Solution: A Two-Step Training Plan

The researchers compared three ways of training the robot:

Pure RL (Trial and Error): The robot guessed on its own. Score: 18.2%
Supervised Learning (SFT): A human (or a rule-based computer) showed the robot the correct answers first, like a teacher showing a student the solution key. Score: 34.1%
The Gap: Pure RL was 15.9% worse than the teacher-led approach.

Why the gap exists:

SFT (The Teacher) was great at injecting the "Hidden Codes" and the correct "Format" (how to write the answer). It gave the robot the facts it needed.
RL (The Explorer) was good at learning the logic (when to act and when to stop), but it couldn't invent the facts or codes on its own.

The Verdict: You Need Both

The paper concludes that you can't just rely on the robot to figure everything out by itself in a clinical setting.

Step 1: You must use a "teacher" (Supervised Learning) to inject the necessary facts, codes, and formats into the robot's brain.
Step 2: Then, you let the robot use "Reinforcement Learning" to practice and refine its decision-making skills (knowing when to use those codes).

Summary Analogy

Think of training a new nurse:

Pure RL is like throwing a new nurse into the ER and saying, "Figure it out by trial and error." They will likely make dangerous mistakes or learn to do nothing because it's safer.
SFT is like giving them a textbook and a checklist. They learn the drug names and the forms perfectly.
The Paper's Conclusion: The best approach is to give them the textbook first (SFT) so they know the facts, and then let them practice in the ER (RL) so they learn to apply those facts correctly in real-time situations. Without the textbook, the practice is useless.

Technical Summary: World Feedback for Clinical Agents: Diagnosing RL in FHIR Environments

1. Problem Statement

The paper addresses the application of Reinforcement Learning (RL) from world feedback to clinical protocol-execution tasks. These tasks involve retrieving laboratory values, applying decision thresholds, and placing correctly structured FHIR (Fast Healthcare Interoperability Resources) orders. Unlike general clinical reasoning, these are administrative workflows where decision logic is known and verifiable.

The core challenge is establishing a sound feedback channel for RL. While Supervised Fine-Tuning (SFT) requires manual annotation of every episode, RL from world feedback theoretically allows a single verifier (encoding clinical decision logic) to grade unlimited rollouts automatically. However, the authors identify that applying this paradigm to clinical environments is non-trivial due to two primary structural barriers:

Benchmark Validity: Existing benchmarks (MedAgentBench v1/v2) contained a "silent-finish ceiling" where 41.7% of tasks could be passed by doing nothing, making inaction the dominant RL strategy.
Capability and Knowledge Gaps: Pure RL struggles with tasks requiring exact clinical codes (e.g., SNOMED, NDC) that cannot be discovered through exploration (flat reward landscapes) and tasks where the base model has zero initial capability (zero gradient).

2. Methodology

Benchmark Reconstruction: MedAgentBench-v3 (MAB-v3)

To address the "silent-finish" issue, the authors constructed MAB-v3, a corrected benchmark containing 508 tasks across 20 clinical task types.

Corrections: The authors fixed branch imbalances (enforcing a 1:1 ratio of action vs. no-action instances), patched undocumented format requirements, and fixed a "wall-clock" bug in the verifier.
Result: The silent-finish ceiling was reduced from 41.7% to 8.9%, ensuring that inaction is no longer a dominant strategy.
Environment: The environment uses a deterministic, offline HAPI FHIR server snapshot with ~100 anonymized patients. It includes a rule-based verifier (1,340 lines of code) that grades agents against clinical protocols without human intervention.

Training and Evaluation Protocols

The study evaluated Qwen3-8B (a model substantially smaller than frontier models) under three conditions:

Base Model: Zero-shot performance.
Supervised Fine-Tuning (SFT): Trained on 401 programmatic demos generated by a rule-based agent. These demos provided exact clinical codes and FHIR payload structures in the <tool call> format.
Pure RL (GRPO): Trained directly from the base model using Group Relative Policy Optimization (GRPO) on the world feedback signal. The reward function ( $r$ ) combined terminal rewards, partial credit for correct resource types, and penalties for spurious actions or skipping tools.

Key Design Principles:

Deterministic Episodes: Eliminated environment instability as a confound.
Reward Shaping: Implemented partial credit and penalties for spurious actions to create a gradient for conditional behavior (action vs. no-action).
Format Consistency: Used the standard <tool call> interface rather than the original benchmark's free-form HTTP strings to test native tool-use capabilities.

3. Key Contributions

MAB-v3 and Environment: A corrected 508-task benchmark and a self-contained world feedback environment that eliminates the silent-finish shortcut and provides auditable, deterministic feedback.
Structured Diagnosis of RL Limitations: The paper identifies and quantifies two structural barriers preventing pure RL from matching SFT performance:
- Capability Ceiling: 10 of 20 task types showed 0% base performance, yielding zero gradient for RL.
- Format-Knowledge Barrier: 3 task types required exact clinical codes undiscernible by exploration, resulting in a flat reward landscape.
Task Taxonomy: A framework categorizing tasks into Decision, Lookup, and Format-Knowledge types. This taxonomy predicts RL learnability:
- Decision tasks: High RL learnability (reward varies with the decision).
- Lookup tasks: Weak RL signal (answers are retrieved, not derived).
- Format-Knowledge tasks: No RL signal (exact codes are discrete and uninferrable).

4. Results

Performance Metrics (Pass@1 on Held-Out Test Set)

Base Qwen3-8B: 16.6%
Pure RL (GRPO): 18.2%
SFT: 34.1%
Gap: Pure RL lagged SFT by 15.9 percentage points.

Analysis of the Gap

The 15.9 pp gap is attributed entirely to the structural barriers identified in the taxonomy, not to the RL algorithm or reward design:

Dead Zones: 10 task types (mostly v1 lookup and format-knowledge) had 0% pass rates for both the base model and RL, as the model lacked the initial capability to generate a gradient.
Format-Knowledge Failure: On tasks requiring exact codes (e.g., Naloxone SNOMED codes), RL performance actually decreased (from 20.8% to 16.7%). The model overwrote occasional "lucky" pre-training recalls with incorrect codes that received uniform negative rewards, providing no gradient to the correct identifier.
Decision Gains: RL showed modest improvements on decision tasks (e.g., +4.2 pp on CT follow-up), confirming it can learn conditional logic when a gradient exists.

Frontier Model Comparison

Frontier models (e.g., GPT-5.5, Gemini 3.1 Pro) showed high performance on decision tasks but struggled significantly with format-knowledge tasks (0% pass on specific NDC/SNOMED requirements), reinforcing that exact code knowledge is not reliably emergent even in large models without specific injection.

5. Significance and Claims

The paper claims that pure RL from world feedback is insufficient for clinical protocol execution when the base model lacks specific format knowledge or initial capability. The 15.9 pp performance gap is not a failure of RL as a paradigm but a diagnosis of its structural limits in this domain.

Key Prescriptions:

SFT + RL Combination: The authors propose a hybrid approach where SFT is used to "inject" format knowledge and exact clinical codes, followed by RL to refine conditional reasoning and decision logic using world feedback.
Benchmark Hygiene: The study emphasizes that clean world feedback requires a clean benchmark. The "silent-finish" shortcut in MAB-v2 rendered RL ineffective regardless of the algorithm used.
Scope: The findings are specific to protocol-execution tasks (administrative workflows) and do not generalize to open-ended clinical reasoning or differential diagnosis.

The paper concludes that while world feedback offers a scalable alternative to per-episode annotation, it cannot replace the need for supervised injection of discrete, non-inferable knowledge (codes and formats) in clinical agents.