🤖 AI

JustLLMGRPO: Radiographic Control for Chest X-Ray Generation

The paper introduces JustLLMGRPO, a method that optimizes text prompts for chest X-ray generation using Group Relative Policy Optimization on a large language model while keeping the image generator frozen, achieving state-of-the-art image quality and alignment by refining radiographic descriptions to emphasize visible findings and remove non-renderable content.

Original authors: Pengxiang Cai, Xiaohan Li, Anglin Liu, Qingyuan Zeng, Zexun Li, Jintai Chen

Published 2026-08-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pengxiang Cai, Xiaohan Li, Anglin Liu, Qingyuan Zeng, Zexun Li, Jintai Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-talented artist who has spent years learning to paint perfect pictures of the human chest, specifically X-rays. This artist knows exactly how bones look, how shadows fall, and how to make a picture that looks real enough to fool a doctor. But there's a catch: this artist is a bit literal. If you tell them, "Draw a chest X-ray, but also mention that the patient's lungs looked the same last week, and maybe the fluid is probably there, but we aren't 100% sure," the artist gets confused. They might try to paint "last week" or "maybe," resulting in a messy, blurry image that doesn't match what you actually wanted to see.

This is the world of Text-Conditioned Chest X-Ray Generation. It's a branch of artificial intelligence where computers try to create medical images based on written descriptions. The big challenge has always been: how do we get the computer to ignore the "fluff" in a doctor's report (like comparisons to old scans or uncertain guesses) and focus only on the things that can actually be drawn in a single picture? For a long time, scientists thought the only way to fix bad images was to retrain the artist (the image generator) to be smarter. But what if the artist is already perfect, and the problem is just that we're giving them bad instructions?

The "JustLLMGRPO" Solution: Fixing the Prompt, Not the Painter

This paper introduces a clever new idea called JustLLMGRPO. Instead of trying to retrain the image generator (the artist), the researchers decided to hire a "Prompt Editor" (a Large Language Model, or LLM) to rewrite the instructions before they reach the artist.

Think of it like this: You have a strict, frozen-in-time sculptor who can only carve stone if you give them a very specific list of shapes. If you hand them a long, rambling story about a mountain that might have a cave, the sculptor gets confused and carves a weird, lopsided rock. The researchers found that if they let a smart AI editor rewrite that story into a crisp, clear list of shapes ("Carve a mountain. Carve a cave on the left side."), the sculptor's work becomes amazing, even though the sculptor hasn't changed at all.

The Big Discovery
The team tested this on a generator called Sana, which had already been trained to make chest X-rays. They froze the generator so it couldn't learn anything new.

The Problem: When they fed the generator raw reports from doctors, the resulting images were often blurry or wrong. The "RadDINO-FID" score (a measure of how realistic the image looks compared to real X-rays) was 54.225.
The First Fix (The Editor): They asked an unmodified AI (Qwen3-4B) to simply rewrite the doctor's reports into shorter, clearer instructions, removing things like "last week," "maybe," and "unchanged." Just by doing this, the image quality jumped! The score dropped to 27.572, nearly cutting the error in half.
The Catch: However, this simple rewriting had a side effect. The new instructions were so different from the original reports that the AI lost track of the original meaning. The "alignment" score (how well the image matched the original doctor's intent) dropped from 0.695 to 0.609. It was like the editor changed the story so much that it wasn't the same story anymore.

The Final Polish: JustLLMGRPO
To fix this, the researchers introduced JustLLMGRPO. They used a special training method called Group Relative Policy Optimization (GRPO).

How it works: Imagine the Prompt Editor tries to write five different versions of the instructions. The frozen artist draws all five images. A "Judge" (a radiology-aware scoring system) looks at the five images and says, "This one looks best, but it still missed the point. This one is okay but looks weird. This one is perfect!"
The system then tells the Prompt Editor: "You did a good job on that one, but you need to keep the original meaning while making it clear."
The result? The Prompt Editor learned to write instructions that were short and clear but still perfectly matched the original doctor's report.

The Results
With JustLLMGRPO, the team achieved the best of both worlds:

Image Quality: The RadDINO-FID score dropped even further to 26.780 (a 50.6% improvement over just using the raw prompts).
Accuracy: The alignment with the original report stayed high at 0.696 (almost exactly the same as the original, fixing the drop seen in the simple rewrite).
Utility: The images were so good that if you trained a separate AI to diagnose diseases on them, it performed better than on images from other top methods.

What This Means
The paper explicitly argues against the idea that we must constantly retrain the image generator to get better results. They showed that a huge amount of potential was hidden in how we ask for the image. By keeping the generator frozen and only optimizing the "Prompt Policy" (the instructions), they got state-of-the-art results.

They also ruled out the idea that simply making the instructions shorter is enough; without the specific GRPO training, the meaning gets lost. And they showed that trying to retrain the generator itself (the "Sana-GRPO" control) actually made the images worse in terms of realism, even if the alignment score looked higher.

In short, the researchers found that sometimes, the best way to get a better picture isn't to hire a better artist, but to learn how to give the artist better instructions. The code for this new method is now public, inviting others to try this "prompt-first" approach on their own AI art projects.

Technical Summary: JustLLMGRPO: Radiographic Control for Chest X-Ray Generation

Problem Statement

Text-conditioned Chest X-Ray (CXR) generation aims to synthesize realistic radiographs that faithfully depict specified clinical findings. Existing approaches have predominantly focused on improving the image generator itself through domain adaptation, specialized conditioning, and reinforcement learning. A critical assumption in this trajectory is that once a generator is adapted to the CXR domain, the input prompts (typically derived from radiology reports) are fixed.

However, radiology reports are clinical communication documents, not literal image descriptions. They contain non-renderable content such as temporal comparisons ("unchanged," "stable"), uncertainty ("likely," "possible"), and longitudinal discourse. When these reports are used directly as prompts for a static, adapted generator, this non-renderable content dilutes the signal from visible findings, leading to suboptimal image quality. This paper posits that a significant optimization dimension remains underexplored: the formulation of the prompt itself, independent of generator parameter updates.

Methodology: JustLLMGRPO

The authors propose JustLLMGRPO, a framework that optimizes the prompt policy using a Large Language Model (LLM) while keeping the image generator frozen. The method leverages Group Relative Policy Optimization (GRPO) to refine prompts based on radiology-aware image feedback.

Core Components

Frozen Generator: The system uses a CXR-adapted Sana generator (frozen throughout the process). This isolates the optimization to the prompt formulation.
LLM Prompt Policy: A Qwen3-4B model acts as the prompt planner. Given a source prompt (e.g., from LLaVA-Rad), the LLM generates candidate prompts that are concise, visually focused, and preserve the original clinical meaning without introducing new findings.
Matched-Seed Rendering: To ensure fair comparison, the system renders both the source prompt and the candidate prompts using the same deterministic seed ( $z_c$ ). This controls for stochastic variation in the diffusion process.
Radiology-Aware Reward: Each generated image is scored using a composite metric $Q(I, c, y)$ $Q (I, c, y)$ :
- BioViL-T Alignment ( $b$ ): Measures alignment with the source prompt (preserving intent).
- CheXpert Label Consistency ( $\ell$ ): Ensures the image matches the ground-truth pathology labels.
- RadDINO Fidelity ( $d$ ): Measures visual realism and feature density.
- The reward is a weighted sum: $0.45b + 0.10\ell + 0.45d$ .
GRPO Optimization: Instead of using a separate critic, the system samples a group of $K=5$ candidate prompts for each source. The relative improvement of each candidate over the matched-seed source rendering is calculated. Standard GRPO updates the LLM prompt policy to maximize these relative advantages, effectively learning to suppress non-renderable report language while emphasizing visible radiographic findings.

Key Contributions

Identification of an Underexplored Dimension: The authors demonstrate that for a frozen, CXR-adapted generator, a single-pass LLM reformulation of prompts can nearly halve the RadDINO-FID (from 54.225 to 27.572) without updating the generator.
Analysis of Prompt Reformulation: The study reveals that unconstrained LLM rewriting improves visual fidelity but degrades alignment with the source prompt (dropping BioViL-T alignment from 0.695 to 0.609). This trade-off highlights the need for constrained optimization.
JustLLMGRPO Framework: The introduction of a GRPO-based prompt policy that successfully balances visual focus with source-prompt alignment. It achieves state-of-the-art performance on the CheXGenBench benchmark while keeping the generator frozen.

Experimental Results

Evaluated on the CheXGenBench development set, JustLLMGRPO demonstrates superior performance across multiple metrics compared to existing generators (including Stable Diffusion variants, RadEdit, and LLM-CXR):

Fidelity: Achieves a RadDINO-FID of 26.780, a 50.6% improvement over direct prompting with the same Sana generator (54.225).
Alignment: Maintains BioViL-T alignment at 0.696, effectively recovering the alignment loss seen in unconstrained rewriting (0.609) and matching the original Sana baseline (0.695).
Distribution Coverage: Achieves the best PRDC coverage (0.588) among synthetic generators.
Downstream Utility: Trained on 20,000 synthetic images, a ResNet-50 classifier achieves a macro AUC of 0.746 on held-out real CXRs, outperforming other synthetic baselines and approaching the performance of models trained on real data (0.731).
Condition-Level Fidelity: JustLLMGRPO achieves the best conditional RadDINO-FID for five categories (including Cardiomegaly, Consolidation, and Pneumonia) and ranks second for five others.

Significance and Claims

The paper argues that prompt-policy optimization is a distinct and powerful axis for text-conditioned CXR generation, separate from generator adaptation. The results suggest that:

Latent Performance: Substantial performance gains remain latent in how radiographic information is expressed to an adapted generator.
Efficiency: High-quality generation can be achieved without the computational cost of retraining or fine-tuning the image generator, provided the prompt is optimized to filter out non-renderable clinical discourse.
Alignment vs. Fidelity: While unconstrained rewriting improves fidelity, it sacrifices semantic alignment. JustLLMGRPO resolves this trade-off by using group-relative feedback to learn a prompt policy that is both visually focused and semantically faithful to the source.

The authors conclude that target-domain adaptation does not exhaust the optimization space and that optimizing the expression of radiographic information via LLM policies is a viable path to state-of-the-art medical image generation.