Prototype-Based Knowledge Guidance for Fine-Grained Structured Radiology Reporting

The paper proposes ProtoSR, a prototype-based framework that leverages an instruction-tuned LLM to extract visual prototypes from free-text radiology reports, thereby injecting unstructured knowledge to significantly improve fine-grained structured report generation and achieve state-of-the-art performance on the Rad-ReStruct benchmark.

The Gist

Imagine you are a radiologist looking at a chest X-ray. Your job is to write a report describing what you see.

The Problem: The "Free Text" vs. "Checklist" Dilemma
Traditionally, doctors write these reports in free text, like a story: "There is a patchy opacity in the lower left lung, suggesting pneumonia." This is great for detail, but it's messy for computers to read and hard to compare across thousands of patients.

Hospitals want structured reports, which are like filling out a strict checklist:

• Is there an opacity? [Yes/No]
• Where is it? [Upper Lobe / Lower Lobe / Diffuse]
• How bad is it? [Mild / Severe]

The problem is that while we have millions of "story" reports (free text), we have very few "checklist" reports (structured data) to teach computers how to fill them out correctly. It's like trying to teach a student to fill out a complex tax form, but you only have a few examples of the form filled out, while you have a library full of essays about taxes.

The Solution: ProtoSR (The "Smart Librarian")
The authors of this paper, ProtoSR, came up with a clever way to use those millions of messy "story" reports to help the computer fill out the "checklist" perfectly.

Think of their system as a Smart Librarian with a special trick.

Step 1: Building the "Prototype Library" (Mining the Knowledge)

First, the system takes a massive library of free-text reports (from a dataset called MIMIC-CXR) and uses a super-smart AI (an LLM) to read them.

• The Analogy: Imagine the AI is a translator. It reads a story saying, "The heart looks enlarged," and translates that into the specific checklist item: "Cardiomegaly: Yes."
• It does this for thousands of examples. For every possible answer on the checklist (e.g., "Lower Lobe," "Patchy," "Severe"), it gathers a small group of X-ray images that match that description.
• These groups of images become "Prototypes" (or "Visual Flashcards"). If the computer needs to decide if an opacity is in the "lower lobe," it can look at its "Lower Lobe Flashcard" to see what that actually looks like.

Step 2: The "Second Opinion" (The Architecture)

Now, the system tries to fill out the checklist for a new patient.

1. The Base Doctor: A standard AI looks at the new X-ray and makes a first guess. Let's say it guesses, "The opacity is in the upper lobe."
2. The Librarian Checks the Flashcards: The system asks, "Wait, does this image look more like the 'Upper Lobe' flashcards or the 'Lower Lobe' flashcards?"
3. The Correction: If the new image looks suspiciously like the "Lower Lobe" flashcards, the Librarian whispers to the Base Doctor: "Hey, I'm seeing strong evidence here that this is actually the lower lobe. Let's adjust the score."
4. The Final Decision: The system combines the Base Doctor's guess with the Librarian's "second opinion" to make the final, more accurate choice.

Why This is a Big Deal

• Solving the "Rare" Problem: In medical data, common things (like "no pneumonia") are easy to learn. Rare things (like a specific type of rare lung texture) are hard because there are very few examples in the structured data. But those rare things do appear in the millions of free-text stories. ProtoSR digs those rare examples out of the stories and turns them into flashcards.
• The "Long Tail" Fix: The paper shows that this method works best on the tricky, detailed questions (the "long tail" of rare attributes). It's like having a specialist who has read every single case file in history, helping you spot the rare details you might miss.

The Result

When they tested this on a benchmark called Rad-ReStruct, ProtoSR beat all previous methods. It didn't just get the easy questions right; it got the hard, detailed questions right by using the "wisdom of the crowd" from those millions of free-text reports.

In a nutshell:
ProtoSR is a system that teaches a computer to fill out a strict medical checklist by first reading millions of messy doctor's notes, turning those notes into visual "flashcards," and then using those flashcards to give the computer a helpful "second opinion" whenever it's unsure about a rare or detailed finding.

Technical Summary

1. Problem Statement

Automated Structured Radiology Reporting (SR) aims to organize findings into predefined fields and standardized answer options, offering better consistency and machine-readability than free-text reports. However, current automated SR systems face significant challenges:

• Data Scarcity & Sparsity: High-quality structured datasets (e.g., Rad-ReStruct) are small and imbalanced. They lack sufficient examples for rare findings and fine-grained attributes (e.g., specific locations, appearances, or severities), leading to poor supervision for long-tail classes.
• Information Gap: While large-scale free-text datasets (e.g., MIMIC-CXR) contain hundreds of thousands of paired images and detailed reports, the unstructured nature of this text makes it difficult to map directly to strict structured taxonomies.
• Limitations of Existing Methods: Generalist Vision-Language Models (VLLMs) often struggle with the specific constraints of hierarchical SR. Previous attempts to integrate knowledge often operate in unstructured output spaces or fail to effectively inject retrieved evidence into discrete decision-making pipelines.

2. Methodology: ProtoSR

The authors propose ProtoSR, a prototype-conditioned late-fusion framework that leverages unstructured free-text reports to enhance structured prediction. The method consists of two main stages:

A. Knowledge Base Construction (Mining & Normalization)

The system transforms a large corpus of free-text reports (MIMIC-CXR) into a multimodal knowledge base aligned with a specific structured template (Rad-ReStruct).

1. Terminology Expansion: An instruction-tuned Large Language Model (LLM) generates synonyms, abbreviations, and alternative phrasings for every target label in the structured template to handle clinical variability.
2. Template-Constrained Extraction: The LLM processes free-text reports hierarchically. It first determines if a finding exists, then extracts specific attribute values based on the template. Constrained decoding ensures outputs strictly adhere to the allowed answer options.
3. Post-Processing & Filtering: Rule-based filters remove uncertain or inconsistent extractions (e.g., removing a parent label if child labels are unsupported).
4. Prototype Bank Assembly: For each label $\ell$, the system aggregates image embeddings from all matching studies. It samples up to $K$ images and uses element-wise max pooling to create a single "visual prototype" embedding that captures the strongest signals for that specific label.

B. Knowledge-Enhanced Late Fusion Architecture

ProtoSR integrates this knowledge into a hierarchical Vision-Language Model (based on Rad-ReStruct) via a late-fusion mechanism:

1. Base Model: A backbone model (EfficientNet-B5 + RadBERT) processes the image and question context to produce base logits ($z_{base}$).
2. Prototype-Conditioned Branch:
   • Given the fused image-text representation, the model retrieves relevant prototypes from the knowledge bank using cosine similarity.
   • Retrieved prototypes are summarized into a visual feature vector ($v$) and an answer support vector ($u$) (a weighted sum of one-hot label vectors).
   • These vectors are concatenated and passed through an MLP to generate a support bias ($b_{sup}$), representing a "second opinion" based on similar cases.
3. Fusion: The final prediction is calculated by adding a scaled version of the support bias to the base logits:
   $$z_{final} = z_{base} + s \odot b_{sup}$$
   where $s$ is a learnable scaling vector that calibrates the influence of the retrieved evidence per answer dimension. This allows for targeted corrections on specific attributes without disrupting the base model's overall behavior.

3. Key Contributions

1. LLM-Driven Mining Pipeline: A novel pipeline that converts massive, unstructured free-text report collections into a structured, multimodal prototype knowledge base aligned with fine-grained reporting templates.
2. Prototype-Conditioned Late Fusion: A lightweight architectural module that retrieves visual prototypes and converts them into a residual logit correction signal. This enables the model to selectively correct predictions for rare or fine-grained attributes while preserving the base model's performance on common cases.
3. Data-Driven Second Opinion: The approach effectively leverages the "long tail" of knowledge present in routine free-text reports to solve the data scarcity problem in structured datasets.

4. Experimental Results

The method was evaluated on the Rad-ReStruct benchmark (3,597 chest X-ray studies) using MIMIC-CXR for knowledge mining.

• Knowledge Base Quality: The extraction pipeline achieved high coverage: 100% for Level 1 (coarse findings), 96% for Level 2 (specific findings), and 82% for Level 3 (fine-grained attributes), significantly expanding the available training signal for rare attributes.
• Performance Gains:
  • Overall F1: ProtoSR achieved 34.4, outperforming the previous best (Context-VQA at 32.9) and generalist VLLMs like CheXagent (32.4).
  • Fine-Grained Attributes (L3): The most significant improvement was observed in Level 3 questions (detailed attributes), where ProtoSR achieved an F1 of 7.4 compared to 3.2 for the next best method. This represents a relative improvement of +72.1% over the base model without knowledge integration.
• Ablation Studies:
  • Removing the knowledge branch dropped performance to the baseline.
  • Replacing prototypes with random noise or Gaussian noise resulted in baseline performance, confirming that the gains come from the meaningful semantic content of the prototypes, not just increased model capacity.
  • Early fusion (injecting knowledge directly into the input sequence) was less effective than the proposed late-fusion residual approach.

5. Significance

ProtoSR demonstrates that routine free-text reports, often considered a barrier to structured automation due to their variability, can be systematically mined and transformed into a powerful knowledge source. By using prototypes to inject "evidence" into the decision-making process, the model effectively bridges the gap between the abundance of unstructured data and the scarcity of structured annotations. This approach is particularly valuable for fine-grained medical tasks where rare attributes are critical for diagnosis but underrepresented in standard datasets, offering a scalable path toward more accurate and consistent automated radiology reporting.