⚡ electrical engineering

LLaMA-XR: A Novel Framework for Radiology Report Generation using LLaMA and QLoRA Fine Tuning

The paper introduces LLaMA-XR, a novel framework that combines LLaMA 3.1, DenseNet-121 image embeddings, and QLoRA fine-tuning to efficiently generate accurate and clinically coherent radiology reports, achieving state-of-the-art performance on the IU X-ray dataset.

Original authors: Md. Zihad Bin Jahangir, Muhammad Ashad Kabir, Sumaiya Akter, Israt Jahan, Minh Chau

Published 2026-04-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Md. Zihad Bin Jahangir, Muhammad Ashad Kabir, Sumaiya Akter, Israt Jahan, Minh Chau

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a radiologist, a doctor who looks at X-ray images to find out what's wrong with a patient's lungs. Every day, you have to stare at these black-and-white pictures and then write a detailed, professional report describing exactly what you see. It's like being a detective who has to write a police report after solving a crime, but you have to do it for hundreds of patients a day. It's tiring, it takes a long time, and if you're tired, you might miss a tiny clue.

This paper introduces a new "AI Assistant" called LLaMA-XR designed to help doctors write these reports faster and more accurately. Here is how it works, broken down into simple parts:

1. The Two-Brain System

Think of LLaMA-XR as a team of two specialists working together:

The "Eye" (DenseNet-121): This is a computer program trained to look at X-rays. Instead of just seeing a picture, it acts like a super-organized assistant who scans the image and creates a checklist. It doesn't write sentences; it just gives numbers. For example, it says, "There is a 60% chance of fluid in the lungs, a 20% chance of a broken bone, and 0% chance of a collapsed lung." It turns the complex image into a simple list of 18 different medical possibilities.
The "Writer" (LLaMA 3.1): This is a very smart language model (like a super-charged version of Siri or ChatGPT) that is an expert at writing. It takes the checklist from the "Eye" and turns those numbers into a full, professional medical report. It knows how to say, "The lungs are clear," or "There is a small shadow indicating pneumonia," using the correct medical words.

2. The Magic Trick: "QLoRA" (The Lightweight Suit)

Usually, teaching a giant AI like LLaMA 3.1 to be a medical expert requires a massive, expensive supercomputer. It's like trying to teach a giant elephant to dance; you need a huge stage and a lot of energy.

The researchers used a clever trick called QLoRA. Imagine you have a giant library of books (the AI's knowledge). Instead of rewriting every single book to teach it about X-rays, you just add a few sticky notes to the pages that matter.

The Result: The AI stays huge and smart, but it only needs to learn a tiny bit of new information. This makes the training process 60% faster and requires much less computer memory. It's like teaching the elephant to dance by just giving it a pair of dance shoes instead of building a whole new dance floor.

3. How They Taught It (The "Fill-in-the-Blank" Game)

To teach the AI, they used a dataset of real X-rays and the reports doctors wrote for them. They set up a game:

Input: They gave the AI the "checklist" (the numbers from the Eye) and a prompt saying, "Here are the findings, please write the report."
Output: The AI tried to write the report.
Correction: They compared the AI's report to the real doctor's report. If the AI missed a detail or used the wrong word, it learned from the mistake.

They did this thousands of times until the AI got really good at it.

4. The Results: Better Than the Competition

The researchers tested LLaMA-XR against other AI models that have been used in the past.

The Score: In the world of AI, they use scores like ROUGE-L and METEOR to see how well the AI's writing matches a human's. LLaMA-XR got the highest scores ever recorded for this specific task.
Why it matters: Some older AIs were good at matching words (like finding the same words in a sentence), but they often sounded robotic or missed the meaning. LLaMA-XR is better at understanding the context. It doesn't just say "lung bad"; it writes a coherent story that a real doctor can trust.

5. Why This is a Big Deal

Speed: It can draft a report in seconds, freeing up doctors to focus on the patient.
Accuracy: It reduces the chance of a tired doctor missing a small detail.
Accessibility: Because of the "QLoRA" trick, this powerful AI can run on standard computers, not just billion-dollar supercomputers. This means even smaller hospitals could use it.

The Bottom Line

LLaMA-XR is like giving every radiologist a tireless, hyper-smart intern who can look at an X-ray, make a checklist, and draft a perfect report in seconds. The doctor then just reviews it, signs off, and moves on to the next patient. It's not replacing the doctor; it's giving them a superpower to work faster and with fewer mistakes.

1. Problem Statement

Automated radiology report generation aims to assist radiologists by converting chest X-ray (CXR) images into descriptive clinical reports. However, existing methods face significant challenges:

Complexity of Medical Language: Generating reports requires not just describing visual features but correlating them with precise medical terminology and clinical context.
Limitations of Previous Models: Early approaches using CNN-RNN architectures struggled with long-range dependencies and complex sentence structures. While Transformer-based models and earlier Large Language Models (LLMs) improved performance, they often lacked the scale to capture nuanced medical relationships or required excessive computational resources for full fine-tuning.
Resource Constraints: Deploying massive models (e.g., 7B+ parameters) for medical fine-tuning is often computationally prohibitive due to high memory overhead, limiting their use in resource-constrained clinical environments.

2. Methodology: The LLaMA-XR Framework

The authors propose LLaMA-XR, a modular framework that integrates visual feature extraction with a quantized Large Language Model (LLM) for report generation. The architecture consists of three main components:

A. Visual Feature Extraction (DenseNet-121)

Model: A pre-trained DenseNet-121 (specifically DenseNet121-res224-all) is used to extract visual features from CXR images.
Input Processing: Images are resized to 224×224 pixels.
Output Representation: Instead of using high-dimensional raw embeddings, the model outputs a 36-dimensional vector. This vector consists of confidence scores for 18 specific medical conditions (e.g., Atelectasis, Pneumonia, Cardiomegaly) derived from two views: Frontal and Lateral (18 scores × 2 views = 36 dimensions).
Rationale: This approach provides clinically interpretable priors, reduces input dimensionality for the LLM, and aligns with standard radiology terminology while maintaining semantic richness.

B. Language Generation (LLaMA 3.1 8B)

Base Model: The framework utilizes LLaMA 3.1 8B, a state-of-the-art autoregressive decoder-only model with 8 billion parameters.
Prompt Engineering: The 36-dimensional confidence scores are formatted into an Alpaca-style prompt. The prompt includes a specific instruction to generate a medical report based on the provided classification scores for Frontal and Lateral views.
Architecture: The LLM processes the tokenized input through transformer blocks (self-attention and feedforward networks) to generate the final text report autoregressively.

C. Efficient Fine-Tuning (QLoRA & SFT)

To make the fine-tuning of the 8B parameter model feasible on limited hardware (e.g., a single A100 GPU), the authors employ Quantized Low-Rank Adaptation (QLoRA):

4-bit Quantization: The pre-trained weights of LLaMA 3.1 are quantized to 4 bits (NF4), reducing memory footprint by ~60%.
Low-Rank Adaptation (LoRA): Only small, trainable low-rank matrices ( $A$ and $B$ ) are updated during training, while the main weights ( $W_0$ ) remain frozen.
Supervised Fine-Tuning (SFT): The model is trained on the IU X-ray dataset (7,470 images, 3,955 reports) using the Unsloth library, which optimizes the training speed and memory efficiency of quantized models.
Training Hyperparameters: 3 epochs, batch size 8 (accumulated to 32), learning rate $2 \times 10^{-6}$ , and AdamW 8-bit optimizer.

3. Key Contributions

Novel Framework: Introduction of LLaMA-XR, which bridges the gap between visual data and textual generation by combining DenseNet-121 confidence scores with the LLaMA 3.1 8B model.
Parameter Efficiency: Demonstration that QLoRA allows for the fine-tuning of large 8B parameter models on medical datasets with significantly reduced memory overhead and training time, making high-performance AI accessible on standard hardware.
Superior Performance: The framework establishes new benchmarks on the IU X-ray dataset, outperforming numerous State-of-the-Art (SOTA) methods in semantic coherence and clinical relevance.
Clinical Relevance: The use of specific medical condition scores as input ensures the generated reports are grounded in clinical findings, reducing hallucinations compared to models relying solely on raw image embeddings.

4. Results

The model was evaluated on the IU X-ray dataset using standard NLP metrics and compared against 30+ SOTA methods (including R2Gen, PGT, UAR, and CvT-DistilGPT2).

Quantitative Metrics:
- ROUGE-L: 0.433 (Highest reported; +4.34% improvement over the previous best, SGF).
- METEOR: 0.336 (Highest reported; +54.13% improvement over UAR).
- BLEU-4: 0.180 (Slightly lower than some recent models, but the authors argue BLEU is less suitable for clinical text due to its focus on n-gram overlap rather than semantic meaning).
Qualitative Analysis:
- Faithfulness: LLaMA-XR demonstrated higher fidelity to source content, correctly identifying the absence of pathologies (e.g., "no pneumonia," "no effusion") where other models failed or hallucinated.
- Coherence: Generated reports showed better linguistic fluency and structured phrasing compared to models like CvT-DistilGPT2, which often produced redundant or generic text.
- Hallucination Reduction: The framework significantly reduced the generation of extraneous or inaccurate findings compared to baseline models.

5. Significance and Conclusion

The LLaMA-XR framework represents a significant step forward in medical AI by demonstrating that large language models can be effectively and efficiently adapted for specialized clinical tasks.

Clinical Utility: By prioritizing semantic accuracy (ROUGE-L/METEOR) over surface-level word matching (BLEU), the model generates reports that are more clinically actionable and reliable.
Scalability: The integration of QLoRA and 4-bit quantization proves that powerful 8B parameter models can be deployed in resource-constrained environments, democratizing access to advanced diagnostic tools.
Future Impact: The study highlights the potential of combining vision-language models to assist radiologists, particularly in settings with limited specialist availability. It suggests a pathway toward more accurate, efficient, and interpretable AI-assisted radiology, though future work should focus on validation with larger, diverse datasets (e.g., MIMIC-CXR) and real-world clinical deployment.