💻 computer science

Instruction-Guided Lesion Segmentation for Chest X-rays with Automatically Generated Large-Scale Dataset

This paper introduces instruction-guided lesion segmentation (ILS) for chest X-rays, presenting the large-scale, automatically generated MIMIC-ILS dataset and the ROSALIA model to enable accurate, user-friendly segmentation and explanation of diverse lesions via simple text instructions.

Original authors: Geon Choi, Hangyul Yoon, Hyunju Shin, Hyunki Park, Sang Hoon Seo, Eunho Yang, Edward Choi

Published 2026-03-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Geon Choi, Hangyul Yoon, Hyunju Shin, Hyunki Park, Sang Hoon Seo, Eunho Yang, Edward Choi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant library of chest X-rays, but they are all locked in a dark room. Inside the room, there are thousands of doctors' notes (radiology reports) describing what they see, but no one has ever drawn a picture of exactly where the problems are.

For a long time, computers trying to learn from these X-rays were like students who could read the notes but couldn't point to the spot on the map. They needed a human to draw a circle around every single spot of pneumonia or fluid, which is slow, expensive, and hard to do for millions of images.

This paper introduces a new system called ROSALIA (think of it as a super-smart medical assistant) and a massive new "textbook" called MIMIC-ILS that teaches it how to do this job automatically.

Here is the breakdown using simple analogies:

1. The Problem: The "Expert-Only" Barrier

Previously, if you wanted a computer to find pneumonia in an X-ray, you had to give it a very specific, expert-level description like: "Find the two infected areas in the upper right lung and the lower left lung."

The Issue: Regular people (or even busy doctors) don't want to write essays. They just want to say, "Show me the pneumonia," or even, "Is there pneumonia in the left lung?"
The Gap: Computers were great at finding things if you gave them a complex map, but terrible at understanding simple, conversational requests.

2. The Solution: The "Magic Photocopier" (MIMIC-ILS)

The researchers needed a way to teach the computer without hiring thousands of doctors to draw circles on millions of images. So, they built a fully automated pipeline.

Think of this pipeline as a Magic Photocopier that works in three steps:

Read the Notes: It takes the doctor's written report (e.g., "There is fluid in the bottom of the left lung") and uses a smart AI to turn that sentence into a structured list of facts.
The "Before and After" Trick: It uses a special AI tool (like a photo editor) to "erase" the disease from the X-ray, creating a "healthy" version. Then, it subtracts the healthy version from the original. The difference? That's the disease! It's like finding a stain on a shirt by comparing it to a clean shirt.
Cross-Checking: It checks if the "stain" it found actually matches the location mentioned in the report. If the report says "left lung" but the stain is on the "right," it discards that data.

The Result: They created MIMIC-ILS, a dataset with 1.1 million examples. It's like a massive textbook where every page has an X-ray, a simple question (e.g., "Segment the pneumonia"), and the correct answer (a digital mask showing exactly where the pneumonia is).

3. The Star Player: ROSALIA

Once they had this massive textbook, they trained a model called ROSALIA.

What it does: ROSALIA is like a detective with a highlighter.
How it works: You show it an X-ray and ask a simple question.
- You ask: "Show me the pneumonia in the right lung."
- ROSALIA: Highlights the pneumonia in the right lung and says, "Here it is."
- You ask: "Show me the pneumonia in the left lung." (But there isn't any).
- ROSALIA: Says, "I looked, and there is no pneumonia in the left lung," and shows you an empty screen.
- You ask: "Show me the opacity." (A general term for any white spot).
- ROSALIA: Highlights all the white spots and says, "These are the opacities."

4. Why This Matters

No More "Expert" Language: You don't need to be a radiologist to use it. You can ask simple questions.
Speed: It can process millions of images automatically, something humans couldn't do in a lifetime.
Accuracy: In tests, ROSALIA was incredibly good at finding the right spots and correctly saying "nothing here" when a disease wasn't present. Other models often failed, either drawing the wrong thing or drawing the whole lung instead of just the disease.

The Big Picture

Imagine you have a flashlight that can instantly find a lost toy in a messy room. Before this paper, the flashlight only worked if you gave it the exact GPS coordinates of the toy. Now, thanks to ROSALIA and MIMIC-ILS, you can just say, "Find the red ball," and the flashlight will instantly point to it, even if the ball is hidden under a blanket.

This technology brings medical AI closer to being a helpful tool for everyone, not just a tool for computer scientists and expert radiologists.

1. Problem Statement

Current automated lesion segmentation models for Chest X-rays (CXRs) face two primary limitations:

Data Scarcity & Label Constraints: Existing datasets rely heavily on manual expert annotation, resulting in small scales and limited lesion types (often focusing on a single pathology like COVID-19 or pneumothorax). Most provide only coarse bounding boxes rather than pixel-level segmentation masks.
Rigid Input Requirements: Existing Vision-Language Models (VLMs) for medical RIS (Referring Image Segmentation) require complex, expert-level textual descriptions (e.g., "Bilateral pulmonary infection, two infected areas...") as input. This creates a barrier for non-experts and limits usability, as users must already interpret the image to formulate the prompt.

The authors propose Instruction-Guided Lesion Segmentation (ILS), a task where a model segments lesions based on simple, user-friendly natural language instructions (e.g., "Segment the pneumonia in the right lung" or "Is there atelectasis in the left base?"), including the ability to confirm the absence of a lesion.

2. Methodology

The paper introduces a fully automated pipeline to construct a large-scale dataset and a specialized model to utilize it.

A. Dataset Construction: MIMIC-ILS

The authors constructed MIMIC-ILS, the first large-scale instruction-answer dataset for CXR lesion segmentation, using the MIMIC-CXR database (192K images and corresponding radiology reports). The pipeline is fully automated (zero human intervention) and consists of four stages:

Report Structuring & Location Mapping:
- Large Language Models (LLMs) parse radiology reports to extract structured tuples: (Entity, Sentence Index, Presence, Certainty, Location, Predicted Lesion Type).
- Locations are mapped to standardized anatomical labels (e.g., "lower lung" $\rightarrow$ "right lung base").
Spatial Information Extraction:
- Anomaly Map: Generated using RadEdit (a diffusion model) by editing the image to remove lesions ("No acute cardiopulmonary process") and computing the difference ( $x - \hat{x}$ ).
- Anatomy Masks: Generated using CXAS (anatomy segmentation model) on the edited images to define lung zones.
- Lesion Detection: A pretrained YOLO model detects bounding boxes for various lesions.
Grounded Lesion Mask Generation:
- A filtering algorithm (Algorithm 1) integrates the anomaly map, anatomy masks, and YOLO bounding boxes.
- It selects high-confidence candidates based on overlap with reported locations, confidence scores, signal intensity in the anomaly map, and size relative to lung masks.
- Connected components are extracted and refined (noise removal, expansion) to create the final segmentation mask.
Instruction-Answer Pair Generation:
- Positive Pairs: Generated for 7 major lesion types (Cardiomegaly, Pneumonia, Atelectasis, Opacity, Consolidation, Edema, Effusion). Instructions include Basic (specific location), Global (entire lung), and Lesion Inference (predict type of opacity).
- Negative Pairs: Generated by pairing lesion types with "empty locations" (regions where the report confirms absence) to train the model to detect non-existence.
- Verification: A location verification step ensures the generated mask spatially overlaps with the reported location. If not, it is discarded or marked as an empty sample.

Dataset Statistics:

1.1 Million instruction-answer pairs.
Derived from 192K images and 91K unique segmentation masks.
Expert evaluation on a subset showed a 96.4% acceptance rate, validating the quality of the automated pipeline.

B. Model: ROSALIA

The authors fine-tuned LISA (a VLM integrating a large language model with the Segment Anything Model - SAM) to create ROSALIA (RadiOlogy Segmentation Assistant).

Architecture: The VLM backbone (LLaVA) processes the CXR image and text instruction, outputting a special [SEG] token and a textual description. The embedding of [SEG] is fed into the SAM decoder to generate the mask.
Training: Fine-tuned on MIMIC-ILS using a combined loss function: Language Loss (Cross-Entropy) + Mask Loss (Binary Cross-Entropy + DICE).
Capabilities: ROSALIA can handle specific segmentation, global segmentation, and absence confirmation, providing both a mask and a textual explanation.

3. Key Contributions

Novel Automated Pipeline: A fully automated framework to generate pixel-level lesion masks and diverse instruction-answer pairs directly from raw image-report pairs, eliminating the need for expensive manual annotation.
MIMIC-ILS Dataset: The first large-scale dataset (1.1M samples) specifically designed for Instruction-Guided Lesion Segmentation in CXRs, covering seven major lesion types with high-quality, expert-validated masks.
ROSALIA Model: The first VLM trained for ILS in CXRs. It demonstrates the ability to interpret diverse user intents (specific, global, negative) and produce accurate segmentations and textual reasoning.

4. Results

The model was evaluated on the MIMIC-ILS test set against baselines from both general domains (LISA, Text4Seg, PixelLM) and medical domains (BiomedParse, RecLMIS, IMIS-Net).

Segmentation Performance:
- ROSALIA achieved a gIoU of 71.2% and cIoU of 75.6%, significantly outperforming all baselines (the next best medical baseline, BiomedParse, scored 23.8% gIoU).
- Performance was consistent across lesion types, with the lowest gIoU (Pneumonia) still at 57.2%.
Absence Confirmation (Negative Samples):
- ROSALIA achieved 91.8% accuracy in correctly identifying empty targets (N-Acc).
- Most baselines failed completely on negative cases (N-Acc near 0%), highlighting the necessity of the specific ILS training data.
Textual Accuracy:
- ROSALIA achieved 94.4% overall accuracy in generating correct textual descriptions (location, presence/absence, and lesion type inference).

5. Significance

Democratization of Medical AI: By enabling segmentation via simple natural language instructions (e.g., "Show me the pneumonia"), the system lowers the barrier for non-experts (e.g., triage nurses, patients) to interact with CXR analysis tools.
Scalability: The fully automated pipeline proves that high-quality, pixel-level medical datasets can be constructed at scale without human annotators, addressing the bottleneck of data scarcity in medical AI.
Foundation for Grounding: MIMIC-ILS serves as a foundational resource for fine-grained lesion grounding, enabling future research into more complex diagnostic reasoning and multi-modal medical AI applications.
Clinical Utility: The ability to confirm the absence of a lesion is crucial for clinical workflows, allowing the model to act as a reliable screening tool rather than just a detection tool.