💻 computer science

GLeVE: Graph-Guided Lesion Grounding with Proposal Verification in 3D CT

The paper introduces GLeVE, a graph-guided framework that leverages anatomical priors and octree-based refinement to achieve precise 3D CT lesion grounding by bridging the semantic-spatial gap between radiology reports and volumetric anatomy.

Original authors: Shuo Jiang, Yuhao Hong, Chunbo Jiang, Weihong Chen, Huangwei Chen, Shenghao Zhu, Beining Wu, Mingxuan Liu, Zhu Zhu, Feiwei Qin, Min Tan, Yifei Chen

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shuo Jiang, Yuhao Hong, Chunbo Jiang, Weihong Chen, Huangwei Chen, Shenghao Zhu, Beining Wu, Mingxuan Liu, Zhu Zhu, Feiwei Qin, Min Tan, Yifei Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a doctor looking at a 3D scan of a patient's body (a CT scan) while reading a long, detailed report written by a radiologist. The report says things like, "There is a small, dark spot in the left kidney, about the size of a grape, and another one in the liver."

The Problem:
Currently, if a doctor wants to find those specific spots on the 3D scan, they have to act like a detective searching through hundreds of slices of images, one by one. It's slow, tiring, and easy to miss things. Existing computer programs that try to help are like "general search engines": they might highlight the whole kidney or the whole liver, but they struggle to pinpoint the exact grape-sized spot mentioned in the text. They often get confused when there are multiple spots or when the text is complex.

The Solution: GLeVE
The authors of this paper created a new system called GLeVE (Graph-Guided Lesion Grounding with Proposal Verification). Think of GLeVE as a highly trained, super-organized medical assistant that doesn't just "read" the report but "understands" the relationships between the words and the 3D image.

Here is how GLeVE works, broken down into three simple steps using analogies:

1. The "Social Network" of Symptoms (Graph Reasoning)

Instead of treating the report as a simple list of words, GLeVE builds a social network map for the lesions.

How it works: It takes a sentence like "A large spot in the liver and a small one in the kidney" and draws a map connecting them. It knows that the "large spot" belongs to the "liver" and the "small spot" belongs to the "kidney." It also understands how they relate to each other.
The Analogy: Imagine you are trying to find two specific people at a crowded party. A normal search might just look for "people." GLeVE, however, creates a map that says, "Person A is wearing a red hat and standing near the DJ; Person B is wearing a blue shirt and standing near the snack table." This map helps the computer distinguish between similar-looking things (like two different spots in the liver) by understanding their unique "social connections" in the text.

2. The "Security Checkpoint" (Proposal Verification)

Once the system has a list of potential spots based on the text, it doesn't just guess. It runs them through a security checkpoint.

How it works: The system generates several possible locations for a lesion. Then, it checks each one against the report's details (like size, location, and density) and the actual anatomy of the body. If a candidate spot is in the wrong organ or doesn't match the size described, it gets rejected.
The Analogy: Imagine you are hiring a bodyguard based on a description: "He is tall, has a scar, and carries a briefcase." You might see three tall men. A normal system might pick the first one it sees. GLeVE acts like a strict bouncer who checks IDs: "You are tall, but you don't have a scar. You're out. You have a scar, but you're carrying a backpack, not a briefcase. You're out." Only the one who matches all the criteria gets to stay. This ensures a perfect "one-to-one" match between the text and the image.

3. The "Zoom-In" Lens (Octree Refinement)

Even after finding the right spot, the edges might be blurry, especially for tiny lesions. GLeVE uses a hierarchical zoom-in technique.

How it works: It starts by finding the general area (the "coarse" view) and then progressively zooms in, refining the edges of the spot layer by layer, like peeling an onion or zooming in on a map from a country level down to a street level.
The Analogy: Think of looking at a blurry photo on your phone. First, you see a blob. Then you pinch to zoom in a little; it's still a bit fuzzy. You zoom in again, and suddenly you can see the exact outline of the object. GLeVE does this mathematically, starting with a rough guess and repeatedly sharpening the edges until the computer's outline perfectly matches the actual shape of the lesion in the 3D scan.

The Results

The researchers tested GLeVE on a massive dataset of abdominal scans (the "AbdomenAtlas 3.0"). They found that:

It's more accurate: It finds the exact spots mentioned in reports much better than previous computer models.
It handles crowds: It is very good at telling the difference between multiple lesions in the same organ (like distinguishing two different spots in the liver).
It works with less data: Even when the computer wasn't shown perfect "answer keys" (masks) for every single case during training, it still performed very well, suggesting it learns the logic of the reports rather than just memorizing images.

In summary: GLeVE turns a confusing mix of text and 3D images into a clear, verified map. It connects the dots between what the doctor says and what the machine sees, ensuring that when a report mentions a specific lesion, the computer points to exactly that spot with high precision.

Technical Summary: GLeVE – Graph-Guided Lesion Grounding with Proposal Verification in 3D CT

Problem Statement

The core challenge addressed by this work is the "semantic-spatial gap" between free-text radiology reports and 3D Computed Tomography (CT) volumes. While reports provide structured descriptions of lesions (organ attribution, sub-regions, size, Hounsfield Units), clinicians currently must manually and iteratively search through CT slices to verify these findings. This process is inefficient, experience-dependent, and prone to error, particularly in multi-organ or multi-lesion scenarios where salient information is diluted by routine text.

Existing solutions face significant limitations:

General Segmentation Models: Methods like SwinUNETR or MedSAM produce global masks but lack interpretable, lesion-by-lesion alignment with specific report descriptions.
Phrase-Level Alignment: Current text-grounding approaches often align individual phrases rather than treating a full lesion description as a coherent semantic unit, leading to unreliable localization.
Annotation Scarcity: Many clinical datasets lack pixel-level masks, making it difficult to train models that enforce precise one-to-one correspondence between text and anatomy without heavy supervision.

Methodology: GLeVE Framework

The authors propose GLeVE (Graph-Guided Lesion Grounding with Proposal Verification), a framework designed to achieve verifiable, pixel-accurate lesion localization. The architecture consists of three primary modules:

1. Lesion Semantic Modeling and Querying (LeQu)

Instead of processing text linearly, GLeVE treats each lesion description as an atomic semantic unit.

Structured Parsing: A frozen Large Language Model (Qwen3-8B) extracts structured fields (organ, sub-location, size, HU) from reports.
Graph Construction: These fields are organized into a semantic graph $G_i = (V_i, E_i)$ containing lesion, anatomical, and attribute nodes. Edges encode relations such as lesion-organ attribution and inter-lesion constraints.
Relation-Aware Reasoning: A Graph Transformer with relation-type-aware attention mechanisms processes the graph. This allows the model to disambiguate multi-lesion cases by leveraging cross-lesion comparisons.
Query Generation: The final representation of the lesion node is mapped to a set of "lesion-wise queries" ( $q_i$ ) via a QueryBank Generator, serving as the semantic summary for localization.

2. Anatomy-Prior Lesion Proposal and Verification (AnVer)

This module ensures that generated candidates adhere to anatomical constraints and enforces a strict one-to-one text-lesion alignment.

Soft Modulation: An organ segmenter provides soft masks which are used to modulate visual features via Feature-wise Linear Modulation (FiLM). This injects anatomical awareness into the voxel features, suppressing irrelevant regions while preserving global spatial structure.
Candidate Generation: The lesion queries are matched against the organ-aware feature map using voxel-level cosine similarity to generate a response map. Top-K high-response components are selected as initial candidates.
Region-Level Verification: To mitigate noise from voxel-level peaks, candidates undergo region-level verification. The system constructs evidence vectors encoding anatomical coverage and report-attribute consistency (e.g., volume and HU overlap). A scoring mechanism selects the single best match per description, minimizing the entropy of the candidate distribution to enforce a dominant match.

3. Octree Autoregressive Lesion Refinement (OcRe)

To achieve pixel-accurate boundaries, especially for small or low-contrast lesions, GLeVE employs a coarse-to-fine refinement strategy.

Octree Construction: Starting from the selected candidate, an octree is built to hierarchically subdivide the region of interest.
Autoregressive Refinement: A conditional autoregressive mechanism uses the prediction from the previous level as a structural prior for the next. A shared 3D residual refinement decoder processes concatenated local features and upsampled previous predictions to progressively refine the mask.
Weak Supervision: In the absence of pixel-level masks, the model is optimized using a report-driven weak supervision loss. This loss penalizes discrepancies in predicted volume and HU against reported values and enforces separation between lesions, alongside a unimodality loss to ensure single dominant matches.

Key Contributions

Graph-Guided Semantic Modeling: The introduction of a relation-aware graph transformer that treats full lesion descriptions as atomic units, enabling the disambiguation of complex multi-lesion scenarios.
Anatomy-Aware Verification: A novel proposal verification mechanism that combines soft anatomical modulation with region-level consistency checks to enforce one-to-one text-lesion alignment, reducing spurious responses.
Octree-Based Refinement: An autoregressive, hierarchical refinement strategy that improves boundary delineation for small lesions without incurring prohibitive computational costs.
Weakly Supervised Training: A report-driven objective function that allows the model to learn from data lacking pixel-level annotations by leveraging reported quantitative attributes (volume, HU).

Experimental Results

Experiments were conducted on AbdomenAtlas 3.0, a dataset comprising 9,262 cases with reports reviewed by radiologists and voxel-aligned tumor masks for liver, kidney, and pancreas lesions.

Performance: GLeVE consistently outperformed classical multimodal foundation models (e.g., CT-CLIP, M3D-LaMed) and universal segmentation methods (e.g., nnUNet, MedFormer).
- Under full supervision (100% masks), GLeVE achieved a 76.2% Lesion Recall (LR) and a 35.3% Lesion Localization Score (LLS).
- It demonstrated a 7.26% improvement in Dice score over the best universal segmentation method and reduced the 95th percentile Hausdorff Distance (HD95) by 7.3 mm compared to advanced report-assisted baselines.
Robustness: Under weak supervision (10% or 25% masks), GLeVE remained competitive, often outperforming alternatives, demonstrating robustness to annotation scarcity.
Ablation Studies: Removing any of the three core modules (LeQu, AnVer, OcRe) resulted in significant performance drops, confirming that semantic modeling, verification, and hierarchical refinement are complementary and essential for robust localization.

Significance and Claims

The paper claims that GLeVE provides a verifiable, pixel-accurate solution for grounding radiology reports in 3D CT. By establishing a traceable, one-to-one alignment between key report descriptions and imaging evidence, the framework supports reliable clinical interpretation and quantitative evaluation. The authors emphasize that their approach bridges the gap between free-text narratives and volumetric anatomy, offering a practical tool to reduce the manual burden on clinicians in verifying critical findings. The work suggests that future extensions could apply this framework to broader anatomical regions and diverse disease categories.