💻 computer science

LUX: A Lesion-Aware Graph-Conditioned Visual - Language Architecture for Explainable Endoscopic Captioning

LUX is a novel graph-conditioned vision-language architecture that enhances explainable endoscopic captioning for ulcerative colitis by constructing lesion-centric scene graphs to guide token-level generation, thereby improving clinical accuracy, interpretability, and grounding while reducing hallucinations compared to existing models.

Original authors: Alexis Ivan Escamilla-Lopez, Gilberto Ochoa-Ruiz, Salvador Hinojosa, Sharib Ali

Published 2026-08-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alexis Ivan Escamilla-Lopez, Gilberto Ochoa-Ruiz, Salvador Hinojosa, Sharib Ali

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Inside the human colon, a chronic condition known as ulcerative colitis causes the lining to become inflamed, red, and fragile. To understand how severe this inflammation is, doctors rely on a procedure called an endoscopy, where a flexible camera is passed through the digestive tract to capture images of the tissue. A specialist then examines these pictures, looking for specific signs like bleeding, ulcers, or a loss of the normal blood vessel pattern. Based on what they see, they assign a severity score and write a detailed description of the disease's state. This process is vital for deciding on treatment, but it is also difficult and subjective. Two different doctors might look at the same image and disagree on how sick the patient is, or one might miss a subtle sign of trouble that another would catch.

For years, scientists have tried to build computer programs that can read these medical images and write their own descriptions, hoping to help doctors be more consistent and accurate. Early attempts used artificial intelligence to look at the whole picture at once, compressing the entire image into a single summary before trying to write a sentence. While these systems could produce fluent text, they often failed to connect their words to the actual spots of disease in the image. They might describe bleeding that wasn't there or miss a severe ulcer because they were looking at the "big picture" rather than the specific details. This lack of connection made the computer's reports unreliable for clinical use, as the machine could not explain why it wrote what it did.

A team of researchers has now developed a new system called LUX that changes how computers approach this task. Instead of treating an endoscopic image as a single, blurry whole, LUX breaks the image down into specific, meaningful parts. When the system looks at a picture of an inflamed colon, it first identifies the exact locations of the problem areas, such as a patch of redness or a small sore. It then treats these spots as individual characters in a story, mapping out how they relate to one another. If a sore is next to a bleeding area, the system notes that connection. It builds a structured map of the disease, where every node represents a specific lesion and every line between them represents a relationship, like proximity or similarity.

This map of lesions becomes the foundation for the computer's writing. Rather than guessing what to say based on a general impression of the image, the system uses this map to guide every word it generates. As it constructs a sentence, it constantly checks its work against the specific spots it has identified. If it writes the word "bleeding," the system ensures that this word is directly tied to a specific area in the image where bleeding was detected. This method forces the computer to ground its language in visual evidence, preventing it from inventing symptoms that do not exist. The result is a description that not only sounds professional but also points directly to the physical evidence supporting each claim.

The researchers tested this approach against many other methods, including systems based on massive, general-purpose artificial intelligence models that have read millions of documents. They found that LUX was significantly better at describing the images accurately. It produced fewer errors, such as describing a healthy colon as diseased or missing a severe ulcer. In tests measuring how well the computer's descriptions matched those written by human experts, LUX outperformed all other systems. It was particularly successful at reducing "hallucinations," a term researchers use for when a computer confidently describes something that is not actually there. While other systems made these mistakes about 9.4 percent of the time, LUX reduced that rate to just 5.3 percent.

The study also showed that this new method helps the computer understand the severity of the disease more accurately. In ulcerative colitis, the severity is often determined by how different signs appear together and where they are located. Because LUX maps out the relationships between these signs, it can tell the difference between a mild case and a severe one with greater precision than systems that only look at the image as a whole. When human doctors reviewed the descriptions generated by LUX, they rated them as highly accurate and clinically useful, noting that the system correctly identified specific features like vascular patterns and tissue texture.

This work suggests that for medical imaging, especially in complex fields like endoscopy, simply making a computer smarter or giving it more data is not enough. The system needs to be taught to look at the image the way a doctor does: by focusing on specific, localized problems and understanding how they fit together. By building a structured map of the disease before it tries to write a sentence, LUX bridges the gap between seeing an image and understanding what it means. This approach offers a path toward artificial intelligence that is not just fluent, but also trustworthy and transparent, providing doctors with a tool that can explain its reasoning just as clearly as it describes the patient's condition.

Technical Summary: LUX – A Lesion-Aware Graph-Conditioned Visual–Language Architecture for Explainable Endoscopic Captioning

1. Problem Statement

The interpretation of endoscopic imagery for ulcerative colitis (UC) is a complex, subjective process heavily reliant on the assessment of subtle mucosal inflammation, vascular patterns, and lesion distribution. While deep learning has advanced automated analysis, existing vision–language models (VLMs) for medical image captioning suffer from critical limitations:

Global Embedding Reliance: Most models compress heterogeneous pathological patterns into a single global visual embedding. This approach neglects the localized and relational nature of pathological evidence, leading to captions that lack lesion-level grounding.
Hallucination and Lack of Traceability: Generated descriptions often conflate distinct findings or introduce unsupported pathological statements (hallucinations) because the linguistic generation is not explicitly constrained by specific visual evidence.
Interpretability Gap: Current explainability methods (e.g., Grad-CAM) provide post-hoc visualizations but do not integrate interpretability into the reasoning process itself. Clinicians require systems that associate localized findings with coherent diagnostic descriptions, not just highlight diffuse attention regions.
Domain Specificity: General-purpose foundation models (LLMs/VLMs) rely heavily on large-scale language priors rather than explicit pathological evidence, which is insufficient for the nuanced, evidence-driven reasoning required in gastrointestinal endoscopy.

2. Methodology

The authors propose LUX (Lesion-aware Unified eXplainable captioning), a framework that reframes endoscopic captioning as an evidence-grounded clinical reasoning process rather than a direct image-to-text translation. The architecture consists of five interacting components:

A. Visual Hybrid Encoder (ResNet-50 + CBAM)

Backbone: A ResNet-50 initialized with ImageNet weights, adapted for colonoscopic imaging.
Attention Mechanism: A Convolutional Block Attention Module (CBAM) is integrated after each residual stage. This selectively amplifies discriminative mucosal patterns (e.g., vascular attenuation, ulceration) while suppressing irrelevant background information.
Multi-scale Fusion: Features from different receptive fields ( $conv3\_x, conv4\_x, conv5\_x$ ) are concatenated and projected to ensure compatibility with the T5 decoder.

B. Scene–Lesion Graph Construction

Lesion Localization: Candidate pathological regions are localized using Grad-CAM activation maps, regularized during training by a Dice-based loss against manually annotated lesion masks (from a subset of the LIMUC dataset).
Graph Formation: Localized regions are abstracted into graph nodes ( $x_i$ ). Edges ( $w_{ij}$ ) encode both spatial proximity (Euclidean distance between centroids) and semantic similarity (cosine similarity of node embeddings).
Relational Reasoning: A Graph Convolutional Network (GCN) propagates contextual information across connected lesions, capturing interactions such as the co-occurrence of ulceration and bleeding. A global graph descriptor is obtained via attention pooling.

C. Lesion-Aware Captioning (T5 Decoder + Dual Cross-Attention)

Dual Cross-Attention: The T5 decoder employs a dual attention mechanism where each generated token attends jointly to global visual features and the lesion graph embeddings.
Dynamic Gating: A learnable gating parameter ( $\gamma_t$ ) dynamically balances the influence of visual context and lesion-centric reasoning at each decoding step.
Token-to-Lesion Alignment: An explicit alignment matrix ( $M_{tok-les}$ ) is computed to quantify the contribution of specific lesion nodes to each generated token, ensuring traceability between language and pathological evidence.

D. Linguistic Formatting

A lightweight, deterministic post-processing module standardizes the output. It maps generated tokens to a curated clinical vocabulary (e.g., UCEIS descriptors), enforces grammatical consistency, and reorders findings to follow an "evidence-first" clinical narrative structure (mucosal appearance $\to$ vascular pattern $\to$ lesions $\to$ bleeding).

E. Explainability

The system provides pixel-level explanations via Grad-CAM heatmaps and token-level lesion attribution, linking every generated word to specific pathological regions.

Training Strategy
The framework utilizes a dual-dataset approach:

UC-Caption: A curated dataset of 500 images with expert-authored captions and MES labels, used for supervised caption generation.
LIMUC: A large-scale dataset (11,276 images) with MES labels but no captions. It is used for pretraining the visual encoder and GCN, and for semi-supervised pseudo-caption augmentation. A manually annotated subset (400 images) provides lesion-level supervision for the localization branch.

3. Key Contributions

Lesion-Aware Graph-Conditioned Architecture: A novel framework that integrates localized pathological evidence directly into the caption generation process via a scene-lesion graph.
Structured Scene–Lesion Graph: A representation that models spatial and semantic relationships among inflammatory findings, moving beyond isolated lesion detection to relational reasoning.
Dual Cross-Attention Mechanism: A decoding strategy that enables token-level grounding, allowing generated language to attend explicitly to lesion-level representations.
Evidence-Grounded Formulation: The formulation of endoscopic captioning as a multimodal reasoning problem constrained by visual evidence, reducing reliance on linguistic priors.
Comprehensive Evaluation: A unified benchmark comparing LUX against classification-oriented models, standard captioning baselines, and state-of-the-art foundation VLMs (e.g., Qwen2-VL, LLaVA-Med).

4. Experimental Results

Evaluated on the UC-Caption test set and validated on the LIMUC dataset, LUX demonstrated superior performance across multiple metrics:

Caption Quality: LUX outperformed all baselines, achieving a CIDEr score of 0.92 (vs. 0.79 for the strongest baseline) and a SPICE score of 0.25 (vs. 0.19). The significant gains in SPICE indicate improved relational and scene-level semantic correctness.
Hallucination Reduction: LUX reduced the hallucination rate to 5.3%, a 44% relative reduction compared to the strongest foundation VLM baseline (Qwen2-VL at 9.4%).
Clinical Consistency: The model maintained high accuracy in Mayo Endoscopic Subscore (MES) classification (84.7%) while generating clinically faithful descriptions.
Ablation Studies: Removing the lesion graph or the dual cross-attention mechanism resulted in the most significant performance degradation, confirming that relational reasoning at the decoder level is the primary driver of clinical faithfulness.
Human Evaluation: Expert reviewers rated LUX highly for clinical correctness (4.4/5), lesion specificity (4.6/5), and hallucination avoidance (4.7/5), noting its ability to describe vascular attenuation, friability, and ulceration accurately without unsupported severity inflation.

5. Significance and Claims

The paper claims that LUX offers a transparent and clinically meaningful step toward explainable AI for gastrointestinal endoscopy. Its primary significance lies in demonstrating that explicit lesion-level grounding and structured relational reasoning provide complementary benefits to model scale.

Beyond Scale: The authors argue that in specialized clinical domains with limited annotated data, architectural inductive biases (lesion localization, graph encoding) are more effective for ensuring clinical reliability than simply scaling up foundation models.
Traceable Reasoning: Unlike post-hoc explainability, LUX integrates interpretability into the generation process, ensuring that every clinical statement is directly aligned with localized pathological evidence.
Clinical Utility: By reducing hallucinations and improving the correspondence between generated text and visual findings, LUX addresses a critical barrier to the deployment of AI in clinical workflows, offering a trustworthy decision-support tool for ulcerative colitis assessment.

The authors conclude that while foundation models excel in linguistic fluency, they lack the explicit mechanisms to associate textual findings with localized evidence. LUX bridges this gap, providing a principled path toward evidence-grounded multimodal AI that is both accurate and interpretable.