💻 computer science

Conv-Guided Mamba Vision Transformer and SPP-DenseNet121 based Hybrid Multi-Feature Fusion and Query Expansion in Content-Based Medical Image Retrieval

This paper proposes a novel Content-Based Medical Image Retrieval framework that integrates a Conv-Guided Mamba Vision Transformer with SPP-DenseNet121 for hybrid multi-feature fusion and employs query expansion to achieve high retrieval accuracy on endoscopic bladder tissue and gastrointestinal datasets.

Original authors: Mathana Gopal Arulsamy, Pallikonda Rajasekaran Murugan, Md. Jakir Hossen, Wai Kit Wong, Poh Kiat Ng, Gomathy Nayagam M

Published 2026-07-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mathana Gopal Arulsamy, Pallikonda Rajasekaran Murugan, Md. Jakir Hossen, Wai Kit Wong, Poh Kiat Ng, Gomathy Nayagam M

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking through a massive, endless library where every single book is a photograph of the inside of a human body. If you were a doctor trying to find a picture of a specific type of bladder tumor to help a patient, you wouldn't want to flip through millions of pages one by one. You'd want a super-smart librarian who could instantly grab the exact photos you need. This is the world of Content-Based Medical Image Retrieval (CBMIR). Instead of searching by text labels like "cancer" or "polyp," this system searches by looking at the actual picture content—the colors, shapes, and textures. The challenge is that medical images are tricky; two pictures might look almost identical but mean very different things to a doctor, or two pictures of the same disease might look wildly different depending on how the camera was held or the lighting. The goal is to build a digital detective that understands both the tiny details and the big picture to find the right medical images quickly and accurately.

In this study, the researchers built a new kind of digital detective called a Hybrid Multi-Feature Fusion and Query Expansion framework. Think of it as a team of two expert detectives working together to solve a case. The first detective is a specialist in local details, using a tool called SPP-DenseNet121. This detective is great at spotting tiny clues like the texture of a tissue or the edge of a lesion, no matter how big or small the spot is. The second detective is a master of global context, using a new, efficient tool called Conv-Guided Mamba Vision Transformer (CG-MViT). This detective looks at the whole scene to understand how different parts of the image relate to each other over long distances, kind of like understanding the layout of a whole room rather than just a single brick.

Usually, when you combine two different sets of clues, you might end up with a messy pile of redundant information. To fix this, the team used a special math trick called Discriminative Multiple Canonical Correlation Analysis (DMCCA). Imagine this as a super-organized filing system that takes the notes from both detectives, removes the duplicates, and merges them into one perfect, crystal-clear report. But the team didn't stop there. They knew that sometimes the first guess isn't perfect, so they added a "second-guess" step called Pseudo-Relevance Feedback (PRF). This is like asking the computer, "Hey, the top 10 pictures you found look good, right? Let's assume they are all correct and use them to refine our search." This helps the system understand the doctor's intent better and find even more relevant images.

The researchers tested their new detective team on two real-world medical photo libraries: the Kvasir dataset (which contains images of the gastrointestinal tract) and the Endoscopic Bladder Tissue dataset. The results were impressive. On the Kvasir dataset, the system found the right images with a high degree of accuracy, scoring a 97.72% success rate when looking at the top 5 results, and still maintaining a strong 91.02% even when looking at the top 100 results. On the more challenging bladder tissue dataset, it achieved 92.60% accuracy for the top 5 results and 66.82% for the top 100.

To make sure their new system was actually the hero and not just lucky, the researchers ran a series of "ablation studies." This is like taking apart a car engine to see which part makes it go fastest. They tested the system with just the first detective, then just the second, then both without the filing system, and finally the full team. They found that every single part contributed to the success; the full team with the filing system and the "second-guess" step performed the best. The study suggests that by combining local and global views, organizing the data smartly, and letting the system learn from its own initial guesses, medical image retrieval can become much more reliable. This could help doctors find similar past cases faster, aiding in diagnosis and treatment planning, though the authors note that more testing on larger and different types of medical images is needed to confirm how well this works in every hospital setting.

Technical Summary: Conv-Guided Mamba Vision Transformer and SPP-DenseNet121 based Hybrid Multi-Feature Fusion and Query Expansion in Content-Based Medical Image Retrieval

Problem Statement
Content-Based Medical Image Retrieval (CBMIR) is critical for clinical decision-making, yet retrieving specific endoscopic bladder tissue images from massive repositories remains challenging. The primary difficulties stem from high intra-class variations, similarities between different pathological classes, and diverse tissue appearances across imaging modalities. Traditional CBMIR methods relying on handcrafted features (e.g., LBP, HOG) fail to capture complex medical structures. While deep learning models like CNNs excel at local feature extraction, they often struggle with long-range dependencies and global contextual information. Conversely, pure Transformer-based models capture global context but suffer from high computational costs and may overlook local structural details necessary for distinguishing pathological variations. Furthermore, existing systems often face a "semantic gap," where visually similar images are retrieved but lack clinical relevance, and feature fusion methods frequently introduce redundancy without effectively leveraging heterogeneous feature spaces.

Methodology
The authors propose a novel CBMIR framework that integrates multi-scale local feature extraction, global contextual learning, discriminative feature fusion, and query refinement. The architecture operates in two phases: offline feature learning and online retrieval.

Preprocessing: Input images undergo resizing and adaptive contrast enhancement to standardize illumination and improve visualization of tissue structures.
Dual-Branch Feature Extraction:
- Local Multi-Scale Branch (SPP-DenseNet121): Utilizes a DenseNet121 backbone with dense connections to facilitate feature reuse and mitigate vanishing gradients. A Spatial Pyramid Pooling (SPP) layer is integrated to capture abnormalities at multiple scales (e.g., 1×1, 2×2, 4×4 grids), ensuring that both global image details and local lesion features are preserved regardless of lesion size.
- Global Contextual Branch (CG-MViT): Employs a Conv-Guided Mamba Vision Transformer. Unlike standard Transformers that rely on self-attention, this branch uses a Mamba State Space Model with selective state space mechanisms to capture long-range dependencies with linear computational complexity. A lightweight convolutional embedding layer precedes the Mamba layers to maintain local spatial consistency and provide inductive bias for anatomical boundaries before tokenization.
Hybrid Feature Fusion (DMCCA): The feature vectors from the SPP-DenseNet121 and CG-MViT branches are fused using Discriminative Multiple Canonical Correlation Analysis (DMCCA). This method projects heterogeneous feature spaces into a unified, discriminative subspace, maximizing correlations between the two sets while minimizing redundancy.
Online Retrieval and Refinement:
- Initial Retrieval: A query image is processed through the same dual-branch pipeline. Similarity is computed using a composite metric combining Euclidean distance and Cosine similarity.
- Query Expansion via Pseudo-Relevance Feedback (PRF): To bridge the semantic gap, the system assumes the top-ranked retrieved images are relevant. The average feature vector of these top- $P$ images is computed and used to expand the original query vector.
- Re-ranking: The expanded query vector is compared against the database again using the composite similarity metric to generate a final, refined ranking of results.

Key Contributions

Novel CBMIR Paradigm: Introduces a framework specifically designed for endoscopic bladder tissue retrieval, facilitating computer-aided decision-making.
Dual-Stream Architecture: Combines SPP-DenseNet121 for scale-invariant local features and CG-MViT for efficient global context learning, addressing the limitations of using either architecture alone.
SPP Integration: Utilizes Spatial Pyramid Pooling within DenseNet121 to effectively represent abnormalities of varying sizes and structures common in endoscopic procedures.
Efficient Global Modeling: Implements CG-MViT to learn global semantic and spatial relations with reduced computational overhead compared to traditional Transformer models.
Discriminative Fusion: Proposes DMCCA for fusing heterogeneous features, creating a redundant-free, discriminative feature space.
Semantic Gap Reduction: Integrates PRF-based query expansion to refine query representation, reducing ambiguity and aligning visual features with clinical semantics.
Comprehensive Validation: Validates the framework through extensive ablation studies and qualitative assessments, confirming the contribution of each component.

Experimental Results
The framework was evaluated on two publicly available datasets: the Kvasir gastrointestinal endoscopic dataset and the Endoscopic Bladder Tissue dataset.

Kvasir Dataset: The proposed model achieved Mean Average Precision (mAP) scores of 97.72% (P@5), 96.81% (P@10), 95.78% (P@20), 93.15% (P@50), and 91.02% (P@100). Class-wise analysis showed high precision across all eight categories, with "Ulcerative-Colitis" and "Dyed-Lifted-Polyps" performing particularly well.
Endoscopic Bladder Tissue Dataset: The model attained mAP scores of 92.60% (P@5), 90.68% (P@10), 84.67% (P@20), 70.79% (P@50), and 66.82% (P@100). High-grade cancer (HGC) and Low-grade cancer (LGC) classes demonstrated superior retrieval precision compared to non-tumor classes.
Ablation Studies: Experiments confirmed that the full framework outperforms individual components (SPP-DenseNet121 alone, CG-MViT alone) and intermediate configurations (e.g., fusion without DMCCA or without PRF), validating the necessity of the hybrid approach and query refinement.

Significance and Claims
The authors claim that the proposed framework offers high robustness and generalizability for clinical image retrieval. By effectively combining local multi-scale features with global contextual dependencies and refining queries through feedback, the system successfully narrows the semantic gap between visual appearance and medical interpretation. The results suggest the framework is a viable solution for computer-aided diagnosis, case-based reasoning, and clinical decision support. The authors note that while the current results are promising, future work could explore self-supervised training, lightweight architectures for real-time hospital systems, and expansion to other modalities like MRI, CT, and histopathology.

More like this