🤖 AI

Focal Modulation and Bidirectional Feature Fusion Network for Medical Image Segmentation

The paper proposes FM-BFF-Net, a hybrid medical image segmentation network that integrates convolutional and transformer components with focal modulation and bidirectional feature fusion to effectively capture global context and enhance boundary precision, achieving state-of-the-art performance across eight diverse medical imaging datasets.

Original authors: Moin Safdar, Shahzaib Iqbal, Mubeen Ghafoor, Tariq M. Khan, Imran Razzak, Thantrira Porntaveetus, Hamid Alinejad-Rokny

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Moin Safdar, Shahzaib Iqbal, Mubeen Ghafoor, Tariq M. Khan, Imran Razzak, Thantrira Porntaveetus, Hamid Alinejad-Rokny

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Super-Scanner" for Medical Images

Imagine a doctor looking at an X-ray, an ultrasound, or a skin photo trying to find a specific problem, like a polyp in the colon or a tumor in the breast. To do this accurately, they need to draw a perfect line around the problem area. This is called medical image segmentation.

For a long time, computers have tried to do this using "Convolutional Neural Networks" (CNNs). Think of these as a magnifying glass. A magnifying glass is great at seeing tiny details right in front of it (local features), but it struggles to see the whole picture at once. It might miss how a small spot connects to a larger shape or fail to understand the context of the whole image.

On the other hand, there are newer models called "Transformers" that act like a wide-angle drone. They can see the whole landscape and understand how everything connects (global context), but they sometimes miss the tiny, fine details needed to draw a perfect edge.

The Problem: Medical images are tricky. Lesions (problems) come in all shapes, sizes, and contrasts. Sometimes they blend into the background. The "magnifying glass" models get lost in the details, and the "drone" models get lost in the big picture.

The Solution: The authors of this paper built a new system called FM-BFF-Net. Think of it as a hybrid vehicle that combines the best of both worlds: the sharp focus of a magnifying glass and the broad view of a drone.

How It Works: The Three Secret Ingredients

The paper describes three main "gadgets" inside this new system that make it work so well:

1. The "Smart Spotlight" (Focal Modulation)

The Analogy: Imagine you are in a dark room looking for a specific object. A regular camera takes a picture of the whole room, but everything looks a bit flat. A "Smart Spotlight" shines a beam exactly where the object is, adjusting its brightness based on how important that spot is.
In the Paper: This is called the Focal Modulation-based ConvFormer Attention Block (FMCAB). It looks at the image and says, "Hey, this specific area is crucial! Let's focus our attention there and ignore the noise." It helps the computer refine the details so it doesn't get confused by similar-looking textures nearby.

2. The "Two-Way Highway" (Bidirectional Feature Fusion)

The Analogy: Imagine a construction crew building a house. The "Encoder" team is digging the foundation and gathering raw materials (looking at the image from a distance). The "Decoder" team is painting the walls and adding the finishing touches (drawing the final outline). Usually, the Decoder team only gets a one-way memo from the Encoder.
In the Paper: This is the Bidirectional Feature Fusion Module (BiFFM). It builds a two-way highway between the digging team and the painting team. They constantly talk to each other. The painting team says, "I need more detail from the foundation here," and the digging team says, "Here is the raw data you need." This ensures the final outline is perfect because the "big picture" and the "tiny details" are constantly being mixed together.

3. The "Global Brain" (Vision Transformer)

The Analogy: Think of this as the project manager sitting in the middle of the construction site. While the workers are focused on their specific tasks, the manager looks at the entire blueprint to make sure the house makes sense as a whole.
In the Paper: This is the Vision Transformer (ViT) placed in the middle of the network. It uses a special "self-attention" mechanism to look at the entire image at once. It helps the system understand long-range connections, like realizing that a small bump on the left side of the image is actually part of a large shape on the right side.

The Results: Did It Work?

The authors tested this new "hybrid vehicle" on eight different datasets (collections of medical images). They looked at:

Polyps (growths in the colon)
Skin Lesions (moles or tumors on the skin)
Ultrasounds (breast lumps and thyroid nodules)

They compared their new system against the current "best" methods (State-of-the-Art).

The Verdict:
The paper claims that FM-BFF-Net consistently beat the competition.

It was better at drawing the exact edges of the problems (boundary precision).
It handled weird shapes and sizes better than the old models.
It worked well even when the images were blurry or had low contrast (like trying to see a shadow against a dark wall).

In simple terms: If the old models were like a student who got 85% on a test, this new model got 95% or higher, especially on the hardest questions.

The Limitations (The "But...")

The authors are honest about where the system still struggles.

The "Ghost" Problem: If a lesion has extremely low contrast (meaning it looks almost exactly the same color as the healthy tissue around it), the system sometimes still has trouble drawing the perfect line. It's like trying to find a white cat in a snowstorm; even the best eyes can struggle.
The Claim: The paper admits that while it is better than everyone else, it isn't perfect in these specific "ghostly" situations.

Summary

The paper presents a new tool for medical image analysis that mixes the "zoom-in" power of traditional computers with the "zoom-out" power of modern AI. By using a Smart Spotlight to focus on details, a Two-Way Highway to share information, and a Global Brain to understand the whole picture, it creates a more accurate map of medical problems than previous tools. It has been proven to work better on polyps, skin issues, and ultrasound scans, though it still finds it hard to see things that blend perfectly into the background.

1. Problem Statement

Medical image segmentation is critical for clinical tasks such as diagnosis, treatment planning, and monitoring. However, existing methods face significant challenges:

Limitations of CNNs: Traditional Convolutional Neural Networks (CNNs) excel at capturing local spatial features but struggle with global context and long-range dependencies due to their localized receptive fields. This leads to difficulties in segmenting structures with complex borders, varying sizes, and low contrast.
Limitations of Transformers: While Vision Transformers (ViTs) effectively capture global context via self-attention, they often lack the ability to preserve fine-grained local spatial details essential for precise boundary delineation.
Clinical Variability: Medical images (MRI, CT, Ultrasound, etc.) exhibit high variability in image quality, contrast, and anatomical structures, making it difficult for models to generalize across different scales and modalities.

2. Methodology: M-BFF-Net Architecture

The authors propose a hybrid deep learning framework that integrates the strengths of CNNs (local feature extraction) and Transformers (global context awareness). The architecture follows an encoder-decoder structure with the following key components:

A. Backbone and Encoder

Backbone: Uses EfficientNetV2S1 as the backbone to maximize computational efficiency.
Encoder Blocks: Comprises four encoder blocks ( $B_{enc}^{1}$ to $B_{enc}^{4}$ ) that extract hierarchical features.
Skip Connections: Aggregated skip connections are established between the encoder and decoder to preserve spatial information.

B. Focal Modulation-based ConvFormer Attention Block (FMCAB)

Inserted between encoder blocks and skip connections, this module refines feature flow:

Mechanism: It combines convolutional operations with a focal modulation attention mechanism.
Function: It uses Global Average Pooling (GAP) and Global Max Pooling (GMP) to compute attention weights, allowing the model to dynamically focus on crucial regions while adaptively modulating context.
Goal: Enhances the integration of local and global contexts, improving the model's ability to handle varying lesion sizes and shapes.

C. Vision Transformer Module (ViTM)

Placed at the bottleneck layer (between encoder and decoder):

Components: Integrates Token-based Self-Attention (TSA) and Global Spatial Attention (GSA).
Function: TSA captures long-range dependencies using Query, Key, and Value matrices, while GSA captures global position dependencies. This allows the model to understand relationships across the entire image, addressing the limitations of local CNN operations.

D. Bidirectional Feature Fusion Module (BiFFM)

Located in the decoder stage to merge features from skip connections and the reconstructed decoder features:

Mechanism: Takes inputs from the encoder skip connection ( $s_n$ ) and the decoder branch. It processes these through parallel pathways involving $1\times3$ and $3\times1$ convolutions and Channel Shuffling (CS).
Function: Facilitates efficient bidirectional interaction between encoder and decoder representations. It harmonizes local details (from skip connections) with global semantic information (from the decoder), ensuring precise boundary reconstruction.

E. Decoder

Utilizes Feature Refinement Modules (FRM) and the BiFFM to progressively upsample and fuse features.
The final output is generated via a $1\times1$ convolution followed by a sigmoid activation to produce the segmentation mask.

3. Key Contributions

Hybrid Architecture: Successfully combines CNNs and Transformers to balance local feature extraction with global context awareness.
FMCAB: Introduces a novel attention block that modulates feature flow, enhancing the model's adaptability to complex anatomical structures.
BiFFM: Develops a bidirectional fusion module that enriches context at every decoder stage, significantly improving boundary precision.
Efficiency: Employs EfficientNetV2S1 as a backbone to ensure the model remains computationally efficient, making it suitable for resource-constrained clinical environments.

4. Experimental Results

The model was evaluated on eight public datasets across three medical imaging domains:

Polyp Segmentation: Kvasir-SEG, CVC-ClinicDB, CVC-ColonDB.
Skin Lesion Segmentation: ISIC2016, ISIC2017, ISIC2018.
Ultrasound Segmentation: BUSI (Breast), DDTI (Thyroid).

Performance Metrics:
The model was compared against State-of-the-Art (SOTA) methods including U-Net, UNet++, Swin-Unet, Meta-Polyp, and others.

Polyp Segmentation: M-BFF-Net achieved a Jaccard Index of 92.96% on Kvasir-SEG, outperforming the next best method (Meta-Polyp at 92.10%) and showing improvements of up to 34.54% on the CVC-300 dataset.
Skin Lesion Segmentation: Consistently outperformed competitors on all ISIC datasets, with Jaccard improvements ranging from 0.05% to 11.61%.
Ultrasound Segmentation: Achieved superior results on BUSI and DDTI datasets, with Jaccard improvements of 0.64% to 11.67% over existing methods.
Visual Quality: Qualitative analysis showed M-BFF-Net produced masks that closely aligned with Ground Truth (GT), particularly in cases with low contrast, irregular shapes, and occlusions.

5. Significance and Conclusion

Clinical Impact: By improving segmentation accuracy and boundary precision, the model aids in more reliable disease diagnosis and treatment planning (e.g., tumor border delineation).
Robustness: The hybrid approach demonstrates high adaptability across different imaging modalities (endoscopy, dermoscopy, ultrasound) and varying lesion characteristics.
Future Work: The authors suggest extending the model to 3D volumetric data and multimodal imaging, as well as integrating uncertainty quantification to provide confidence estimates for clinical decision-making.

In summary, M-BFF-Net represents a significant advancement in medical image segmentation by effectively bridging the gap between local feature extraction and global context modeling, resulting in a robust, high-accuracy solution for diverse clinical applications.