💻 computer science

MFil-Mamba: Multi-Filter Scanning for Spatial Redundancy-Aware Visual State Space Models

MFil-Mamba is a novel visual state space architecture that addresses the limitations of existing SSM-based vision models by employing a multi-filter scanning backbone and adaptive weighting mechanism to capture unique spatial information with minimal redundancy, thereby achieving state-of-the-art performance across diverse computer vision benchmarks.

Original authors: Puskal Khadka, KC Santosh

Published 2026-03-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Puskal Khadka, KC Santosh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to "see" and understand the world, like recognizing a cat in a photo or finding a stop sign on a busy street.

For a long time, we used two main ways to teach robots:

The "Local" Way (CNNs): Like looking at a photo through a tiny keyhole. You see a small patch clearly, then move the keyhole a little bit to see the next patch. It's great for details but slow to connect the dots across the whole picture.
The "Global" Way (Transformers): Like having a super-powerful brain that looks at the entire photo at once and instantly understands how every pixel relates to every other pixel. This is powerful, but it's incredibly expensive and slow, like trying to read every book in a library simultaneously.

Recently, a new technology called Mamba came along. It's like a super-efficient reader that can read a long story very quickly (linear speed) while still remembering the context. The problem? Mamba was built for reading text (which is a straight line), but images are 2D grids (like a chessboard).

The Old Solution: The "Zig-Zag" Problem

To make Mamba read images, previous researchers tried to force the 2D image into a 1D line. They did this by scanning the image in specific patterns: left-to-right, top-to-bottom, or even in a spiral or a "Z" shape.

The Analogy: Imagine you have a jigsaw puzzle. To show it to a friend who only understands a single line of description, you try to describe it by tracing a specific path over the pieces.

The Problem: If you trace a "Z" pattern, you might describe the top-left corner, then jump to the bottom-right, then back to the middle. You lose the natural flow of the picture. You end up describing the same relationships multiple times (redundancy) and distorting how the pieces actually fit together. It's like trying to describe a painting by only looking at it through a twisted straw.

The New Solution: MFil-Mamba

The authors of this paper, Puskal Khadka and KC Santosh, said, "Why force the image into a line at all? Let's look at it differently."

They created MFil-Mamba (Multi-Filter Mamba). Here is how it works, using simple metaphors:

1. The "Multi-Lens Camera" (Multi-Filter Scanning)

Instead of forcing the image into a single, twisted line, MFil-Mamba looks at the image through four different "lenses" or filters at the same time:

Lens 1 (The Original): Just looks at the picture as is.
Lens 2 (The Horizontal Edge): Highlights horizontal lines (like the horizon or a table).
Lens 3 (The Vertical Edge): Highlights vertical lines (like a tree trunk or a building).
Lens 4 (The Smart Filter): A special lens that learns to find unique patterns specific to the task (like the curve of a cat's ear).

The Analogy: Imagine you are trying to understand a complex machine.

Old Way: You walk around it in a zig-zag pattern, trying to describe it piece by piece in a specific order.
MFil-Mamba Way: You put on four different pairs of glasses at once. One pair highlights the bolts, one highlights the wires, one highlights the casing, and one highlights the moving parts. You get a rich, complete picture of how the parts relate to each other without having to force them into a single line.

2. The "Smart Mixer" (Adaptive Weighting)

Once the robot looks through these four lenses, it has four different versions of the image. How does it combine them?

Old Way: It just averages them out (50% this, 50% that).
MFil-Mamba Way: It has a Smart Mixer. It learns to say, "For this specific part of the image, the 'Vertical Edge' lens is most important, so I'll listen to that one more. For this other part, the 'Original' lens is best."
The Analogy: It's like a DJ mixing music. Instead of playing four songs at the exact same volume, the DJ (the AI) knows when to turn up the bass (vertical lines) and when to turn up the vocals (original details) to create the perfect track.

Why is this a big deal?

The paper shows that this new method is faster, cheaper, and smarter than the previous "zig-zag" methods.

Better Accuracy: On the famous ImageNet test (a giant photo quiz), their tiny model got an 83.2% score, beating many larger, more expensive models.
Better Vision: When tested on finding objects (like cars in a parking lot) or splitting a scene into parts (like separating a road from a sidewalk), it performed better than the current state-of-the-art models.
No Distortion: Because it doesn't force the image into a weird line, it understands the "shape" of the world much better. The "Effective Receptive Field" (how much of the image the robot can "see" at once) is smooth and natural, not jagged and artificial like the old methods.

The Bottom Line

MFil-Mamba is like upgrading a robot's vision from a "tunnel vision" approach (looking at a picture one twisted line at a time) to a "multi-perspective" approach (looking at the picture through specialized lenses that understand edges, shapes, and context simultaneously).

It proves that you don't need to force square pegs (images) into round holes (text-based AI) to make them work. Instead, you can build a new tool that respects the natural 2D shape of the world, making AI faster, more efficient, and better at seeing what's really there.

1. Problem Statement

While State Space Models (SSMs), particularly the Mamba architecture, have achieved remarkable success in 1D sequence modeling (e.g., NLP) due to their linear time complexity, adapting them to Computer Vision remains challenging.

The Core Issue: Visual data is inherently 2D with complex spatial dependencies, lacking the natural sequential order of text.
Limitations of Existing Approaches: Current Vision Mamba variants (e.g., VMamba, ViM) attempt to linearize 2D images into 1D sequences using predefined directional traversal strategies (e.g., bidirectional, four-way cross-scanning, Hilbert curves, zigzag).
Consequences: These fixed traversal methods introduce information redundancy, distort original spatial relationships, and fail to capture diverse spatial dependencies efficiently. They often create artificial sequential orders that do not align with the intrinsic structure of visual features.

2. Methodology: MFil-Mamba

The authors propose MFil-Mamba, a novel architecture that replaces rigid directional traversal with a Multi-Filter Scanning strategy. Instead of reordering pixels, the model extracts complementary spatial information using dynamic filters before processing them through the SSM.

Key Architectural Components:

Multi-Filter Scanning Strategy:
- Instead of traversing the image in specific directions, the input feature map is processed through four distinct representations:
  - Original Input ( $I$ ): Preserves raw visual cues.
  - Horizontal Filter ( $F_h$ ): Captures vertical edge information using Sobel-based kernels.
  - Vertical Filter ( $F_v$ ): Captures horizontal edge information.
  - Dynamic Filter ( $F_{k,l,t}$ ): A learnable depthwise separable convolution block that adapts to task-specific high-level patterns during training.
- These four maps are concatenated to form a rich, multi-perspective feature representation ( $F_{concat}$ ) without imposing an explicit traversal order.
Selective State Space Module (MFil-SSM):
- The concatenated features are fed into a Selective SSM block. This allows the model to capture diverse spatial dependencies in parallel, leveraging the efficiency of Mamba while operating on a structurally enriched 2D representation.
Adaptive Weighting Mechanism:
- To fuse the outputs from the different filtered scans, the authors introduce a lightweight adaptive weighting module.
- It uses trainable scalar weights ( $\alpha_i$ ) computed via Softmax to dynamically aggregate the filtered outputs. This allows the network to emphasize the most informative spatial features for a given context while suppressing less relevant ones.
Hierarchical Architecture:
- The model follows a four-stage hierarchical design (similar to Swin Transformer) with downsampling layers between stages.
- It replaces the standard MLP in the feed-forward network with a Convolutional Feed-Forward Network (ConvFFN) to better capture local spatial cues.

Theoretical Justification

The paper provides a theoretical analysis showing that directional traversal merely permutes the spatial covariance matrix ( $\Sigma_{ij} = P_i \Sigma_x P_j^T$ ), which introduces redundancy without capturing new dependencies. In contrast, the multi-filter approach projects inputs into orthogonal and adaptive subspaces ( $\Sigma_{ij} = F_i \Sigma_x F_j^T$ ), effectively reweighting and reorienting the underlying covariance to capture distinct spatial dependencies without distortion.

3. Key Contributions

Novel Scanning Paradigm: Introduction of MFil-Mamba, which abandons fixed directional traversal in favor of a multi-filter strategy that extracts complementary structural and contextual cues directly from 2D feature maps.
Adaptive Fusion: Development of a lightweight adaptive weighting mechanism that dynamically fuses multi-filter outputs, enhancing spatial feature integration.
State-of-the-Art Performance: Demonstration of superior performance across multiple vision tasks (classification, detection, segmentation) with three model variants (Tiny, Small, Base) that outperform existing Transformer and SSM-based models.

4. Experimental Results

The model was evaluated on standard benchmarks: ImageNet-1K (Classification), MS COCO (Object Detection & Instance Segmentation), and ADE20K (Semantic Segmentation).

Image Classification (ImageNet-1K):
- MFil-Mamba-T (Tiny): Achieved 83.2% Top-1 accuracy, outperforming Swin-T (81.3%), VMamba-T (82.6%), and ConvNeXt-T (82.1%).
- MFil-Mamba-B (Base): Achieved 84.2% Top-1 accuracy, surpassing Swin-B (83.5%) and VMamba-B (83.9%).
Object Detection & Instance Segmentation (MS COCO):
- MFil-Mamba-T: Achieved 47.3% Box AP and 42.7% Mask AP, significantly outperforming Swin-T and PlainMamba-L1.
- MFil-Mamba-B: Achieved 49.0% Box AP and 43.8% Mask AP.
Semantic Segmentation (ADE20K):
- MFil-Mamba-T: Achieved 48.5% mIoU (Single Scale), outperforming Swin-T (44.4%) and VMamba-T (47.9%).
Ablation Studies:
- Confirmed that replacing single-directional scan with multi-filter scanning improved accuracy by ~0.43%.
- Showed that adaptive weighting provided a further ~0.12% gain.
- Demonstrated that simply increasing model depth/width (state dimension) does not guarantee better performance; balanced hyperparameter coordination is crucial.

5. Significance and Impact

Efficiency vs. Performance: MFil-Mamba achieves linear complexity (like Mamba) while overcoming the spatial distortion issues of previous Vision Mamba models, offering a more efficient alternative to the quadratic complexity of Vision Transformers (ViTs).
Structural Coherence: By avoiding arbitrary pixel reordering, the model preserves the intrinsic 2D spatial relationships of images, leading to better Effective Receptive Field (ERF) characteristics as visualized in the paper.
Generalizability: The architecture serves as a versatile backbone for diverse downstream tasks, proving that SSMs can be effectively adapted to vision without relying on complex, fixed traversal heuristics.
Future Direction: The work opens new avenues for SSM research in vision, suggesting that filter-based representation learning is a more promising path than direction-based sequentialization.

In summary, MFil-Mamba represents a significant step forward in Vision SSMs by shifting the paradigm from "how to scan" (directional traversal) to "what to extract" (multi-filter representation), resulting in state-of-the-art performance with high efficiency.