🤖 AI

Norm or Direction? Decoding Vision Mambas for High-Resolution Vision

This paper reveals that while MambaOut encodes class-discriminative information primarily in the magnitude of foreground tokens, VMamba uniquely distributes semantic evidence across token directions and background regions, a distinct encoding strategy that grants it superior stability and performance in high-resolution dense prediction tasks.

Original authors: Jin Yu, Juyoun Park

Published 2026-07-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jin Yu, Juyoun Park

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize a cat in a picture. For a long time, the best way to do this was to have the robot look at every single pixel and compare it to every other pixel, like a detective checking every clue against every other clue. This is powerful but slow, especially for huge, high-definition photos. Recently, scientists invented a new, faster way called "Vision Mamba." Think of it like a robot that reads a picture like a book, scanning it line by line to understand the story without getting overwhelmed. But then, another team built a robot that doesn't read the book at all; instead, it uses a different kind of "gated" filter to spot the cat. Surprisingly, this new robot is just as good at finding cats in standard-sized photos. This leaves scientists with a big mystery: If both robots get the job done, are they actually seeing the world in the same way? Or is one of them using a secret superpower that the other is missing? This question matters because as we move from small photos to massive, detailed images (like satellite maps or medical scans), the way these robots "see" might determine if they can handle the job or if they start to get confused.

This paper, titled "Norm or Direction? Decoding Vision Mambas for High-Resolution Vision," dives into that mystery to see how two specific robot brains—VMamba and MambaOut—actually process information. The researchers discovered that while both models are great at spotting objects, they store the "clues" about what they see in completely different ways.

Think of a token (a tiny piece of the image the robot analyzes) as a little messenger carrying a note. Every messenger has two things: how loud they shout (their "magnitude" or strength) and the direction they are pointing (their "direction"). The study found that MambaOut is like a team of shouters. It concentrates all its important information into a few very loud, high-energy messengers that stand right on top of the object (the cat). If you silence the quiet messengers in the background, MambaOut is still fine because the loud ones are doing all the work.

VMamba, on the other hand, is more like a choir. It doesn't rely on a few loud shouters. Instead, it spreads the important information out across the whole room, including the background. The loud messengers in VMamba's team often end up standing in the background (like the sky or grass), not on the cat itself. If you just look at who is shouting the loudest, you might think VMamba is confused. But here is the twist: the direction the messengers are pointing holds the secret. Even if you turn down the volume on all the messengers so they are all whispering, VMamba can still tell you it's a cat because the direction of their whispers is perfectly aligned. MambaOut, however, falls apart if you turn down the volume; it needs the shouting to work.

The researchers tested this by making the robots look at much bigger, higher-resolution pictures. When the image gets huge, there are thousands more messengers to manage. MambaOut, which relies on a few loud shouters, starts to stumble because it's hard to pick the right few loud ones out of a crowd of thousands. VMamba, with its choir of directional whispers, handles the crowd much better. It spreads the evidence out, making it more stable and reliable as the picture gets bigger.

This difference becomes even clearer when the robots are asked to do a harder job: not just say "there is a cat," but point out exactly where the cat is in the picture (a task called segmentation). When the researchers let the robots learn from scratch on these detailed tasks, VMamba pulled ahead. It turned out that VMamba's ability to reorganize its "directional" clues made it much easier to teach how to point at specific parts of an image. MambaOut, which relies on the loudness of specific spots, was harder to teach for this detailed work.

The paper suggests that the secret to building better robots for high-resolution tasks isn't just about the scanning mechanism (the "Mamba" part) but about how they organize their information. It seems that for big, detailed jobs, relying on the "direction" of the data is more powerful than relying on the "loudness" (magnitude). The authors propose that future robot brains should be designed to pay more attention to these directional clues, perhaps by training them to be better at organizing information across the whole image rather than just focusing on the loudest spots. While the paper doesn't claim to have solved everything (they admit they aren't 100% sure which specific part of the robot's brain causes this background noise), the evidence strongly suggests that shifting focus from "how loud" to "which way" is the key to unlocking better vision for high-resolution images.

Technical Summary: Norm or Direction? Decoding Vision Mambas for High-Resolution Vision

Problem Statement

Vision Mamba models, such as VMamba, have emerged as efficient visual backbones by replacing quadratic self-attention with linear-complexity Selective State Space Models (SSMs). However, recent work (MambaOut) demonstrates that Gated CNN blocks can match or exceed VMamba's performance on standard image classification, challenging the necessity of SSMs for vision tasks. This creates a fundamental ambiguity: do VMamba and MambaOut encode visual information differently at the representation level, and if so, how do these differences impact performance on high-resolution and dense prediction tasks (e.g., semantic segmentation)? While MambaOut argues that SSMs are unnecessary for classification (a non-autoregressive, short-sequence task), VMamba retains advantages on dense prediction benchmarks where spatial sequences are longer. The core problem is to decode the internal representational strategies of these models to understand the source of VMamba's advantage in high-resolution settings.

Methodology

The authors conduct a comparative analysis between VMamba-T and MambaOut-T (Tiny variants) using a multi-faceted approach:

Cross-Model and Within-Model CKA Analysis: The authors apply Centered Kernel Alignment (CKA) to measure similarity between representations across different stages, between the two models, and between classification-pretrained and segmentation-fine-tuned versions.
Token Decomposition (Magnitude vs. Direction): Spatial tokens are decomposed into $\ell_2$ norm (magnitude) and unit vectors (direction). The study investigates where high activations occur and whether class-discriminative information is encoded in magnitude or direction.
Attribution and Localization:
- Grad-CAM: Used to compare token activation maps with class attribution heatmaps.
- Token Replacement Attribution: A perturbation-based method where individual tokens are replaced with the image-wise mean to measure their contribution to the target class logit.
- Unit-Vector Evaluation: Classifiers are tested on normalized (unit-norm) tokens to isolate the information contained purely in token directions.
- High-Norm Token Testing: Classifiers are trained or evaluated using only the top-k highest-norm tokens to test if magnitude alone carries class information.
Task-Specific Evaluation: The models are evaluated on:
- High-Resolution Classification: Fine-tuning on ImageNet-1k at 768×768 resolution.
- Semantic Segmentation: Training on ADE20K and Pascal VOC 2012 under two regimes: frozen backbone and full fine-tuning.
- Ablation: Removing the SS2D module to test its specific contribution to the observed phenomena.

Key Findings

1. Divergent Representational Strategies

CKA Divergence: VMamba's final stage blocks form representations that are qualitatively distinct from both MambaOut and their own preceding blocks. In contrast, MambaOut maintains consistent representations across layers. VMamba also shows greater sensitivity to supervision changes (classification vs. segmentation) at the final stage.
Magnitude vs. Direction:
- MambaOut: Concentrates class-discriminative information in a small set of high-norm foreground tokens. These tokens align spatially with Grad-CAM attributions. Removing magnitude (normalizing tokens) causes a sharp drop in performance (single-token accuracy drops to ~31.5%).
- VMamba: Produces high-norm tokens predominantly in background regions, which are misaligned with Grad-CAM foreground focus. However, discriminative signals are preserved primarily in token directions. Normalizing tokens results in a much smaller performance drop (single-token accuracy remains ~52.5%, significantly higher than MambaOut).

2. Evidence Aggregation and Resolution

Token Replacement: VMamba distributes logit support broadly across object regions, whereas MambaOut relies on sparse, dominant tokens.
High-Resolution Impact: At standard resolution (224×224), MambaOut slightly outperforms VMamba. However, at high resolution (768×768), VMamba surpasses MambaOut. The authors attribute this to the stability of VMamba's broad evidence aggregation, which is less sensitive to spatial sampling variations as token counts increase, compared to MambaOut's reliance on sparse peaks.

3. Dense Prediction and Fine-Tuning

Frozen Backbone: MambaOut performs slightly better on segmentation with a frozen backbone, likely because its magnitude-aligned foreground cues are immediately exploitable by a new head.
Full Fine-Tuning: VMamba significantly outperforms MambaOut when the backbone is fully fine-tuned.
Directional Robustness: In a "unit-token decoder" test (where magnitude is removed), VMamba's segmentation performance remains largely intact (47.9 $\to$ 47.5 mIoU), while MambaOut degrades sharply (47.1 $\to$ 41.5 mIoU).
Spatial Reorganization: Under dense supervision, VMamba's direction-based encoding reorganizes into clearer spatial selectivity, whereas MambaOut's magnitude-concentrated encoding shows little change.

Key Contributions

Decoding Encoding Strategies: The paper identifies that VMamba and MambaOut rely on fundamentally different encoding strategies: MambaOut uses magnitude-aligned foreground concentration, while VMamba utilizes direction-based distribution across spatial tokens.
Explanation of High-Resolution Advantage: The authors demonstrate that VMamba's advantage in dense prediction and high-resolution tasks stems not merely from the SSM mechanism or sequence length, but from how semantic evidence is organized across token magnitude and direction.
Role of Token Direction: The study establishes that token direction serves as a critical, often overlooked, axis for visual representation, particularly for dense supervision tasks where information must be aggregated stably across many tokens.

Significance and Claims

The paper concludes that the success of Vision Mamba models in high-resolution and dense prediction settings is rooted in their directional encoding strategy. This strategy allows for:

Stable Aggregation: More robust evidence aggregation as the number of tokens increases (high resolution).
Adaptability: Greater capacity to reorganize spatial selectivity under dense supervision (segmentation).

The authors suggest that future visual backbone designs should consider token magnitude and directional structure as critical axes. Specifically, they propose that foreground-aligned magnitude regularization could improve standard classification, while angular regularization or token-wise contrastive objectives could enhance performance in dense and high-resolution settings.

The paper remains modest regarding the specific architectural components responsible, noting that while the SS2D module is a candidate, the phenomenon persists even when SS2D is removed, suggesting the effect arises from the broader block structure rather than the selective scan mechanism alone.