💻 computer science

Vision Non-Causal Trapezoidal Mamba: Eliminating Directional Scanning in Vision SSMs with Second-Order Dynamics

The paper introduces Vision Non-Causal Trapezoidal Mamba (VNCT), a second-order non-causal State Space Model that eliminates directional token scanning to achieve orientation-robust representations and superior performance across various vision tasks while maintaining low inference latency.

Original authors: Anvitha Ramachandran, Dhruv Parikh, Haoyang Fan, Rajgopal Kannan, Viktor Prasanna

Published 2026-07-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anvitha Ramachandran, Dhruv Parikh, Haoyang Fan, Rajgopal Kannan, Viktor Prasanna

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to understand a giant, complex puzzle made of thousands of tiny picture pieces (tokens). For a long time, computer vision models acted like a very strict, one-way street. They had to look at the pieces in a specific order: left to right, top to bottom, or in some other fixed path. This is called directional scanning. It's like reading a book where you can't look ahead or flip back; you have to read every word in a line before moving to the next.

The paper introduces a new model called Vision Non-Causal Trapezoidal Mamba (VNCT). Its main finding is that this strict "one-way street" rule is actually unnecessary for seeing well. In fact, getting rid of it makes the model faster and smarter.

The Old Way vs. The New Way

Think of the old models (like many Vision Transformers or earlier "Mamba" models) as a team of detectives who have to walk through a museum room in a single file line. They can only look at the painting in front of them, then the next one, and so on. If they miss a detail on the left because they were focused on the right, they might not catch it until they circle back around, which takes extra time. This creates a bias; the model gets "used" to looking in a specific direction.

VNCT throws out the rulebook. Instead of walking in a line, it lets every single picture piece talk to every other piece all at once in a single pass. It's like turning on a giant, instant spotlight that illuminates the entire museum room simultaneously. Every detective can see every painting instantly, without waiting for a turn. The paper shows that this "all-at-once" approach eliminates the weird biases caused by walking in a line.

The Secret Sauce: Second-Order Dynamics

But how does it talk to everyone so fast without getting confused? The authors use a clever math trick called second-order dynamics (specifically, "trapezoidal" dynamics).

Imagine you are trying to guess the temperature of a room.

First-order (the old way): You check the thermometer once and guess based on that single reading.
Second-order (VNCT's way): You check the thermometer, but you also look at how fast the temperature was changing just a split second ago. You take an average of the "now" and the "just before."

This "trapezoidal" method (averaging the start and end points of a step) gives the model a richer, more detailed understanding of the image. The paper suggests that this extra layer of detail helps the model understand shapes and boundaries much better than the simpler, single-step methods used by previous models.

What the Paper Rules Out

The paper explicitly argues against the idea that vision models need directional scanning to work well. Many previous models tried to fix the "one-way street" problem by adding more complex paths (like walking in a zig-zag or a spiral), but the authors show that you don't need to fix the path at all—you just need to remove the path entirely. They rule out the necessity of sequential scanning for high-performance vision.

The Results: Faster, Stronger, and More Robust

The authors measured their model on several big challenges, and the results were quite clear:

Speed: Because it doesn't have to wait for a line to form, VNCT is faster. On a single image, it took 5.25 milliseconds for the "Micro" version and 7.31 milliseconds for the "Tiny" version to make a prediction. This is faster than the previous non-causal model (VSSD), which took 5.80 ms and 8.01 ms respectively.
Accuracy: It got better scores on standard tests. On the ImageNet-1K test, the "Tiny" version hit 84.2% accuracy, beating the previous best non-causal model (VSSD-Tiny) which got 83.7%.
Turning and Flipping: One of the coolest findings is about "orientation robustness." If you flip an image upside down or rotate it, the old models get a bit confused and their accuracy drops. VNCT is much more chill about it. When the image was rotated or flipped, the average accuracy drop was only 0.3%, compared to 0.9% for VSSD and 1.6% for the directional scanning model. This suggests the model sees the object as a whole, not just a sequence of pixels.
Boundaries: When drawing the outlines of objects (like separating a car from the road), VNCT was sharper. It improved the "Boundary IoU" (a score for how well the edges match) by up to 3.7 points on the Cityscapes dataset compared to VSSD.

The Bottom Line

The paper doesn't just suggest this might work; it measured it across classification, object detection, and segmentation tasks. The authors conclude that the "one-way street" of directional scanning is an artifact of trying to force sequence models onto images, and that a second-order, non-causal approach is a simpler, more efficient, and more robust way to build vision AI. They didn't just tweak the old system; they replaced the traffic rules entirely, and the cars are moving faster and driving straighter.

Technical Summary: Vision Non-Causal Trapezoidal Mamba (VNCT)

1. Problem Statement

State Space Models (SSMs) have emerged as efficient alternatives to Vision Transformers (ViTs) for modeling long-range dependencies with linear-time complexity. However, most existing vision SSMs (e.g., ViM, VMamba) inherit directional token scanning from causal sequence modeling. To apply 1D SSMs to 2D images, tokens must be serialized along predefined paths (horizontal, vertical, or multi-directional).

The authors identify three critical limitations of this approach:

Spatial Bias: Directional scanning imposes an artificial traversal order not intrinsic to visual data, creating orientation-sensitive feature interactions.
Inefficiency: Aggregating bidirectional context often requires multiple scan passes, increasing inference latency.
Misalignment: Visual scenes are non-causal; spatial relationships exist independently of token ordering.

While recent work like Vision State Space Duality (VSSD) introduced non-causal aggregation, it remains limited to first-order dynamics. The paper questions whether directional scanning is necessary for high-performance vision SSMs or if it is merely an artifact of adapting sequence models to images.

2. Methodology: Vision Non-Causal Trapezoidal Mamba (VNCT)

The authors propose VNCT, a hierarchical vision backbone built on a second-order non-causal state-space formulation. The core innovation is the NC-M3 (Non-Causal Mamba-3) layer.

2.1. Core Mechanism: NC-M3

The NC-M3 layer reformulates the exponential-trapezoidal dynamics of Mamba-3 into a global, non-causal aggregation operator.

Second-Order Dynamics: Unlike standard SSMs that use a single-step update, Mamba-3 utilizes a trapezoidal quadrature rule, averaging over both integration endpoints (left and right). This is expressed as:
$h_t = \alpha_t h_{t-1} + \beta_t x_{t-1} + \gamma_t x_t$
where $\beta$ and $\gamma$ weight the left- and right-endpoint observations.
Non-Causal Lift: The authors transform this causal recurrence into a permutation-invariant operator. Instead of sequential state propagation, they replace the lower-triangular causal transition matrix with a fully dense, normalized interaction kernel.
Global Aggregation: The interaction is computed via global sufficient statistics. For an output token $i$ $i$ , the state is derived from a global sum of all input tokens $j$ $j$ , weighted by data-dependent parameters ( $\alpha, \beta, \gamma$ $α, β, γ$ ) projected into an interaction space.
- The aggregation rule is factorized to avoid constructing an $L \times L$ interaction matrix, maintaining linear-time complexity ( $O(L)$ ).
- This allows all image tokens to interact in a single pass without directional scanning.

2.2. Architectural Components

Low-Rank MIMO Parameterization: VNCT incorporates a low-rank Multi-Input Multi-Output (MIMO) mechanism to jointly model spatial and channel-wise interactions efficiently.
2D Rotary Positional Encoding (RoPE): Since the aggregation is order-independent, spatial information is injected via 2D RoPE applied to the source and query projections, encoding relative spatial displacement.
Hierarchical Backbone: The model follows a four-stage architecture (similar to Swin or ConvNeXt) with progressive downsampling. Stages 1–3 utilize NC-M3 blocks, while Stage 4 employs Multi-Head Self-Attention (MHSA) on the lowest-resolution feature map to handle global context where quadratic cost is negligible.
Local Perception: A Local Perception Unit (LPU) using depth-wise convolutions injects local spatial inductive bias before global aggregation.

3. Key Contributions

NC-M3 Layer: A second-order non-causal state-space layer that eliminates directional scanning, enabling single-pass interactions among all image tokens.
VNCT Backbone: A hierarchical vision backbone combining second-order non-causal dynamics with low-rank MIMO parameterization.
Empirical Validation: Demonstration that eliminating directional scanning improves inference latency, orientation robustness, and boundary preservation, consistently outperforming both directional-scanning SSMs and first-order non-causal SSMs.

4. Experimental Results

The authors evaluated VNCT on ImageNet-1K (classification), COCO (object detection and instance segmentation), and ADE20K (semantic segmentation).

Classification (ImageNet-1K): VNCT achieves state-of-the-art Top-1 accuracy across Micro, Tiny, Small, and Base scales.
- Example: VNCT-Tiny achieves 84.2% Top-1 accuracy, outperforming VSSD-T (83.7%) and VMamba-T (82.5%).
- Efficiency: VNCT demonstrates lower single-image inference latency than VSSD at Micro, Tiny, and Small scales.
Dense Prediction (COCO & ADE20K):
- Detection/Segmentation: VNCT-T surpasses all baselines in Box AP and Mask AP on COCO, with significant gains at AP75 (indicating improved localization precision).
- Semantic Segmentation: VNCT achieves the highest mIoU on ADE20K across all model scales (e.g., VNCT-S achieves 50.1% mIoU vs. 49.2% for VSSD-S).
Boundary Preservation: VNCT significantly improves Boundary IoU (BIoU).
- On ADE20K, VNCT-T improves BIoU by +2.3 points over VSSD-T.
- On Cityscapes, the improvement is +3.7 points.
- Qualitative analysis shows VNCT produces more complete and geometrically accurate contours.
Orientation Robustness: VNCT exhibits superior robustness to geometric transformations (flips and rotations) without retraining.
- On ImageNet-1K, VNCT-T shows an average accuracy drop of only 0.3% across four transformations, compared to 0.9% for VSSD-T and 1.6% for VMamba-T.
Receptive Field Analysis: Effective Receptive Field (ERF) visualization reveals that VNCT produces a smooth, symmetric, and globally uniform receptive field, unlike the directional artifacts seen in VMamba or the irregular coverage of VSSD.

5. Significance and Claims

The paper argues that directional scanning is unnecessary for high-performance vision SSMs. The authors claim that the limitations of previous vision SSMs stem from the causal directional scan rather than state-space modeling itself.

By replacing causal scanning with second-order non-causal state-space modeling, VNCT offers a simple, efficient, and robust alternative for visual recognition. The results suggest that:

Global non-causal aggregation combined with second-order dynamics yields stronger spatial representations for dense prediction tasks.
Removing directional bias leads to more orientation-robust features and better boundary preservation.
High-performance vision backbones can be achieved without the computational overhead of multiple scan passes or the quadratic complexity of full attention mechanisms (except at the final stage).

The authors conclude that this approach motivates non-causal, higher-order state-space operators as a promising direction for future vision backbones.