💻 computer science

Manta: Enhancing Mamba for Few-Shot Action Recognition of Long Sub-Sequence

The paper proposes Manta, a framework that enhances Mamba for few-shot action recognition of long sub-sequences by integrating a Matryoshka Mamba architecture for hierarchical local and temporal feature modeling with a hybrid contrastive learning paradigm to mitigate intra-class variance, achieving state-of-the-art performance across multiple benchmarks.

Original authors: Wenbo Huang, Jinghui Zhang, Guang Li, Lei Zhang, Shuoyuan Wang, Fang Dong, Jiahui Jin, Takahiro Ogawa, Miki Haseyama

Published 2026-03-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenbo Huang, Jinghui Zhang, Guang Li, Lei Zhang, Shuoyuan Wang, Fang Dong, Jiahui Jin, Takahiro Ogawa, Miki Haseyama

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

🎬 The Big Problem: Watching a Movie in a Hurry

Imagine you are trying to teach a robot to recognize human actions, like "diving off a cliff."

The Old Way (Transformers): Most AI models today are like students who can only read one page of a book at a time. If you show them a whole movie scene (a long video), their brain gets overloaded, and they crash. They are forced to look at tiny, 8-second clips.
- The Flaw: If you only show the robot the 2 seconds where the person is falling, it might think the action is just "falling." It misses the context of the run-up or the splash. Also, if the robot sees two different people diving, it gets confused because one starts diving at second 2 and the other at second 5. It doesn't know how to line them up.
The New Hero (Mamba): Recently, a new type of AI called Mamba came out. It's like a super-fast reader that can read a whole book without getting tired. It's great at handling long videos.
- The Flaw: But Mamba is too "lazy" with details. It looks at the whole video and says, "Oh, this is a diving video," but it ignores the tiny, crucial moments (like the specific hand position) that actually define the action. It also doesn't know how to sync up two different videos that happen at different speeds.

🐙 The Solution: Meet "Manta"

The authors created a new framework called Manta (Matryoshka MAmba and CoNtrasTive LeArning). Think of Manta as a Russian Nesting Doll (Matryoshka) combined with a Detective.

Here is how it works, broken down into two main tricks:

1. The "Nesting Doll" Trick (Matryoshka Mamba)

Instead of looking at the whole video at once (too blurry) or just one frame (too small), Manta looks at the video in layers of detail, like a set of nesting dolls.

The Inner Dolls (Local Features): Imagine you are looking at a "Diving" video. The most important part is the "fall." Manta has tiny "Inner Modules" that zoom in on small chunks of the video (like just the fall, or just the jump). They act like microscopes, finding the tiny, critical details that the big picture misses.
The Outer Doll (Global Alignment): Once the microscopes find the important bits, the "Outer Module" acts like a conductor. It takes all those tiny details and lines them up perfectly in time. Even if Person A dives at second 2 and Person B dives at second 5, the conductor says, "Okay, let's slide Person B's fall over so it matches Person A's fall."
Why it's cool: It doesn't just look at the whole video; it looks at the parts and then stitches them together perfectly.

2. The "Detective" Trick (Hybrid Contrastive Learning)

Here is the second problem: Intra-class variance.
Imagine you have 100 videos of people "diving."

Video 1: A pro diver in a pool (blue water, sunny).
Video 2: A kid jumping off a dock (murky water, cloudy).
Video 3: A person diving into a swimming pool at night.

To a computer, these look very different. They might think they are three different actions. This is called "noise" or "variance."

Manta uses a Detective approach to solve this:

The Supervised Detective: It looks at the labeled videos (the "Support" set) and says, "Okay, these 5 videos are all 'diving.' Even though they look different, they belong in the same group."
The Unsupervised Detective: It looks at the new, unlabeled videos (the "Query" set) and says, "This new video looks a bit like the 'diving' group, even if the lighting is weird. Let's group it there."
The Result: It forces the AI to ignore the "noise" (bad lighting, different angles) and focus only on the essence of the action. It teaches the AI that a dive in the rain is still a dive.

🏆 Why is this a Big Deal?

The paper tested Manta on famous video datasets (like Kinetics and UCF101) with very few examples (only 1 or 5 videos to learn from).

It handles long videos: While other AI models crash or get confused when the video gets longer than 8 seconds, Manta stays calm and accurate even with 128-second videos.
It's robust: If you add "noise" to the video (like making it look rainy, dark, or adding random frames), Manta still gets it right. Other models fail miserably.
It's fast: Because it uses the efficient Mamba architecture, it runs much faster than the old "Transformer" models, which are like heavy trucks compared to Manta's sports car.

🧠 The Takeaway

Manta is like a smart teacher who doesn't just memorize the whole book. Instead, they:

Zoom in on the key sentences (Local Features).
Line up the chapters so the story makes sense (Temporal Alignment).
Ignore the typos and bad handwriting to understand the true meaning (Handling Variance).

This allows the AI to learn new actions from just a few examples, even if the videos are long, messy, or filmed in different conditions. It's a huge step forward for making AI understand the real world, not just perfect, short clips.

1. Problem Definition

The paper addresses Few-Shot Action Recognition (FSAR), a task where the model must classify unseen action categories using only a few labeled video samples (support set). While long video sub-sequences naturally provide richer contextual information to describe entire actions (e.g., "Diving cliff") compared to short clips, they present two specific challenges when applied to FSAR:

Inefficiency of Transformers: Mainstream Transformer-based methods suffer from high computational complexity ( $O(N^2)$ ), limiting them to processing very short sub-sequences (typically ~8 frames). They cannot effectively handle the long sequences required for comprehensive action understanding.
Limitations of Direct Mamba Application: While Mamba (based on State Space Models) offers efficient long-sequence modeling ( $O(N)$ $O (N)$ ), directly applying it to FSAR fails because:
- Lack of Local Feature Modeling: Mamba tends to focus on global features, often overlooking critical but small local features (e.g., the "falling" phase in a dive) that are essential for accurate recognition.
- Temporal Misalignment: Critical local features appear at different time steps across different samples of the same class. Mamba lacks an explicit mechanism to align these temporal dependencies.
- Intra-class Variance Accumulation: As sequence length increases, visual discrepancies (due to camera angles, lighting, etc.) accumulate within the same class, making it difficult to cluster samples effectively and leading to misclassification.

2. Methodology: The Manta Framework

The authors propose Manta (Matryoshka MAmba and CoNtrasTive LeArning), a framework designed to enhance Mamba specifically for long sub-sequence FSAR. It consists of two parallel branches:

A. Matryoshka Mamba Branch (Feature Modeling & Alignment)

This branch replaces standard global modeling with a nested structure to handle local features and temporal alignment simultaneously.

Inner Modules (Local Feature Enhancement): The long sub-sequence is divided into non-overlapping fragments. Multiple Inner Modules (based on Mamba-2) process these fragments independently to enhance local features. They utilize bidirectional scanning (forward and backward) to capture local context without sharing parameters between directions, allowing flexibility in modeling variable local features.
Outer Module (Implicit Temporal Alignment): The outputs of the Inner Modules are fed into an Outer Module. This module uses a bidirectional Mamba structure with shared parameters to scan the entire sequence. By fusing the enhanced local features, it captures timeline dependencies, effectively performing implicit temporal alignment across the sequence.
Multi-Scale Fusion: The framework operates across multiple scales (defined by hyper-parameter $O$ ). Learnable weights ( $w_o$ ) are generated via a Conv2D block to dynamically weigh the importance of features at different scales before averaging them to produce the final representation.
Cross-Distance Calculation: To further refine alignment, the model calculates distances between query/support prototypes in both original and inverted (reversed) feature spaces, averaging these distances to determine the final class prediction.

B. Hybrid Contrastive Learning Branch (Variance Mitigation)

To combat the accumulation of intra-class variance in long sequences, Manta employs a hybrid contrastive learning paradigm:

Supervised Contrastive Learning ( $L_{con}^S$ ): Applied to the labeled Support Set. It pulls samples of the same class together and pushes different classes apart.
Unsupervised Contrastive Learning ( $L_{con}^Q$ & $L_{con}^{SQ}$ ): Applied to the Query Set and a combined set of Support + Query. Since queries are unlabeled, positive pairs are constructed based on the assumption that samples within the same task belong to the same underlying distribution, helping to cluster samples without explicit labels.
Total Loss: The training objective combines Cross-Entropy loss ( $L_{ce}$ ) from the Mamba branch and the Hybrid Contrastive loss ( $L_{hc}$ ):
$L_{total} = \lambda L_{ce} + L_{hc}$

3. Key Contributions

Matryoshka Mamba Architecture: The first application of a nested "Matryoshka" structure (Inner Modules for local features + Outer Module for global alignment) using Mamba for FSAR. This design effectively balances local detail extraction with global temporal alignment.
Hybrid Contrastive Learning Paradigm: A novel loss function combining supervised and unsupervised contrastive learning specifically tailored to mitigate the negative effects of intra-class variance accumulation in long video sequences.
State-of-the-Art Performance: Manta achieves new SOTA results on four major benchmarks (SSv2, Kinetics, UCF101, HMDB51), significantly outperforming Transformer-based methods and other Mamba variants, particularly in long sub-sequence settings (up to 128 frames).

4. Experimental Results

Benchmarks: The model was tested on SSv2, Kinetics, UCF101, and HMDB51 under 5-way 1-shot and 5-shot settings.
Performance:
- On SSv2 (1-shot), Manta with ResNet-50 achieved 63.4%, surpassing the previous SOTA (AMFAR) by 1.7%.
- On Kinetics (1-shot), it reached 82.4% (ResNet-50) and 84.4% (VMamba-B), outperforming multimodal methods like AMFAR.
- Long Sequence Capability: Unlike Transformer-based methods that fail (Out of Memory) or degrade significantly when sequence length ( $F$ ) exceeds 16, Manta maintains stable and improving performance up to $F=128$ .
Ablation Studies:
- Removing either the Inner Module, Outer Module, or Contrastive Loss resulted in significant performance drops, confirming the necessity of all components.
- Multi-scale design: Using scales $\{1, 2, 4\}$ yielded the best results; adding more scales introduced redundancy.
- Robustness: Manta demonstrated superior robustness against frame-level noise, sample-level noise, and background noise (Gaussian, rain, lighting) compared to TRX, HyRSM, and MoLo.
Efficiency: Manta is significantly faster than Transformer-based methods (e.g., ~4.25 hours for 10k tasks vs. ~8.67 hours for TRX) due to the linear complexity of Mamba.

5. Significance

This paper bridges the gap between efficient long-sequence modeling (Mamba) and the specific requirements of Few-Shot Action Recognition. By solving the critical issues of local feature neglect and temporal misalignment inherent in standard Mamba applications, and by introducing a mechanism to handle variance accumulation in long videos, Manta enables the use of long, context-rich video sub-sequences in low-data regimes. This opens new avenues for robust action recognition in real-world scenarios where long-duration actions are common and data is scarce.