💻 computer science

Otter: Mitigating Background Distractions of Wide-Angle Few-Shot Action Recognition with Enhanced RWKV

The paper proposes Otter, a novel framework that mitigates background distractions in wide-angle few-shot action recognition by integrating a Compound Segmentation Module to emphasize subjects and a Temporal Reconstruction Module to restore temporal relations, achieving state-of-the-art performance across multiple benchmarks.

Original authors: Wenbo Huang, Jinghui Zhang, Zhenghao Chen, Guang Li, Lei Zhang, Yang Cao, Fang Dong, Takahiro Ogawa, Miki Haseyama

Published 2026-03-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenbo Huang, Jinghui Zhang, Zhenghao Chen, Guang Li, Lei Zhang, Yang Cao, Fang Dong, Takahiro Ogawa, Miki Haseyama

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

🎬 The Big Problem: The "Wide-Angle" Trap

Imagine you are trying to teach a robot to recognize a specific action, like snowboarding.

The Normal View: You show the robot a close-up video of a person on a snowboard. The robot sees the person clearly and says, "Ah, snowboarding!" Easy peasy.
The Wide-Angle View: Now, imagine you zoom out so far that the person looks like a tiny speck in a massive, snowy landscape. The screen is 90% white snow, trees, and sky, and only 10% is the actual snowboarder.

The Problem:
Current AI models get confused here. They look at the video and think, "Wow, there is a lot of snow! This must be about 'winter' or 'nature'!" They get distracted by the background (the scenery) and forget to look at the main character (the person).

This is called Few-Shot Action Recognition (FSAR). It's like trying to teach a child a new game after only showing them two or three examples. If the examples are cluttered with distracting backgrounds, the child (the AI) gets confused.

🦦 The Solution: Enter "Otter"

The researchers built a new AI model named Otter. Think of Otter as a super-focused detective who refuses to be distracted by the scenery. Otter uses a special brain architecture called RWKV (which is like a super-fast, memory-efficient way of reading a story), but Otter adds two special "glasses" to make it work better on wide-angle videos.

Here is how Otter solves the two main problems:

1. The "Spotlight" Glasses (Compound Segmentation Module - CSM)

The Issue: In a wide-angle video, the snowboarder is tiny. The AI usually looks at the big, loud background (the snow) instead of the quiet, small subject.

Otter's Fix:
Imagine the video frame is a giant pizza.

Old AI: Takes a bite of the whole pizza, including the crust, the sauce, and the pepperoni, and tries to guess what it is.
Otter: Uses a laser cutter to slice the pizza into tiny squares. It then looks at each square and asks, "Is this the pepperoni (the person) or just the sauce (the background)?"
The Magic: Otter learns to turn up the brightness on the "pepperoni" squares and dim the "sauce" squares. It effectively puts a spotlight on the person, making them stand out even if they are tiny in the frame.

2. The "Time-Travel" Glasses (Temporal Reconstruction Module - TRM)

The Issue: In wide-angle videos, because the background (snow/trees) looks almost identical in every frame, the AI gets confused about time. It can't tell if the person is moving forward, backward, or just standing still because the background doesn't change enough to give a clue. It's like watching a movie where the background is a static painting; you can't tell if the actors are moving.

Otter's Fix:
Imagine you are watching a movie, but the frames are jumbled up.

Old AI: Tries to watch the movie forward only. If the background is boring, it loses the plot.
Otter: Watches the movie forward AND backward at the same time.
The Magic: By scanning the video in both directions, Otter can piece together the "story" of the movement. Even if the background is boring, the change in the person's position relative to the background becomes clear. It reconstructs the timeline so the AI understands, "Okay, the person moved from left to right," even if the background looks the same.

🏆 How Well Does Otter Work?

The researchers tested Otter on many famous video datasets (like Kinetics, UCF101, and a special badminton dataset).

The Results: Otter beat all the previous "State-of-the-Art" (best-in-class) models.
The Analogy: If previous models were like a student trying to study in a noisy cafeteria, Otter is like that same student putting on noise-canceling headphones and finding a quiet library. It focuses on the "student" (the action) and ignores the "cafeteria" (the background).

💡 Why is this important?

In the real world, we don't always have perfect, close-up cameras. Security cameras, drone footage, and sports broadcasts are often wide-angle.

Health Monitoring: A camera in a living room needs to know if an elderly person is falling, even if they are far away in the corner of the room.
Sports Analysis: A drone filming a badminton match needs to track two players moving fast across a huge court.

Otter proves that by teaching AI to ignore the noise and reconstruct the timeline, we can make it much smarter at understanding human actions, even when the video is messy or wide-angle.

🚀 Summary in One Sentence

**Otter is a new AI that uses "spotlight" technology to find the person in a wide video and "time-travel" technology to understand their movement, allowing it to recognize actions perfectly even when the background is distracting.

1. Problem Definition

The paper addresses the challenge of Few-Shot Action Recognition (FSAR) specifically within wide-angle video scenarios.

Context: Wide-angle videos (Field of View > 80°) provide rich contextual background, which is beneficial for understanding specific action scenarios (e.g., distinguishing "climbing" from "construction" based on the environment).
The Challenge: In wide-angle frames, the primary subject (the actor) occupies a significantly smaller proportion of the frame compared to regular videos. This leads to two critical issues:
1. Background Distractions: Standard global modeling techniques (like standard RWKV) tend to capture massive amounts of secondary background information (e.g., snow, walls) rather than the primary subject, leading to feature representation errors.
2. Degraded Temporal Relations: Similar backgrounds across consecutive frames obscure the evolution of the primary subject, making it difficult to reconstruct temporal relationships and action direction.
Gap: Existing FSAR methods and even advanced recurrent models like RWKV fail to simultaneously highlight the small subject and reconstruct the degraded temporal dynamics caused by wide-angle backgrounds.

2. Methodology: The Otter Framework

The authors propose Otter (CompOund SegmenTation and Temporal REconstructing RWKV), a framework built upon the RWKV (Receptance Weighted Key Value) architecture, which combines the parallel processing of Transformers with the linear complexity of RNNs.

Otter introduces two core modules to address the specific challenges of wide-angle FSAR:

A. Compound Segmentation Module (CSM)

Goal: To adaptively highlight the primary subject and suppress background noise before feature extraction.
Mechanism:
1. Segmentation: Each video frame is segmented into $HW/p^2$ patches.
2. Feature Aggregation: The module applies Spatial Mixing, Time Mixing, and Channel Mixing (core RWKV units) to capture dependencies within and between patches.
3. Adaptive Weighting: It learns adaptive weights ( $lw_\Delta$ ) for each patch via a convolutional layer and residual connections. These weights determine the importance of each patch.
4. Reconstruction: The weighted patches are reassembled to the original frame resolution. This process effectively amplifies the features of the inconspicuous subject while suppressing the dominant background.

B. Temporal Reconstruction Module (TRM)

Goal: To reconstruct the temporal relations degraded by similar backgrounds in wide-angle sequences.
Mechanism:
1. Bidirectional Scanning: TRM performs feature scanning in both ordered (forward) and reversed (backward) directions across the video frames.
2. Temporal Enhancement: It utilizes the learned weights to reconstruct the temporal evolution of the subject.
3. Prototype Construction: The module generates a Temporal-Enhanced Prototype by averaging the results of the bidirectional scans. This allows the model to perceive action direction and align frames more accurately than standard unidirectional approaches.

C. Dual-Prototype Strategy & Training Objective

Dual Prototypes: The model constructs two types of class prototypes:
1. Regular Prototype: Based on standard feature extraction (retaining subject emphasis from CSM).
2. Temporal-Enhanced Prototype: Incorporates the TRM to capture temporal dynamics.
Loss Function: The training objective ( $L_{total}$ $L_{t o t a l}$ ) combines:
- Cross-Entropy Loss ( $L_{ce}$ ): For classification accuracy.
- Prototype Similarity Losses ( $L^1_P, L^2_P$ ): To maximize the distance between different class prototypes, ensuring better class separation.

3. Key Contributions

First Application of RWKV to Wide-Angle FSAR: The paper pioneers the use of RWKV for this specific domain, identifying its potential for global modeling while addressing its limitations regarding subject highlighting.
Compound Segmentation Module (CSM): A novel mechanism that segments frames into patches and learns adaptive weights to highlight small subjects against large backgrounds, solving the "subject proportion" problem.
Temporal Reconstruction Module (TRM): A bidirectional scanning mechanism integrated into prototype construction that recovers temporal relations degraded by static or similar backgrounds.
Comprehensive Evaluation: The method is validated on standard benchmarks (SSv2, Kinetics, UCF101, HMDB51) and a specialized real-world wide-angle dataset (VideoBadminton), demonstrating superior performance.

4. Experimental Results

State-of-the-Art (SOTA) Performance: Otter achieves the highest accuracy across multiple benchmarks.
- SSv2 (1-shot): 64.7% (vs. previous best of 63.4% with Manta).
- Kinetics (1-shot): 90.5% (vs. 87.4% with Manta).
- UCF101 & HMDB51: Consistently outperforms existing methods like SOAP, Manta, and MoLo.
Wide-Angle Specifics: On the VideoBadminton dataset (all wide-angle samples), Otter significantly outperforms other methods (e.g., 71.2% vs. 64.1% for Manta in 1-shot), proving its effectiveness in mitigating background distractions.
Robustness:
- FoV Sensitivity: As the Field of View increases (simulating wider angles), other methods show a drastic performance drop, whereas Otter maintains a much more stable decline.
- Noise Resilience: Otter demonstrates superior robustness against sample-level noise, frame-level noise, and visual noise (zoom, Gaussian, rain) compared to Transformer-based and other RWKV-based baselines.
Efficiency: Due to the linear complexity of RWKV, Otter is faster in inference time (approx. 4.13 hours for 10k tasks) compared to Transformer-based methods (approx. 7.4+ hours).

5. Significance

Bridging the Gap: This work bridges the gap between high-level global modeling (RWKV) and the specific, often overlooked, challenges of wide-angle video analysis.
Practical Applicability: By improving performance on real-world wide-angle scenarios (like sports monitoring or surveillance where cameras have wide FoVs), the method has direct applications in health monitoring, motion analysis, and security.
Methodological Insight: The paper demonstrates that simply applying powerful global models is insufficient; specific architectural adaptations (segmentation for spatial focus and bidirectional scanning for temporal recovery) are necessary to handle the unique noise characteristics of wide-angle data.

In conclusion, Otter represents a significant advancement in Few-Shot Action Recognition by effectively solving the dual problems of subject occlusion by background and temporal degradation in wide-angle videos, setting a new benchmark for performance and robustness in this domain.