💻 computer science

RS-SSM: Refining Forgotten Specifics in State Space Model for Video Semantic Segmentation

This paper introduces RS-SSM, a novel approach for video semantic segmentation that addresses the loss of specific spatiotemporal details in state space models by employing a Channel-wise Amplitude Perceptron and a Forgetting Gate Information Refiner to adaptively recover forgotten information, thereby achieving state-of-the-art performance with high computational efficiency.

Original authors: Kai Zhu, Zhenyu Cui, Zehua Zang, Jiahuan Zhou

Published 2026-03-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kai Zhu, Zhenyu Cui, Zehua Zang, Jiahuan Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a long, complex movie and trying to describe exactly what is happening in every single frame. You need to remember the big picture (the plot, the main characters) but also the tiny details (the texture of a shirt, the flicker of a streetlight, the exact edge of a moving car).

This is the challenge of Video Semantic Segmentation (VSS). Computers need to label every single pixel in a video to understand the scene.

The Problem: The "Overworked Librarian"

In recent years, a new type of AI called a State Space Model (SSM) has become very popular for video tasks. Think of an SSM as a super-efficient librarian trying to summarize a massive library of movies into a single, tiny notebook.

The Good News: This librarian is incredibly fast. They can compress hours of video into a small summary without getting tired or running out of memory.
The Bad News: To fit everything into that tiny notebook, the librarian has to throw away the "boring" stuff. Unfortunately, they often throw away the specific details too. They remember that a "car" was there, but they forget exactly where the tire meets the road or how the paint is peeling.

In technical terms, the model "forgets" the high-frequency details (edges, textures) while keeping the low-frequency details (general shapes). This leads to blurry, inaccurate segmentation where the computer can't tell the difference between a car and the road it's driving on.

The Solution: The "Detail Detective" (RS-SSM)

The authors of this paper, led by Kai Zhu and Jiahuan Zhou, built a new system called RS-SSM (Refining Specifics State Space Model). They realized that instead of trying to make the notebook bigger (which slows everything down), they should teach the librarian how to recover the lost details after the summary is made.

Here is how they did it, using two clever tools:

1. The "Frequency Scanner" (Channel-wise Amplitude Perceptron - CwAP)

Imagine the video data is a song. Some parts are the deep bass (the big shapes), and some parts are the high-pitched cymbals (the sharp edges and textures).

The old librarian just listened to the bass and ignored the cymbals.
The CwAP is like a special microphone that scans the audio and says, "Hey! The high-pitched cymbals are getting lost in channels 3, 5, and 9. We need to pay extra attention to those channels!"
It creates a map showing exactly where the "forgotten details" are hiding so the system knows what to fix.

2. The "Memory Reversal Switch" (Forgetting Gate Information Refiner - FGIR)

In the librarian's notebook, there is a "forgetting gate"—a mechanism that decides what to throw away.

The old system had a gate that was too aggressive; it threw away the details too quickly.
The FGIR is a smart switch. It looks at the map from the Frequency Scanner and says, "Wait, we threw away the details in channel 3. Let's reverse the gate for that channel!"
Instead of forgetting, it now remembers and refines those specific details. It essentially tells the AI: "Don't forget the tire edge this time; focus on it."

The Result: A Perfect Summary

By combining these two tools, the RS-SSM system works like this:

It compresses the video quickly (keeping the speed).
It scans to see what details were lost.
It flips a switch to "un-forget" those specific details.
The final result is a video segmentation that is both fast and incredibly precise.

Why This Matters

Think of it like editing a photo.

Old AI: Takes a photo, shrinks it to a thumbnail to save space, and then tries to guess the details. The edges look fuzzy.
RS-SSM: Takes the photo, shrinks it, but then uses a special "detail brush" to paint the sharp edges back in perfectly, all without making the file size huge.

The paper shows that this method works better than any other current technology on major video datasets, making it a huge step forward for things like self-driving cars (which need to know exactly where the road ends and the grass begins) and video editing tools.

In short: They taught the AI how to remember the little things without slowing down the whole process.

1. Problem Statement

Video Semantic Segmentation (VSS) requires pixel-level spatiotemporal modeling to maintain temporal consistency for semantic objects. While recent State Space Models (SSMs) offer efficient linear-complexity compression for long video sequences, they suffer from a critical limitation:

The "Forgetting" Issue: SSMs use a fixed-size state space to compress information. While this effectively preserves common semantic information (e.g., global structures, smooth regions), it inevitably forgets specific spatiotemporal details (e.g., object boundaries, textures, local variations).
Consequence: This loss of "specifics" leads to coarse segmentation results, degrading pixel-level accuracy and temporal consistency, which is detrimental to VSS tasks.
Existing Alternatives: Transformer-based methods capture these details via attention mechanisms but suffer from quadratic computational complexity, making them inefficient for long videos.

2. Methodology: RS-SSM

The authors propose RS-SSM (Refining Specifics State Space Model), a dual-path architecture designed to complementarily refine the specific information lost during SSM compression. The pipeline consists of three main components:

A. Overall Architecture

The model processes video frames through an image encoder followed by $L$ dual-path SSM layers. Each layer contains two SSM modules ( $\theta_1$ and $\theta_2$ ):

$\theta_2$ (Standard Path): Processes features normally, acting as the baseline SSM.
$\theta_1$ (Refining Path): Receives a refined forgetting gate derived from $\theta_2$ to specifically target and recover forgotten details.
The outputs of both paths are concatenated and fused via an MLP to produce the final segmentation mask.

B. Channel-wise Amplitude Perceptron (CwAP)

To identify what information is being forgotten, the CwAP module analyzes the frequency domain characteristics of the feature maps:

Frequency Analysis: It applies a 2D Fast Fourier Transform (FFT) to projected features.
Spectrum Feature Extraction: It partitions the frequency domain into bands. High-frequency bands correspond to "specifics" (boundaries, textures), while low-frequency bands correspond to "common" semantics.
Quantification: It calculates the energy ratio of high-frequency bands to generate Spectrum Features ( $F_t$ ), which quantify the amount of specific information in each channel.
Alignment Loss ( $L_{ci}$ ): A channel information loss is introduced to maximize the cosine similarity of Spectrum Features across different samples. This forces the model to consistently allocate specific information to the same subset of channels, making them easier to target later.

C. Forgetting Gate Information Refiner (FGIR)

The FGIR module adaptively modifies the forgetting mechanism to recover lost details:

Inversion: It computes an inverted version of the forgetting gate matrix ( $A$ ) from the standard path ( $\theta_2$ ). Inverting the gate reverses the compression degree (highly compressed channels become less compressed).
Adaptive Weighting: It calculates an inverting weight ( $\alpha$ $α$ ) based on:
- Spectrum Features ( $\tilde{F}$ ): Channels rich in specific information.
- Channel Importance ( $\beta$ ): Derived from the original gate, indicating which channels were heavily compressed (and thus lost more info).
- Logic: Channels with high specific content but low importance in the original path receive higher weights for inversion.
Refinement: The final refined forgetting gate ( $A^R$ ) is a weighted interpolation between the original gate and the inverted gate. This $A^R$ is fed into $\theta_1$ , guiding it to focus on complementarily refining the specific information that $\theta_2$ forgot.

3. Key Contributions

Problem Identification: The paper identifies and formalizes the "specifics forgetting" limitation in existing SSM-based VSS methods, where fixed-size state compression sacrifices pixel-level details.
Novel Architecture (RS-SSM): Proposes a dual-path framework where one path explicitly refines the forgotten details of the other.
CwAP & FGIR Modules:
- CwAP: Uses frequency domain analysis to quantify and align channel-wise specific information distribution.
- FGIR: Introduces an adaptive mechanism to invert and refine the forgetting gate, enabling the model to recover high-frequency details without increasing computational complexity significantly.
Efficiency: Maintains the linear computational complexity of SSMs while achieving state-of-the-art segmentation accuracy.

4. Experimental Results

The method was evaluated on four benchmarks: VSPW, NYUv2, CamVid, and Cityscapes.

Performance: RS-SSM achieves State-of-the-Art (SOTA) performance across all datasets.
- On VSPW (large-scale), it outperforms the previous best SSM method (TV3S) by 1.6 mIoU (MiT-B5 backbone) and 1.7 mIoU (MiT-B1 backbone).
- On Cityscapes, it improves mIoU by 2.6 over the SOTA.
- It consistently outperforms Transformer-based methods (e.g., SegFormer, CFFM++) in accuracy while maintaining better efficiency.
Efficiency:
- Despite the added modules, RS-SSM remains highly efficient. On VSPW with MiT-B5, it reduces GFLOPs by 8.2 and increases FPS by 6.2 compared to TV3S.
- The CwAP and FGIR modules are lightweight, preserving the linear complexity advantage of SSMs.
Ablation Studies:
- Removing the dual-path structure (using a single vanilla SSM) drops performance by up to 3.1 mIoU.
- Removing the adaptive refinement (using a simple inverted gate without CwAP) drops performance by 1.4 mIoU, proving the necessity of the frequency-based alignment and adaptive weighting.

5. Significance

Bridging the Gap: RS-SSM successfully bridges the gap between the computational efficiency of State Space Models and the pixel-level precision required for video segmentation.
New Paradigm: It introduces a "refinement" paradigm for SSMs, demonstrating that specific information lost during compression can be recovered through adaptive gate inversion rather than simply increasing model size or using quadratic attention.
Practical Impact: The method offers a viable solution for real-time, long-video processing applications (e.g., autonomous driving, surveillance) where both high accuracy and low latency are critical.

Code Availability: The authors have released the code at https://github.com/zhoujiahuan1991/CVPR2026-RS-SSM.