💻 computer science

Information-Guided Spatiotemporal Scanning for Efficient Video State-Space Modeling

This paper introduces AdaScan, an information-guided adaptive scanning framework that dynamically optimizes the spatial granularity, region selection, and serialization order of video tokens based on content-specific information, thereby enhancing the efficiency and performance of State-Space Models (SSMs) in video understanding tasks.

Original authors: Tian Ye, Wang Lu, Chaowei Yang, Yahui Kang, Jun Miao, Shuxuan Li

Published 2026-10-22

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Tian Ye, Wang Lu, Chaowei Yang, Yahui Kang, Jun Miao, Shuxuan Li

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Watching a video is a continuous stream of information, but for a computer, it is a massive, repetitive pile of data. Every second of footage contains thousands of tiny squares of pixels, and in a standard video, most of these squares do not change much from one moment to the next. A wall in the background, a patch of sky, or a table surface often remains static while a person moves across the scene. To understand what is happening, a computer needs to process this stream, but doing so for every single pixel in every frame is incredibly expensive and slow. In recent years, a new type of mathematical engine called a state-space model has emerged as a way to handle these long sequences of data more efficiently than older methods. These models work by reading information in a specific order, updating their internal understanding step by step as they go. The critical insight is that the order in which the computer reads the video matters just as much as the content itself. If the computer reads the unchanging background first and the moving action last, it wastes time and memory on things that do not change, potentially missing the important details that arrive later in the sequence.

For a long time, computers have read video frames using a fixed, rigid pattern, much like a person reading a book line by line from top to bottom, regardless of whether the text contains a story or just blank space. This method works well for uniform text, but it fails to account for the fact that video information is unevenly distributed. In a real scene, the interesting parts—where a person is walking, an object is falling, or a face is turning—occupy only a small portion of the screen, while the rest is often predictable and redundant. A fixed reading order treats a quiet patch of sky the same as a fast-moving car, forcing the computer to process the sky with the same intensity as the car. This inefficiency means the computer spends too much energy on things it already knows and not enough on the new, surprising evidence that actually helps it understand the scene.

To solve this, researchers have developed a new approach called AdaScan, which acts as a smart guide for how a computer should look at a video. Instead of following a pre-set path, this system decides in real-time where to look, how closely to look, and in what order to visit different parts of the screen. It operates on the principle that a computer should prioritize the parts of the video that are changing or unpredictable. The system first identifies which areas of the video contain useful information by looking for three specific things: the static structure of the scene, the movement of objects, and, most importantly, the "surprise" factor. This surprise factor measures how much the current view differs from what the computer expected based on the previous frames. If a computer predicts that a ball will be in a certain spot based on its previous movement, but the ball is actually somewhere else, that difference is a high-value piece of information that needs immediate attention.

Once the system knows where the interesting parts are, it adjusts the size of the "lens" it uses to look at them. In areas where nothing is happening, it uses a coarser, broader view to save time, grouping large areas together. In areas where action is taking place, it switches to a fine-grained view, breaking the image into smaller pieces to capture every detail. Finally, it plans a path to visit these selected areas. This path is not random, nor is it a simple straight line. Instead, it connects the interesting spots in a smooth, logical sequence that minimizes the distance the computer has to jump between points. This ensures that the computer builds a coherent story of the video, moving from one piece of evidence to the next without wasting time jumping back and forth across the screen or revisiting areas it has already seen.

The researchers tested this new method on several challenging video tasks, including recognizing human actions, understanding fine details in daily activities, and identifying objects in complex scenes. In a standard test for recognizing actions, the new system achieved a success rate of 88.5 percent, outperforming other advanced methods that use much larger and more complex models. In a test designed to distinguish between very similar cooking activities, it reached 97.6 percent accuracy. For a task involving identifying every object in a video frame, it achieved a score of 58.7 percent, which was the highest among the compared methods. These results show that by changing the order in which information is processed, rather than just making the computer smarter, the system can learn more effectively and quickly.

The study also compared this adaptive approach against several other ways of scanning video. A random scanning method, which picks spots without any logic, performed poorly because it often missed important connections between frames. A standard grid-based scan, which looks at every spot in a fixed order, was consistent but wasted a lot of time on unimportant background areas. A greedy approach that simply picked the most interesting spots without planning a path resulted in a chaotic sequence that jumped erratically across the screen, confusing the computer's internal memory. The new method succeeded because it balanced the need to find interesting information with the need to move smoothly from one spot to the next. It found that the most effective way to understand a video is not to look at everything equally, but to follow a path that is guided by what is actually happening in the scene.

This work demonstrates that the way a computer organizes its attention is just as important as the algorithms it uses to process that attention. By allowing the scanning process to adapt to the content of the video, the system can focus its resources where they are needed most. The researchers found that the specific combination of identifying surprising changes, adjusting the level of detail, and planning a smooth path was essential; removing any one of these parts reduced the system's performance. The results suggest that for computers to truly understand video, they must stop treating every frame as a static image to be read line by line and start treating it as a dynamic event to be explored intelligently. This shift in approach allows for more efficient and accurate video understanding, paving the way for systems that can process long, complex videos without getting bogged down by redundant data.

Technical Summary: Information-Guided Spatiotemporal Scanning for Efficient Video State-Space Modeling

Problem Statement

State-space models (SSMs), such as Mamba, have emerged as an efficient alternative to attention-based architectures for long-sequence modeling due to their favorable computational complexity. However, extending SSMs to video data requires serializing two-dimensional spatial and temporal visual features into one-dimensional sequences. Existing visual SSMs typically rely on predefined, content-independent scanning patterns (e.g., raster, zigzag, or multidirectional scans).

The authors argue that these fixed strategies fail to account for the non-uniform spatial distribution and temporal evolution of information in natural videos. In dynamic scenes, foreground objects occupy limited regions, large background areas are often predictable from previous frames, and new informative regions emerge as actions evolve. A fixed scanning order assigns equal sequential priority to redundant background patches and critical task-relevant regions, leading to:

Redundant state updates for predictable content.
Delayed integration of newly emerging evidence.
Fragmented spatiotemporal trajectories if high-saliency patches are selected independently without regard for transition consistency.

Methodology: AdaScan

To address these limitations, the paper proposes AdaScan, an information-guided adaptive scanning framework. AdaScan treats visual token serialization as a spatiotemporal decision problem, dynamically determining where to look, at what granularity, and in what order. It operates at the serialization stage, making it conceptually decoupled from the underlying SSM operator.

The framework consists of three complementary components:

1. Information-Aware Weighting and Scoring (IAWS)

IAWS estimates the utility of candidate regions by jointly modeling three cues:

Static Structural Information: Measures appearance diversity (color entropy) and local intensity variance to capture semantically structured regions even in static scenes.
Motion-Compensated Predictive Innovation: Unlike raw motion magnitude, this measures the discrepancy between the current observation and a motion-compensated prediction derived from preceding frames. This emphasizes visual evidence that cannot be inferred from previous observations (e.g., new objects or independent motion) while suppressing predictable content and camera movement.
Residual Motion: Captures explicit local dynamics after removing dominant global motion.
Adaptive Weighting: Instead of fixed coefficients, a lightweight MLP predicts frame-dependent weights to dynamically balance these cues based on the specific video content.

2. Adaptive Spatial Resolution (ASR)

Recognizing that uniform patch sizes are suboptimal, ASR dynamically adjusts the spatial granularity of visual tokens.

It calculates an information activity metric for a video subset, combining information scores and residual motion magnitude.
It optimizes patch size ( $N_p$ ) by balancing representation loss (information loss from coarse patches) against execution cost (latency/complexity).
Result: Information-rich or highly dynamic regions are represented with finer patches, while predictable or redundant areas use coarser representations, reducing sequence length without sacrificing critical details.

3. Information-Guided Spatiotemporal Path Planning (IGSPP)

IGSPP organizes selected regions into coherent scanning trajectories to avoid the fragmentation caused by greedy selection of high-score patches.

Graph Construction: A directed graph is built where nodes are patches and edges represent spatial neighbors or motion-consistent temporal transitions.
Transition Regularization: The cost of moving between patches includes spatial displacement, temporal distance, and motion inconsistency. This discourages arbitrary long-range jumps.
Receding-Horizon Optimization: To avoid the combinatorial complexity of global optimization, a receding-horizon solver with a look-ahead depth ( $L$ ) is employed. It selects the next patch by maximizing a value function that considers immediate information gain, transition cost, and the potential to reach future informative regions.
Objective: The strategy seeks a trajectory that maximizes accumulated information gain while minimizing redundant revisits and transition costs.

Key Contributions

The paper summarizes its contributions as follows:

Problem Formulation: It reframes visual token serialization in video SSMs as an information-guided spatiotemporal scanning problem, allowing the process to adapt to non-uniform and evolving information distributions.
Unified Framework (AdaScan): It proposes a framework integrating information-aware scoring (with predictive innovation), adaptive spatial resolution, and spatiotemporal path planning. This jointly determines the regions to process, their representation granularity, and their sequential order.
Empirical Validation: It validates AdaScan across three distinct tasks: action recognition, fine-grained video understanding, and video semantic segmentation, demonstrating that content-adaptive scanning outperforms conventional predefined strategies.

Experimental Results

The authors evaluated AdaScan on three benchmarks: Kinetics-400 (action recognition), Breakfast (fine-grained understanding), and VSPW (video semantic segmentation).

Kinetics-400: AdaScan achieved 88.5% Top-1 accuracy with 43M parameters and 8 input frames. This outperforms larger models (e.g., VideoMAP-L with 301M parameters) and other recent SSM-based methods (e.g., VideoMamba++ at 84.5%).
Breakfast: Achieved 97.6% frame-level accuracy using 8 input frames, surpassing methods that use significantly more frames (e.g., 64 frames).
VSPW: Achieved 58.7% mIoU with 8 input frames, outperforming models with larger backbones (e.g., Swin-L based models).

Ablation Studies confirmed the complementary nature of the components:

Predictive Innovation was identified as the strongest individual information cue.
Path Planning provided the most significant boost when combined with information scoring, highlighting the importance of trajectory coherence over simple greedy selection.
Adaptive Resolution reduced latency and computational cost while maintaining accuracy.
Compared to baselines (Random, Zigzag, Greedy), AdaScan achieved the best balance between information acquisition (highest gain) and trajectory efficiency (lowest redundant visits and jump ratios).

Significance and Claims

The paper claims that AdaScan offers a fundamental shift from fixed architectural conventions to content-adaptive serialization. By restructuring the input sequence rather than redesigning the recurrent operator, the method allows state evolution to focus on informative and temporally evolving visual evidence.

The authors assert that their approach demonstrates:

Generalizability: The benefits of adaptive scanning extend across classification, fine-grained temporal understanding, and dense semantic prediction.
Efficiency: The method achieves a favorable trade-off between informative content acquisition and scanning efficiency, reducing redundant traversal and computation compared to fixed strategies.
Modularity: The framework is conceptually modular and can be integrated into existing visual SSM backbones without altering the core state-space transformation logic.

The work concludes that effective video modeling with SSMs requires not just efficient operators, but intelligent serialization strategies that align with the heterogeneous and dynamic nature of video content.