🤖 AI

video-SALMONN S: Memory-Enhanced Streaming Audio-Visual LLM

The paper introduces video-SALMONN S, a memory-enhanced streaming audio-visual LLM that utilizes test-time training to continuously convert short-term representations into long-term memory, enabling it to process over 3-hour videos and significantly outperform existing models on long-duration and episodic learning benchmarks.

Original authors: Guangzhi Sun, Yixuan Li, Xiaodong Wu, Yudong Yang, Wei Li, Zejun Ma, Chao Zhang

Published 2026-02-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Guangzhi Sun, Yixuan Li, Xiaodong Wu, Yudong Yang, Wei Li, Zejun Ma, Chao Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand a movie that is three hours long, but you can only show it one frame per second (a very slow, choppy view) and the picture is a bit blurry (360p resolution). Furthermore, the robot has a very small "brain" (memory) that can only hold a tiny amount of information at once.

Most current AI robots forget the beginning of the movie by the time they reach the end. This paper introduces a new robot named video-SALMONN S that solves this problem. Here is how it works, using simple analogies:

1. The Problem: The "Leaky Bucket"

Imagine trying to fill a bucket with water (video information) while the bucket has a hole in the bottom.

Old AI models are like buckets with big holes. If you pour in a 3-hour video, the water (information) leaks out before you get to the end. They have to throw away most of the video to fit it in their memory, so they forget important details.
Streaming models (the current best) are better, but they still lose information over time because they have to constantly delete old data to make room for new data.

2. The Solution: The "Self-Editing Notebook" (TTT)

The secret sauce of video-SALMONN S is a technique called Test-Time Training (TTT).

The Analogy: Imagine you are reading a very long book. Instead of just reading and forgetting, you have a magical notebook. Every time you read a new page, you don't just write it down; you rewrite your own brain based on what you just read. You update your understanding of the story while you are reading it.
How it works: As the video plays, the model constantly tweaks its own internal settings (its "weights") to store the story's details. It turns short-term memories (what just happened) into long-term memories (the whole story) by changing its own structure. It's like the robot is "learning" the video in real-time, rather than just watching it.

3. The "Long-Range Prediction" Trick

To make sure the robot doesn't forget the beginning of the movie, the authors added a special rule called Long-Span Prediction.

The Analogy: Imagine you are playing a game of "Telephone" with a friend, but the game is 3 hours long. Usually, the message gets garbled. This model has a rule: "Every time you pass a message, you must also check if you can still remember what was said 30 minutes ago."
The Result: This forces the robot to keep the early parts of the video clear in its mind, even while it's processing the latest frames.

4. The "Smart Librarian" (Memory Reader)

Even with a magical notebook, you can't read every single page of a 3-hour book when someone asks you a specific question. That would take too long.

The Analogy: When a user asks, "What happens next?" the robot acts like a smart librarian. Instead of pulling out the entire 3-hour archive, it quickly scans its memory, finds the exact few pages relevant to the question, and ignores the rest.
The Benefit: This keeps the robot fast and efficient, even though it has watched hours of video.

5. The New Test: "ELViM" (The Memory Challenge)

The authors realized that existing tests were too easy; they only asked questions about videos the robot was watching right now. They created a new test called ELViM (Episodic Learning from Video Memory).

The Scenario: The robot watches a video of someone baking a cake. Then, it watches 30 minutes of other videos (like nature documentaries). Finally, the baking video comes back up, paused right before the baker cracks an egg.
The Question: "What is the next step?"
The Goal: The robot must remember the baking step from 30 minutes ago, ignore the nature documentaries in between, and answer correctly.
The Result: video-SALMONN S crushed this test, getting 15% better than the best previous models. It proved the robot could actually "remember" skills it learned hours earlier.

Summary of Achievements

It watches long videos: It can handle over 3 hours of video at a low frame rate without running out of memory.
It remembers better: It beats both "streaming" models (that watch live) and "offline" models (that see the whole video at once) by a significant margin (3-7% better on standard tests, 15% better on the memory test).
It's efficient: It doesn't need a super-computer to do this; it fits within a standard memory budget.

In short, video-SALMONN S is the first AI that can watch a long movie, learn from it, and remember the details hours later, all while keeping its memory usage small and steady.

Technical Summary: video-SALMONN S

Problem Statement

Long-duration streaming video understanding is a critical capability for future AI agents, yet current state-of-the-art (SOTA) video Large Language Models (LLMs) face significant limitations. Most existing models operate in an offline manner, requiring a fixed maximum number of input visual tokens, which leads to substantial information loss for videos exceeding this limit. While recent streaming approaches attempt to handle continuous video inputs via token merging or downsampling, they often incur cumulative information loss over long durations (e.g., 3+ hours), resulting in performance inferior to offline models. Furthermore, existing methods relying on external memory or specialized frame selection often require complex designs or specialized training data, limiting scalability. A key gap remains in modeling long-term dependencies where an agent must learn procedural knowledge from video streams observed hours earlier and apply it to real-time decision-making.

Methodology: video-SALMONN S

The authors propose video-SALMONN S, a memory-enhanced streaming audio-visual LLM designed to process videos exceeding 3 hours at 1 FPS and 360p resolution under a fixed memory budget. The core innovation is the integration of Test-Time Training (TTT) as a streaming memory mechanism, a first for video LLMs.

1. TTTMEM Layer

Instead of relying solely on token compression, video-SALMONN S employs a TTTMEM layer that transforms short-term multimodal representations into long-term memory embedded within the model's parameters (fast weights).

Mechanism: The layer functions as a recurrent neural network updating an MLP's fast weights ( $W_t$ ) for each incoming mini-batch of video tokens ( $X_t$ ).
Loss Functions: The weights are updated to minimize two objectives:
1. Multiview Reconstruction Loss: Reconstructs the current input.
2. Long-Span Prediction Loss: Predicts inputs from a distant timestamp ( $X_{t-T}$ ), acting as a temporal consistency regularizer to prevent the overwriting of critical historical information.
Token Management: Unlike methods that merge tokens (which can cause over-smoothing), video-SALMONN S uses a similarity-based discarding strategy. It combines new memory tokens with previous ones and discards tokens with the highest cosine similarity to their neighbors, preserving the long-span context representation ability of the TTT layer.

2. Two-Stage Training Scheme

To address GPU memory constraints during training with long sequences, a two-stage strategy is employed:

Stage 1 (Cold-Start): The TTTMEM parameters (projection matrices) and the LLM are jointly trained with medium-length contexts to establish the fast-weight update mechanism.
Stage 2 (Scale-Up): The TTTMEM projection parameters are frozen, but the fast-weight update rule is maintained. The model is then trained on significantly longer sequences and increased memory token counts, optimizing the LLM's ability to extract information from the expanded memory.

3. Modality-Aware Memory Reading

To handle the fixed memory budget while responding to specific prompts, the model employs a modality-aware, prompt-dependent memory reading mechanism.

Instead of processing all memory tokens, the LLM compresses the Key-Value (KV) cache by selecting only the most relevant tokens based on attention scores between the prompt and the memory chunks.
This mimics the human process of retrieving specific information from long-term memory into working memory, ensuring constant memory usage while adhering to streaming constraints.

Key Contributions

video-SALMONN S Model: The first streaming video LLM to utilize Test-Time Training (TTT) as a memory mechanism. It achieves SOTA performance on videos >3 hours long (1 FPS, 360p) under a fixed memory budget, outperforming both streaming and non-streaming baselines.
TTTMEM Layer & Training: Introduction of a novel layer incorporating long-span prediction objectives and a two-stage training scheme to enhance long-range dependency modeling and memory capacity without information loss.
ELViM Benchmark: The Episodic Learning from Video Memory (ELViM) benchmark is introduced to explicitly evaluate whether streaming video LLMs can acquire procedural knowledge from past video experiences (observed hours earlier) and reuse it for real-time decision-making. This fills a gap in existing benchmarks that focus primarily on short-term perception.

Results

The model was evaluated on general long-video benchmarks (Video-MME, LVBench, VideoEvalPro) and the proposed ELViM benchmark.

Long Video Benchmarks: video-SALMONN S consistently outperformed strong streaming and non-streaming baselines by 3–7% in absolute accuracy. Specifically, it achieved improvements of 1.7%, 2.9%, and 5.4% over its non-streaming counterpart on Video-MME (Long), LVBench, and VideoEvalPro, respectively.
ELViM Performance: The model demonstrated a 15% absolute accuracy improvement over strong non-streaming models on ELViM, highlighting its superior ability to learn from and recall video memory.
Efficiency: The TTTMEM layer adds minimal overhead (approx. 0.1s) to inference time, and the model maintains a fixed GPU memory footprint (~24GB for 16k tokens) comparable to baseline streaming methods.
Ablation Studies: The long-span prediction loss was shown to be critical for retaining information over long temporal distances. The modality-aware reading mechanism provided consistent improvements over training-free KV-cache compression methods like AdaReTaKe.

Significance and Claims

The paper claims that video-SALMONN S represents a significant step toward future AI agents capable of operating continuously over streaming inputs with bounded memory. By shifting from static token compression to dynamic, parameter-based memory via TTT, the model mitigates the cumulative information loss inherent in current streaming approaches.

The introduction of the ELViM benchmark is highlighted as a principled evaluation tool for continual learning and long-term memory in multimodal systems, moving beyond simple perception tasks to assess an agent's ability to learn procedural skills from extended audio-visual experiences. The authors position this work as foundational for applications requiring long-horizon decision-making, instructional assistance, and accessibility tools for users relying on extended audio-visual information.