Technical Summary: SLVMBench - Skill Learning from Video Memory

Problem Statement

Recent advancements in audio-visual large language models (video-LLMs) have demonstrated proficiency in understanding short videos (minutes) for tasks like captioning and question answering. However, a critical gap exists in evaluating these models for real-world embodied AI applications, which require accurate real-time perception and effective long-term memory.

Current evaluation paradigms are fragmented:

1. Real-time benchmarks (e.g., StreamingBench, OVOBench) focus on short-term perception (10–20 minutes) without requiring long-term memory retention.
2. Long-video benchmarks (e.g., VideoMME, LVBench) test passive comprehension and information extraction over hours but do not require real-time processing or the application of procedural knowledge.
3. Skill-learning benchmarks often rely on short, immediate demonstrations or lack guaranteed knowledge coverage, failing to test the full pipeline of memorizing, extracting, and transferring procedural knowledge from long-term memory to real-time tasks.

There is a lack of a unified benchmark that evaluates whether video-LLMs can learn a skill from a tutorial embedded in a long stream of irrelevant content and apply that skill to a real-time task after a significant temporal delay.

Methodology: SLVMBench Design

The authors propose SLVMBench (Skill Learning from Video Memory), the first benchmark designed to jointly evaluate skill acquisition from long video memory and its application to real-time tasks.

Data Construction

The dataset consists of 2,261 human-generated question-answer pairs derived from 1,220 videos across 11 real-world domains (ranging from daily life to professional skills). The construction pipeline involves:

• Video Curation: Videos were crawled from YouTube using procedural queries, strictly filtered to be published after January 2025 to prevent data contamination from pre-trained model knowledge.
• Task Structure: Each sample is a quintuple $T = (V_{tut}, D, V_{tar}, t_c, Q)$:
  • $V_{tut}$: A tutorial video demonstrating the specific skill.
  • $D$: A sequence of distractor videos (unrelated content) inserted between the tutorial and the target.
  • $V_{tar}$: A target video showing a real-world task execution.
  • $t_c$: A precise temporal cutoff (decision point) within $V_{tar}$.
  • $Q$: A multiple-choice question (5 options) posed at $t_c$.
• Temporal Gap: The tutorial is embedded arbitrarily within a stream of 1 to 2 hours of distractor videos before the target task begins. The interval between the relevant knowledge and the query is at least 30 minutes, often up to 2 hours.
• Annotation Rigor: A team of 99 annotators performed sub-second temporal calibration, ensured logical consistency between tutorial and target, and engineered "anti-guessing" questions where common sense or visual cues alone are insufficient to answer correctly without the tutorial.

Task Taxonomy

The benchmark categorizes tasks into three thematic clusters to evaluate different cognitive dimensions:

1. Procedural Mastery: Next step prediction, step ordering, and subgoal prediction.
2. Constraints & Tool Logic: Parameter recall, tool configuration, and safety checks.
3. Diagnostics & Adaptation: Mistake detection and conditional branching (adapting to "if-then" scenarios).

Evaluation Paradigms

The authors evaluate models under three distinct settings to isolate specific capabilities:

1. w/o Tutorial: Only the target video (truncated at $t_c$) is provided. This measures reliance on common-sense priors.
2. w/ Tutorial: The tutorial is immediately followed by the truncated target. This measures prompt application of learned knowledge.
3. w/ Tutorial (Long): The tutorial is followed by a long sequence of distractors (up to 2 hours) before the target. This tests episodic memory and retrieval amidst high-density noise.

Key Results

The benchmark was evaluated on a diverse suite of proprietary (e.g., Gemini 3.1 Pro, GPT-5.2, GPT-4o) and open-source (e.g., video-SALMONN, Qwen-Omni, VideoLLaMA3) video-LLMs.

Performance Observations

• Skill Acquisition Validity: All models showed significant performance gains ($\Delta$) in the "w/ Tutorial" setting compared to the baseline, confirming that the tasks require specific procedural knowledge not available via common sense.
• Memory Decay: In the "w/ Tutorial (Long)" setting, performance degraded markedly for nearly all models.
  • Proprietary Models: While Gemini 3.1 Pro maintained a gain of 14.00% over baseline, models like GPT-5.2 and GPT-4o saw their gains shrink to 2.21% and 2.48% respectively, effectively reverting to baseline performance.
  • Open-Source Models: Many open-source models (e.g., VideoLLaMA3, Qwen3-VL) showed minimal gains ($\Delta_L < 2\%$) in the long-form setting, indicating a failure to retain procedural knowledge over extended distractors.
• Architecture Impact:
  • Audio-Visual vs. Visual-Only: Models with audio capabilities (e.g., Gemini 3.1 Pro, Qwen2.5-Omni) significantly outperformed visual-only counterparts, highlighting the importance of verbal instructions in tutorials.
  • Streaming vs. Non-Streaming: Specialized streaming architectures (e.g., video-SALMONN-S) demonstrated better robustness ($\Delta_L \approx 14.95\%$) compared to non-streaming models, though performance still dropped significantly compared to immediate application.
• Fine-Grained Analysis: Streaming models excelled in logic-intensive tasks like Conditional Branching and Mistake Detection, whereas non-streaming models struggled with diagnostic reasoning, often dropping to near-random performance levels in these categories.

Significance and Claims

The paper positions SLVMBench as a critical tool for identifying the limitations of current video-LLMs in real-time skill acquisition and application.

• Revealing a Key Limitation: The results demonstrate that while video-LLMs can understand short contexts or passively comprehend long videos, they struggle substantially with learning procedural knowledge from long-term memory and transferring it to real-time tasks. The "memory cliff" observed when temporal gaps increase reveals a fundamental bottleneck in current episodic memory mechanisms.
• Benchmarking Standard: SLVMBench fills the gap between passive long-video understanding and real-time streaming perception, providing the first rigorous test for multimodal episodic memory in the context of skill learning.
• Future Direction: The authors note that current limitations (e.g., 2-hour temporal cap due to sequence length constraints) reflect the state of the art. They suggest that future work should explore video demonstrations as memory for agentic AI systems, moving beyond question-answering to actual interaction in simulated or real-world environments.

The paper concludes that existing video-LLMs are not yet ready for embodied AI scenarios requiring long-term procedural memory, and SLVMBench provides the necessary evaluation framework to drive progress in this area.