💻 computer science

Persistent Object Narratives for Token-Efficient Video Language Models

The paper introduces SlotNarrative, a token-efficient video language model interface that organizes video content into persistent object narratives using compact identity and state tokens, achieving a favorable accuracy-to-token trade-off by decoupling object grouping from frame-wise compression.

Original authors: Junzhe Chen, Siyuan Meng, Xiaojie Guo

Published 2026-08-06

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Junzhe Chen, Siyuan Meng, Xiaojie Guo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to understand a movie. You have a giant library of books (the robot's brain), but it can only read a few pages at a time before it gets overwhelmed. If you try to show the robot every single frame of a movie—thousands of pictures per minute—it chokes. It gets lost in the sheer volume of pixels and forgets that the character running in the first scene is the same one jumping in the last scene. This is the big puzzle scientists are solving in the field of Video Large Language Models. These are AI systems designed to "watch" videos and answer questions about them, but they struggle to keep track of objects over time without using up all their memory. The key challenge is finding a way to summarize a long video into a tiny, efficient story that the robot can actually read, without losing the plot.

Enter SlotNarrative, a new method proposed by researchers at Tianjin University that acts like a clever editor for these AI robots. Instead of feeding the robot a chaotic pile of thousands of video frames, SlotNarrative organizes the video into "persistent object narratives." Think of it like this: if you were describing a soccer game to a friend who missed the whole match, you wouldn't list every single second of the game. Instead, you'd say, "There was a striker named Alex who ran down the field, then he passed the ball, then he scored." You are tracking the story of Alex, not the pixels of his jersey.

The paper suggests that SlotNarrative works by first grouping visual features into "slots"—little buckets that catch things that look like objects. Then, it uses a special, invisible memory system to link these buckets together over time. Even if the object disappears behind a tree or the camera cuts away, the system remembers, "Ah, that's still Alex!" Finally, it writes a tiny, fixed-size report for the robot. This report has two parts: an "Identity Token" (a permanent ID card for the object, like "Alex the Striker") and a set of "State Tokens" (a diary of what happened to him in different parts of the video).

The researchers found that this method is incredibly efficient. They managed to squeeze the entire visual story of a video into just 144 "token" positions (the units of information the robot reads). This is a tiny fraction of the thousands of tokens other methods use. Despite using so little space, the system performed very well on standard video quizzes, often beating other compact methods. The paper suggests that by separating the "who" (identity) from the "what happened" (state), the robot can understand videos much better without getting confused by the sheer amount of data. However, the authors also note that while this works great for tracking objects, it sometimes struggles with very complex, long-range timing or crowded scenes where objects get mixed up, showing that there is still work to be done to make these AI editors perfect.

Technical Summary: Persistent Object Narratives for Token-Efficient Video Language Models

Problem Statement

Video Large Language Models (Video-LLMs) have advanced open-ended video understanding but face two primary limitations in their visual interfaces:

Token Intensity: Current frame-wise construction methods generate visual tokens proportional to temporal and spatial resolution, often resulting in thousands of tokens per video (e.g., LLaVA-OneVision, LLaVA-Video).
Lack of Explicit Object Structure: Existing compression techniques (e.g., sparse memory, adaptive spatiotemporal compression) reduce token counts but fail to explicitly organize recurring object evidence. Consequently, the language model must infer object correspondence and state evolution from compressed sequences without explicit binding, leading to ambiguity when objects reappear after occlusion or scene cuts.

While object-centric learning offers a theoretical basis for grouping features into object-like slots, existing slot-based approaches typically treat slots as local and exchangeable, failing to link recurring observations to persistent object instances across time. Conversely, methods relying on external detection and tracking pipelines introduce dependency on non-learned components.

Methodology: SlotNarrative

The authors propose SlotNarrative, a slot-based, object-centric interface that organizes video evidence into persistent object narratives. The framework operates through three functional modules:

1. Object-Centric Video Representation

A frozen SigLIP encoder extracts patch features from sampled frames. A feature adapter and a recurrent slot encoder group these features into $K=11$ object-like slot vectors ( $S_t$ ) and patch-normalized attention maps ( $A_t$ ).

Local State: The system estimates local motion, geometry, and visibility using mask-weighted patch matching and a causal trajectory encoder.
Scene Handling: At detected scene cuts, recurrent slot initialization is reset to prevent treating edits as extreme object motion, while retaining clip-level memory.

2. Persistent Object Memory

A parameter-free memory module links frame-local slot observations to persistent, clip-level object entries.

Re-identification: When a slot transitions from invisible to visible, the system matches it against inactive memory entries using five complementary cues: slot-feature similarity, pooled-appearance similarity, trajectory-state similarity, recent-observation similarity, and position proximity.
Matching Logic: A weighted, parameter-free score determines re-identification. A greedy one-to-one matching strategy prevents duplicate claims. If an object reappears under a different raw slot index, the memory preserves the persistent ID, effectively bridging missing intervals.
No Trainable Parameters: The memory itself contains no learnable weights; it relies on fixed cue weights (e.g., 0.3 for slot features, 0.2 for appearance) and thresholds.

3. Typed Object-State Tokenization

To interface with a frozen language model, retained object entries are serialized into a fixed-size visual sequence:

Inventory Selection: Entries are ranked by visible duration, state change, and "objectness" (a metric combining slot area and attention entropy). The top $M=16$ entries are retained.
Token Decomposition: Each retained entry is represented by:
- One Identity Token: Summarizes persistent appearance using time-aggregated, visibility-weighted backbone features.
- $J=8$ State Tokens: Encode segment-level appearance, geometry, visibility, trajectory state, and re-identification events.
Fixed Budget: This design allocates exactly $M(1 + J) = 144$ visual-token positions per video, independent of the number of sampled frames. Unused positions are masked.

Training Strategy

The model is trained in three sequential stages:

Stage 1: Learns object-like grouping and local consistency using feature reconstruction, SlotContrast, and Synergistic Representation Learning on unlabeled clips.
Stage 2: Trains trajectory and presence modules via future-observation prediction and confidence-derived targets.
Stage 3: Aligns the typed object-state representation with the language model (Qwen2-7B-Instruct). Only the identity/state projectors, layer normalizations, and typed embeddings are trained; the visual backbone, memory, and language model remain frozen.

Key Contributions

Slot-Based Object-Centric Interface: Organizes video evidence around persistent clip-level object entries rather than frames or exchangeable per-frame slots, using a parameter-free memory to associate recurring observations without external tracking pipelines.
Typed Object-State Representation: Separates persistent object appearance from time-varying state. This decomposition yields a compact, fixed interface of 144 tokens (1 identity + 8 state tokens $\times$ 16 objects) for a frozen Video-LLM.
Efficiency and Performance: Demonstrates a favorable accuracy–visual-token trade-off across multiple benchmarks, establishing persistent object narratives as a structured visual interface.

Experimental Results

The authors evaluated SlotNarrative on three open-ended VideoQA benchmarks: MSVD-QA, MSRVTT-QA, and ActivityNet-QA.

Token Efficiency: With only 144 allocated visual-token positions, SlotNarrative achieves:
- 75.6% accuracy on MSVD-QA.
- 69.8% accuracy on MSRVTT-QA.
- 50.3% accuracy on ActivityNet-QA.
Comparisons:
- Compared to the closest object-centric baseline, Slot-VLM (192 tokens), SlotNarrative uses 25% fewer tokens while improving accuracy by 0.7, 0.2, and 2.0 points respectively.
- It outperforms dense token methods (e.g., LLaVA-OneVision with ~6,000 tokens) on short-video datasets and remains competitive on longer videos (ActivityNet-QA) despite the significant token reduction.
Ablation Studies:
- Trajectory and Memory: Both components significantly improve performance. Removing memory alone drops accuracy by ~3.2 points on MSVD-QA; removing trajectory drops it by ~1.8 points. Their combination yields the best results.
- Typed Tokens: Separating identity and state tokens is crucial. Using only identity tokens (16 tokens) or only state tokens (128 tokens) results in significant accuracy drops (44.1% and 42.4% respectively on ActivityNet-QA) compared to the full 144-token interface (50.3%).
- Budget Sensitivity: Reducing object coverage ( $M$ ) is more costly than reducing temporal resolution ( $J$ ). Increasing the budget beyond 144 tokens yields diminishing returns.

Significance and Claims

The paper claims that SlotNarrative establishes a new paradigm for Video-LLM interfaces by prioritizing persistent object narratives over frame-wise evidence. The significance lies in:

Structured Compression: It provides a compact, structured, and temporally organized visual interface that explicitly links recurring observations, reducing the burden on the language model to infer object correspondence.
Token Efficiency: It achieves competitive performance with a fixed, low token budget (144 tokens), decoupling input length from video duration and resolution.
Learned Association: It demonstrates that object-like units can be discovered and associated across missing intervals directly from visual features without relying on external detection or tracking models.

The authors modestly acknowledge limitations, noting that while the interface is compact, long-range temporal order and fragmentation in crowded scenes remain challenges. They suggest that future work should focus on richer object discovery and motion cues without reverting to long frame-token sequences.