💻 computer science

See More, Store Less: Memory-Efficient Resolution for Video Moment Retrieval

The paper proposes SMORE, a memory-efficient framework for Video Moment Retrieval that utilizes query-guided captions, importance modulation, and adaptive frame compression to achieve state-of-the-art performance on multiple benchmarks while overcoming the memory constraints of dense frame processing.

Original authors: Mingyu Jeon, Sungjin Han, Jinkwon Hwang, Minchol Kwon, Jonghee Kim, Junyeong Kim

Published 2026-01-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mingyu Jeon, Sungjin Han, Jinkwon Hwang, Minchol Kwon, Jonghee Kim, Junyeong Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a 2-hour movie, but you only have a tiny notepad to write down what happens in it. You need to find a specific scene based on a description like, "The moment the puppy chases the red ball."

The Problem:
Most current AI systems try to solve this in one of two ways, and both have flaws:

The "Sparse Snapshot" approach: They take a few random pictures of the movie every few seconds. If the puppy runs across the screen between those snapshots, the AI misses it completely. It's like trying to read a book by only looking at page 1, page 50, and page 100.
The "Full Description" approach: They try to write a detailed summary of every single second of the movie. This captures everything, but it fills up your notepad (memory) so fast that the computer crashes before it can finish.

The Solution: SMORE (See MORE, store less)
The authors propose a new framework called SMORE. Think of it as a super-smart, memory-efficient assistant who knows exactly what you are looking for before they even start reading the script.

Here is how SMORE works, using three simple tricks:

1. The "Customized Highlighter" (Query-Guided Captions)

Instead of writing generic notes like "A dog is walking," SMORE listens to your specific question first.

Old Way: The AI writes, "A dog is walking." (Generic, might not help you find the specific scene).
SMORE Way: If you ask, "Where is the puppy chasing the ball?", the AI looks at the video and writes, "A puppy is chasing a ball."
The Analogy: Imagine a librarian who, instead of just cataloging every book by its color, reads your request ("I need a book about space") and then writes a special note on the relevant books saying, "This one is about space!" This ensures the notes match exactly what you are looking for.

2. The "Volume Knob" (Query-Aware Importance)

Not all notes are equally important. Some scenes are just background noise; others are the main event.

How it works: SMORE gives a "volume knob" to every note it writes. If a note perfectly matches your search, it turns the volume up (high importance). If a note is about something irrelevant (like a cat sleeping in the background when you asked about a puppy), it turns the volume down.
The Analogy: It's like a DJ mixing a playlist. When the song you want to hear comes on, the DJ cranks the volume. When a song you don't care about plays, they fade it out so it doesn't distract you. This helps the AI focus only on the "loud" (important) parts of the video.

3. The "Smart Compressor" (Structured Visual Compression)

Videos often have many frames that look almost identical (e.g., a car driving on a straight road for 10 seconds). Storing every single frame is a waste of space.

How it works: SMORE looks at consecutive frames. If two frames are nearly the same, it doesn't throw one away (which loses information). Instead, it mathematically "squashes" them together into a single, compact summary that keeps the most important details.
The Analogy: Imagine you have a stack of 100 photos of a sunset where the sun barely moves. Instead of keeping all 100 photos, you take the best 5 and blend them into one high-quality "super-photo" that captures the whole movement without needing 100 separate files.

The Results

The paper tested this system on three major video databases (QVHighlights, Charades-STA, and ActivityNet-Captions).

Performance: SMORE beat the current best models (like Chrono and LLaVA-MR) in finding the right video moments.
Efficiency: It did this while using less memory. While other top models needed a massive, expensive computer chip (80GB of memory) to run, SMORE could achieve better results on a smaller, more common chip (48GB of memory).

In a Nutshell:
SMORE is like a detective who doesn't just read the whole case file blindly. Instead, they read the specific clue you give them, highlight the relevant pages, turn down the noise, and summarize the boring parts so they can solve the mystery faster and with less paper.

Technical Summary: SMORE (See MORE, store less)

Problem Statement

Video Moment Retrieval (VMR) requires identifying specific temporal segments within a video that semantically correspond to a natural language query. While Multimodal Large Language Models (MLLMs) have advanced video understanding, VMR remains challenging due to memory constraints associated with processing densely sampled video frames over extended durations.

Existing methodologies face a trade-off between information resolution and memory efficiency:

Sparse Sampling: Traditional approaches often rely on selecting a limited number of keyframes to fit within memory budgets. This risks missing critical, ephemeral visual cues, leading to information loss, particularly in lengthy videos.
Dense Processing with Generic Captions: Alternative strategies attempt to condense visual content into textual captions to reduce memory load. However, conventional zero-shot captioning generates generic descriptions that lack alignment with specific user intents, resulting in sub-optimal retrieval performance.
Memory Bottlenecks: Recent MLLM-based VMR systems (e.g., Chrono, LLaVA-MR) improve temporal awareness but often require substantial GPU memory (e.g., 80GB A100), limiting their accessibility and scalability.

Methodology: SMORE Framework

The authors propose SMORE (See MORE, store less), a framework designed to enhance memory efficiency while maintaining high information resolution. SMORE operates within an interleaved token architecture where video frames are combined with timestamps and enriched with query-guided textual captions. The framework consists of three core components:

1. Query-Guided Caption Generation

To address the misalignment between generic captions and user queries, SMORE introduces a two-stage captioning process:

Query Parsing: The input query is parsed to extract specific objects and actions.
QA-Based Relevance Check: A Question-Answering (QA) module evaluates whether specific objects or actions appear in a given video scene.
Dual-Path Captioning:
- Relevant Scenes: If a scene contains query-relevant elements, a "query-aware prompt" is used to generate a caption specifically aligned with the user's intent (e.g., focusing on "cooking" and "facetime" if those are in the query).
- Irrelevant Scenes: Scenes failing the relevance check receive standard, generic captions.
  This ensures that the model receives dense semantic coverage without generating distracting or misleading text for irrelevant segments.

2. Query-Aware Importance Modulation

Not all generated captions contribute equally to the retrieval task. SMORE employs a weighting mechanism to prioritize informative content:

A similarity score $S_q(f_i, c_i)$ is calculated for each frame-caption pair based on the cosine similarity between the frame and the query, and the query and the caption (incorporating frame-caption similarity).
Caption embeddings are re-weighted by this score before entering the LLM's self-attention layer. This allows the model to focus on segments with high semantic relevance to the query and down-weight background or redundant information.

3. Structured Visual Compression (SVC)

To reduce spatial redundancy without losing temporal fidelity, SMORE utilizes a low-overhead compression strategy based on Singular Value Decomposition (SVD):

Redundancy Detection: Consecutive frame embeddings are compared using cosine similarity against a current anchor frame.
SVD Compression: If the similarity exceeds a threshold $\theta$ , the frames are stacked and compressed using truncated SVD (Rank-1 approximation). The resulting average row serves as a compact embedding that retains dominant spatial characteristics while discarding lower-order redundancies.
Anchor Update: If similarity is low, the current frame is retained as the new anchor.
This approach preserves essential visual dynamics and temporal order better than simple frame selection or average pooling.

Key Contributions

Memory-Efficient VMR Framework: SMORE achieves state-of-the-art performance on standard benchmarks while operating within restricted memory environments (48GB A6000), contrasting with methods requiring 80GB A100 GPUs.
Query-Guided Semantic Abstraction: The introduction of query-guided captioning and importance modulation bridges the gap between dense visual coverage and specific user intent, improving retrieval precision without increasing visual token counts.
Structured Visual Compression: The proposed SVD-based compression method effectively reduces memory usage by eliminating redundant frames while preserving key visual evidence, outperforming naive frame selection and average pooling.

Experimental Results

The authors evaluated SMORE on three representative datasets: QVHighlights, Charades-STA, and ActivityNet-Captions.

QVHighlights: SMORE achieved a new state-of-the-art (SOTA) performance. It improved upon the high-precision model SG-DETR by 4.19% in R1@0.5 and 6.24% in R1@0.7, while also maintaining a higher mean Average Precision (mAP) average (+0.62%). It also surpassed the high-recall model LLaVA-MR in both recall and mAP metrics.
Charades-STA: SMORE set a new SOTA, outperforming LLaVA-MR by 1.12% in mIoU and 0.61% in R1@0.5.
ActivityNet-Captions: The model achieved gains of 1.15% in R1@0.5 and 0.66% in R1@0.7 over the previous SOTA.
Ablation Studies: Progressive ablation confirmed that each component (zero-shot captioning, query-guided captioning, importance modulation, and SVC) contributes synergistically to the final performance, with the full model showing a total improvement of +2.86% in mAP Average over the baseline.
Memory Efficiency: Experiments demonstrated that SMORE consistently outperforms frame-only baselines across various memory budgets (30GB to 45GB), proving its ability to scale to longer videos with richer multimodal information under tight constraints.

Significance and Claims

The paper claims that SMORE demonstrates that high retrieval accuracy and memory efficiency are not mutually exclusive. By leveraging query-guided semantic abstraction and structured visual compression, the framework enables the deployment of powerful video-language models on more accessible hardware (e.g., 48GB GPUs) without compromising fine-grained temporal understanding.

The authors position SMORE as a solution to the fundamental trade-off in VMR between the need for dense temporal reasoning and the limitations of current memory resources. They note that while the framework introduces a slight inference latency due to its modular pipeline, it offers a flexible architecture that can operate in Storage-Efficient (on-demand captioning) or Latency-Efficient (pre-computed captions) modes, making it adaptable for diverse real-world deployments.

The paper concludes by acknowledging limitations, such as performance degradation with highly ambiguous queries or videos, and suggests future work could explore decoder-only models to extend the framework's generalizability.