CoVR-R:Reason-Aware Composed Video Retrieval

1. Problem Definition

Composed Video Retrieval (CoVR) is the task of retrieving a target video ($V_t$) from a gallery given a reference video ($V_r$) and a textual modification instruction ($E$).

The Core Challenge:
Existing CoVR methods largely treat the modification text as a set of literal keywords or direct visual constraints. They often fail to account for implicit after-effects—the causal, temporal, and cinematographic consequences that naturally follow an edit but are not explicitly stated in the text.

• Example: If the edit says "change the cow to a horse," a literal model might just look for a horse. However, a successful retrieval requires reasoning that the grazing behavior changes, the tail movement differs, and the scene context (e.g., terrain) might shift to match the new animal's habitat.
• Gap: Current models miss edits where success depends on predicting state transitions, action phases, camera movements, or temporal ordering, leading to poor performance on complex, real-world queries.

2. Methodology: Reason-Then-Retrieve

The authors propose a zero-shot, reasoning-first framework that leverages Large Multimodal Models (LMMs) to infer consequences before performing retrieval. The architecture operates in two stages (illustrated in Figure 3 of the paper):

Stage 1: After-Effect Reasoning Trait Generation

Instead of directly embedding the edit text, the system uses a frozen LMM (specifically Qwen3-VL-8B) to generate a structured reasoning trace ($R$) based on the reference video ($V_r$) and the edit ($E$).

• Input: $V_r$ and $E$.
• Process: The LMM is prompted to predict specific visual consequences across five dimensions:
  1. States: Object state transitions (e.g., raw $\to$ cooked).
  2. Actions/Phases: Temporal ordering of sub-actions.
  3. Scene: Background and environmental changes.
  4. Camera: Shot scale, framing, and motion (pan/zoom).
  5. Tempo: Pacing and duration changes.
• Output: A compact, schema-constrained reasoning record $R$.

Stage 2: Target Video Embedding Generation & Retrieval

The system converts the reasoning trace into a query for retrieval.

1. Target Description Generation: The LMM is prompted with ($V_r$, $E$, $R$) to generate a natural language description of what the target video should look like.
2. Embedding Extraction:
   • The generated description is tokenized.
   • Token embeddings are extracted from the LLM's final transformer layer.
   • Importance-Weighted Pooling: Instead of simple averaging, the method applies a weighted mean pooling strategy. Tokens representing actions, objects, and states receive higher weights ($\alpha_{high}$), while generic function words receive lower weights ($\alpha_{low}$). This creates a dense video embedding vector $q(V_r, E)$.
3. Retrieval: The query embedding is compared against pre-computed gallery embeddings (generated via the same weighted pooling process on video descriptions) using cosine similarity to rank and retrieve the target video.

Key Design Choice: The LMM backbone remains frozen; no task-specific fine-tuning is performed. The system relies entirely on the LMM's emergent reasoning capabilities.

3. Key Contributions

A. The CoVR-R Benchmark

The authors introduce CoVR-R (Composed Video Retrieval with Reasoning), a new benchmark designed to evaluate reasoning capabilities rather than keyword matching.

• Scale: 2,800 high-quality triplets ($V_r, E, V_t$).
• Data Sources: Derived from Dense-WebVid-CoVR and Something-Something V2.
• Unique Features:
  • Structured Reasoning Traces: Every triplet is paired with ground-truth reasoning annotations covering states, actions, scene, camera, and tempo.
  • Hard Distractors: Negative samples are specifically crafted to match keywords but fail on causal/temporal logic (e.g., correct objects but wrong phase order).
  • Evaluation Metrics: Includes standard Recall@K and a novel Reasoning Score (0–10) evaluated by an LLM judge to measure alignment with ground-truth causal chains.

B. The Reason-Aware Framework

A novel pipeline that decouples reasoning from retrieval, allowing general-purpose LMMs to drive video search without expensive retraining. It explicitly models the "gap" between the edit instruction and the visual reality.

C. Comprehensive Evaluation

The paper provides extensive ablation studies on model scaling, reasoning granularity, and pooling strategies, demonstrating that structured reasoning is superior to raw text fusion for complex edits.

4. Experimental Results

Performance on CoVR-R (Reasoning-Centric)

• Zero-Shot Superiority: The proposed method (Qwen3-VL-8B) achieves 49.88% R@1, outperforming the strongest baseline (BSE-CoVR) by +16.45 percentage points.
• Reasoning Score: The method achieves a reasoning score of 8.31/10, significantly higher than baselines (~6.0), confirming it correctly predicts implicit effects.
• Implicit-Effect Subsets: The model excels specifically on subsets requiring temporal and cinematographic reasoning, where keyword overlap is insufficient.

Performance on Dense-WebVid-CoVR (Standard Benchmark)

• The method also outperforms state-of-the-art on the standard Dense-WebVid-CoVR benchmark, achieving 61.21% R@1 (vs. 48.08% for the best baseline), demonstrating that reasoning capabilities generalize even to datasets with less explicit reasoning requirements.

Ablation Insights

• Model Scaling: Performance scales with model size (Qwen3-VL-72B > 8B > 4B), validating the approach's reliance on LMM capacity.
• Reasoning Granularity: There is a "sweet spot" for reasoning detail.
  • Minimal traces (keywords only) fail to capture causality.
  • Verbose traces (too much detail) introduce noise and degrade retrieval performance.
  • Standard structured traces (approx. 89 tokens) offer the optimal balance between semantic richness and discriminative signal clarity.
• Pooling Strategy: Importance-weighted pooling significantly outperforms simple mean, max, or last-token pooling, proving that semantic weighting is crucial for video representation.

5. Significance and Impact

• Paradigm Shift: Moves the field from "edit-literal" matching (keyword overlap) to "effect-reasoned" retrieval (causal prediction).
• Zero-Shot Efficiency: Demonstrates that high-performance CoVR can be achieved without task-specific fine-tuning by leveraging the reasoning capabilities of frozen LMMs, reducing the need for massive annotated datasets.
• Explainability: The structured reasoning traces provide an interpretable chain of thought, making the retrieval process more transparent and debuggable compared to black-box embedding fusion.
• Future Direction: Highlights the need for adaptive systems that can switch between keyword matching (for simple edits) and deep reasoning (for complex edits) to optimize efficiency.

In summary, CoVR-R establishes that successful composed video retrieval requires a "reasoning-first" approach to bridge the gap between textual instructions and their complex visual consequences, achieving state-of-the-art results through a zero-shot, LMM-driven pipeline.