Technical Summary: VideoHarness-RSI

Problem Statement

Long-video understanding remains a significant challenge for Vision-Language Models (VLMs), even when the underlying models are powerful. The core bottleneck is not necessarily the model's inference capability, but rather context construction: how to select and organize a limited set of visual observations from a video containing thousands of irrelevant frames.

Existing approaches address this via compression, retrieval, memory, or agentic evidence acquisition. However, these systems typically modify multiple components simultaneously (e.g., changing the model, the retrieval mechanism, and the reasoning workflow together). This makes it difficult to isolate a specific question: How much performance gain can be achieved solely by improving the executable program that constructs the context, while keeping the VLM and its interface completely fixed?

Methodology: VideoHarness-RSI

The paper introduces VideoHarness-RSI (Recursive Harness Self-Improvement), a controlled framework for recursively searching for optimal executable context constructors (harnesses) around a frozen VLM.

Core Framework

The system treats the harness as a program $H$ that maps a video-question pair $(V, q)$ to a bounded multimodal context $C_H$, which is then fed to a frozen VLM $M$ to generate an answer $\hat{y}$.
$$\hat{y} = M(H(V, q; K), q)$$
The optimization objective is to maximize development accuracy over the space of executable programs $H_{exec}$, subject to a fixed context constraint $K$.

The Harness Decomposition

The authors analytically decompose any harness into three functional stages, though these are not separately optimized objectives but rather a path for program mutation:

1. Write ($Write_H$): Constructs an addressable representation of the video (e.g., a stream, caption list, or embedding index).
2. Read ($Read_H$): Retrieves evidence conditioned on the question $q$.
3. Pack ($Pack_H$): Orders and formats the bounded context for the VLM.

Recursive Search Protocol

The search operates via an outer loop that proposes, executes, and evaluates harness mutations:

1. Proposer: An LLM-based proposer (Claude Opus 4.6) generates candidate harness code based on a frontier of current best programs ($F_t$) and an archive ($A_t$) of historical code, scores, and execution traces.
2. Evaluation: Candidates are "smoke-tested" and evaluated end-to-end on a development set ($D_{dev}$).
3. Selection: The update rule is strict. A candidate $H_{t,j}$ is promoted to the frontier $F_{t+1}$ only if its development accuracy $S(H_{t,j})$ strictly exceeds the current frontier's accuracy.
4. Constraints: The VLM (Qwen3-VL-8B-Instruct), its decoding configuration, and the evaluation metrics remain fixed throughout the search. The search does not touch the model parameters.

Experimental Setup

• Dataset: LVBench (1,232 QA pairs after filtering). 350 questions used for search/development; 882 reserved for held-out evaluation.
• Baselines: Compared against prespecified uniform sampling, literature-inspired hand-crafted harnesses (e.g., AKS, WorldMM-style, Homer-style), and proprietary VLMs under uniform sampling.
• Cost Metric: Input-side context cost is measured as the sum of visual and auxiliary text tokens per question, excluding offline indexing costs.

Key Results

1. Performance Gains from Search

Recursive search consistently improves performance over both weak and strong baselines:

• From Uniform Sampling: Starting from a random uniform sampling baseline (36.3% accuracy), the search discovered EMBEDNAVIGATE-HYBRID, which achieved 45.4% on the held-out set.
• From Strong Hand-Crafted Baselines: Starting from a strong hand-crafted baseline (AKS, 46.5%), the search discovered TIMESTAMPED-AKS, which improved the held-out accuracy to 50.3%.
• Cross-Benchmark Transfer: The harnesses selected on LVBench were frozen and applied directly to Video-MME and MLVU without further search or adaptation. They outperformed uniform sampling on both benchmarks (e.g., 61.5% vs. 59.9% on Video-MME), suggesting the discovered policies are transferable.

2. Mechanism of Improvement

Analysis of the discovered harnesses reveals distinct strategies:

• Navigation vs. Retrieval: The search discovered that combining temporal navigation (using captions to focus sampling on relevant time regions) with visual similarity retrieval (using CLIP embeddings) yields better results than either alone.
• Evidence Visibility: The hybrid harness improved accuracy not just by exposing more annotated evidence intervals (increasing "visible" frames), but also by improving performance on cases where evidence was missed, suggesting better context organization and density matters.
• Question Type Sensitivity: Navigation strategies were particularly effective for temporal and reasoning questions, while visual retrieval aided entity and key-information questions.

3. Cost-Accuracy Trade-offs

The search revealed a Pareto frontier between accuracy and input token cost.

• EMBEDNAVIGATE-HYBRID achieved high accuracy but incurred a significant text overhead due to an additional navigation pass.
• TIMESTAMPED-AKS achieved the highest accuracy with a smaller input footprint, demonstrating that search can find efficient configurations that hand-crafted baselines might miss.

Significance and Claims

The paper positions VideoHarness-RSI as a controlled baseline for studying automated harness design. Its primary contributions and claims are:

1. Isolation of Variables: It establishes that executable context construction is a distinct optimization layer. Significant gains can be achieved by optimizing the "harness" (the program managing what the model sees) without retraining the model or changing its interface.
2. Reproducibility: The authors provide an auditable archive of candidate code, parentage, execution traces, and scores, offering a reproducible baseline for future research in harness discovery.
3. Transferability: The results demonstrate that harnesses discovered on one benchmark can transfer to others without re-search, indicating that the search discovers generalizable context-construction policies rather than overfitting to specific dataset quirks.
4. Limitations: The authors modestly note that the search frontier often plateaus after early improvements and that the "strict" point-estimate selection does not statistically guarantee generalization, though empirical results on held-out sets support the approach. They also clarify that their cost metrics reflect input token volume, not end-to-end latency or monetary cost, as offline indexing costs vary.

In summary, the paper argues that for long-video understanding, the "system" surrounding a frozen VLM is a critical, optimizable component that can be improved through recursive, automated program search.