EMCompress: Video-LLMs with Endomorphic Multimodal Compression

1. Problem Statement

Video Large Language Models (Video-LLMs) face a fundamental bottleneck in long-video reasoning:

• The Tension: Current models typically use static, sparse frame sampling (uniform sampling) to fit long videos into a limited token budget. This approach either dilutes semantic evidence across irrelevant segments or misses fine-grained temporal semantics entirely.
• The Limitation of Existing Solutions: Traditional "grounding" methods attempt to retrieve query-relevant frames but operate at a perceptual level (matching visual tokens to text). They fail to address semantic mismatches in complex queries involving multi-hop temporal, relational, or causal dependencies. Retrieving a frame without adapting the query often leaves the downstream model with a mismatched context.
• The Core Issue: The current paradigm forces the model to process the entire video stream, introducing "cognitive noise" and reducing the density of relevant evidence per token.

2. Proposed Solution: Endomorphic Multimodal Compression (EMC)

The authors propose a novel task called Endomorphic Multimodal Compression (EMC). Inspired by human cognitive mechanisms (filter-then-reason), EMC aims to compress the input $(V, Q)$ into a compact, sufficient statistic $(v, q)$ while preserving the answer invariance.

Key Definitions & Formulation

• Endomorphic Transformation: Unlike latent-code compression (e.g., Information Bottleneck) which maps inputs to an abstract latent space, EMC maps the multimodal input back into the same task space.
  • Transformation: $F_{EMC}: (V, Q) \to (v, q)$
  • Where $v$ is a trimmed video sub-segment and $q$ is a rewritten query.
• Sufficient Statistic: Under the Markov chain $A \to (V, Q) \to (v, q)$, the goal is to find the most compact $(v, q)$ such that $I((v, q); A) = I((V, Q); A)$. The compressed pair must contain all information necessary to derive the ground-truth answer $A$.
• Admissibility Conditions:
  1. Structural Continuity (C1): The output video $v$ must be a concatenation of non-overlapping contiguous sub-segments to preserve motion and scene dynamics.
  2. Answer Sufficiency (C2): For any reasonable downstream VideoQA agent $M$, $M(q, v) = M(Q, V)$. The answer must remain correct at the instance level.
• Minimality Objectives:
  1. Video-side: Minimize the size (duration/frame count) of $v$.
  2. Query-side: Minimize the reasoning steps required to derive $A$ from $(v, q)$.
  • Resolution: The objectives are resolved lexicographically with video priority; video compression drives the transformation, and query rewriting is the downstream consequence.

3. Methodology: ReSimplifyIt Framework

To realize EMC, the authors introduce ReSimplifyIt, a plug-and-play, model-agnostic multi-agent framework. It operates via an iterative, feedback-driven loop involving three main agents:

1. Launcher (Video-Free):

   • Generates a high-level trimming instruction (e.g., "keep the clip after event X") and a rewritten query based only on the text query and previous history.
   • Uses Success History (SH) and Failure History (FH) to self-correct and avoid repeating failed plans.
   • Operates via abductive reasoning on language without direct video access initially.

2. Validator (Execution & Feedback):

   • Receives the instruction from the Launcher.
   • Determines feasibility by interacting with the Viewer.
   • If the instruction is feasible, it executes the trim; if not, it provides constructive feedback to the Launcher.
   • Embodies a "competence from consequence" philosophy: plans are validated by execution, not just rules.

3. Viewer (Scanning & Localizing):

   • Scanning: Summarizes video snippets given a timestamp range.
   • Localizing: Retrieves timestamp ranges given a text description.
   • Implementation: Uses a two-stage keyframe extraction (MPEG-4 I-frames + dynamic clustering) and an LLM for captioning. It supports bidirectional navigation (top-down and bottom-up).

Workflow: The process iterates. The Launcher proposes a plan $\to$ Validator checks feasibility via Viewer $\to$ If failed, feedback is recorded, and the Launcher retries. If successful, the process terminates with the compressed $(v, q)$.

4. Key Contributions

1. EMC Task Formulation: The first formalization of video compression as an endomorphic problem that jointly reshapes the video and query to preserve answer invariance, distinct from simple temporal grounding.
2. EMCompress Benchmark: A new benchmark built on YouCookII containing 238.2 hours of video and 2,754 questions. It includes dual labels for both the compression task (video trimming + query rewriting) and standard VideoQA.
3. ReSimplifyIt Framework: A robust, multi-agent baseline that outperforms existing temporal localization methods by 0.40 F1 on the benchmark.
4. Empirical Validation: Demonstrated that EMC significantly improves both inference (robustness) and training (alignment) for Video-LLMs.

5. Experimental Results

The authors evaluated EMC across multiple Video-LLMs (e.g., Video-ChatGPT, LLaVA-NeXT, InternVL) and benchmarks (ActivityNet-QA, MLVU, Video-MME, etc.).

• Inference Performance: Integrating EMC as a front-end yielded relative gains of 33.7% in inference accuracy across benchmarks. For example, on ActivityNet-QA, Video-ChatGPT improved from 50.5 to 58.5.
• Training Performance: Training Video-LLMs on data pre-processed by EMC (using ground truth or predicted compression) resulted in relative gains of 7.33% compared to training on raw data. This confirms that EMC purifies the supervision signal.
• Efficiency & Density:
  • EMC achieves massive density amplification. On the LVBench dataset, sampling 8 frames from the compressed segment is equivalent to sampling 2,566 frames from the original video.
  • Cost Ratio: EMC reduces the effective frame-sampling cost by up to 100x (CostRatio $\approx$ 0.01) while maintaining or improving accuracy.
• Ablation Studies:
  • Removing modular design (ReSimplifyIt-simple) slightly reduces performance but maintains high efficiency.
  • Removing video access entirely (ReSimplifyIt-blind) causes a significant drop, highlighting the necessity of the feedback loop between text and vision.

6. Significance and Impact

• Cognitive Alignment: EMC mirrors human cognition by filtering redundancy before deep reasoning, addressing the "superfluous cognition noise" in current Video-LLMs.
• Paradigm Shift: It moves beyond "retrieving evidence" to "reconstructing the problem." Instead of asking the model to find the answer in a noisy video, EMC asks the model to solve a cleaner, semantically aligned version of the problem.
• Scalability: As a model-agnostic front-end, EMC can be plugged into any existing VideoQA pipeline without architectural changes, offering a scalable path toward interpretable and efficient video-language understanding.
• Future Direction: The paper identifies spatial compression and end-to-end integration (where the encoder learns to compress) as the next frontiers.

In summary, EMCompress establishes that information compression is a critical, under-explored mechanism for advancing Video-LLMs, offering a principled way to handle long-context reasoning by transforming the input space rather than just expanding model capacity.