💻 computer science

Scaling Video Understanding via Compact Latent Multi-Agent Collaboration

The paper introduces MACF, an end-to-end multi-agent collaboration framework that enables scalable and high-fidelity long-video understanding by decoupling local perception budgets from global complexity through a novel agent-native latent communication protocol and curriculum training strategy.

Original authors: Kerui Chen, Jinglu Wang, Jianrong Zhang, Ming Li, Yan Lu, Hehe Fan

Published 2026-05-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kerui Chen, Jinglu Wang, Jianrong Zhang, Ming Li, Yan Lu, Hehe Fan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a two-hour movie, but your brain (or your computer) can only hold a few minutes of visual information in its memory at once. If you try to watch the whole thing at full speed, you get overwhelmed. If you try to watch it by only glancing at a few random frames, you miss the plot.

This is the problem MACF (Multi-Agent Collaboration Framework) solves. It's a new way for AI to watch and understand long videos without getting "brain fog" or losing important details.

Here is how it works, using a simple analogy:

The Problem: The "Overloaded Detective"

Imagine a single detective (a standard AI model) trying to solve a mystery in a massive, 100-room mansion (a long video).

The Limit: The detective can only look at one room at a time and has a strict limit on how many photos they can carry in their pocket.
The Old Way (Sampling): To fit the whole mansion in their pocket, the detective takes one blurry photo from every room. They miss the clues hidden in the corners.
The Other Old Way (Retrieval): The detective asks a secretary to write a summary of every room. But the secretary might miss a crucial detail (like a specific color of a rope) or write it down poorly, leading to a wrong conclusion.

The Solution: The "Specialized Team"

MACF changes the game by hiring a team of detectives instead of relying on one.

Splitting the Work: The mansion is divided into sections. Detective A watches the first 10 minutes, Detective B watches the next 10, and so on. Each detective only needs to remember a small, manageable chunk of the video. They don't have to sacrifice detail because their "memory budget" is only for a short clip.
The Secret Handshake (Latent Communication): This is the paper's big innovation.
- Old Teams: The detectives would write notes to each other. "I saw a red rope." But words are clumsy. If Detective A saw a "brown-and-white dog" and wrote "white dog," Detective B might get confused about which dog they are talking about.
- MACF Team: Instead of writing notes, the detectives pass each other compact, high-tech "memory chips" (latent tokens). These chips don't use words; they contain the raw feeling and visual essence of what was seen. They can say, "Here is the exact visual pattern of the rope," without losing the color or texture in translation.
The Commander: A central Commander (the Coordinator Agent) receives these memory chips from all the detectives. Because the chips are in a shared, efficient language, the Commander can piece together the whole story perfectly, even though no single detective saw the whole movie.

How They Learned to Work Together (The Training)

You can't just hire a team and expect them to work instantly. The paper describes a three-step training camp to teach them:

Learning the Language: First, they practice passing chips that describe simple captions (like "a dog is running"). This ensures everyone speaks the same "visual language."
Learning to Focus: Next, they practice looking at a picture and a question, then passing a chip that only contains the answer to that specific question. They learn to ignore irrelevant details.
Teamwork Drills: Finally, they practice with the full video. The Commander learns how to take chips from Detective A, Detective B, and Detective C, and combine them to solve the mystery.

Why It's Better

The paper tested this against the best AI models available.

Accuracy: When the video is long, MACF gets the right answer much more often than the "single detective" or teams that use text notes.
No Lost Details: In one test, a text-based team failed to identify the color of a dog's leash because the description was too vague. MACF's "memory chips" kept the visual detail sharp, leading to the correct answer.
Efficiency: It's faster and uses less computing power than other methods that try to send huge amounts of data between agents.

The Bottom Line

MACF is like upgrading from a single person trying to memorize an entire library to a team of librarians who each read a few books and pass tiny, perfect summaries to a head librarian. It allows AI to understand long, complex videos without getting tired, without losing details, and without needing a supercomputer to do it.

1. Problem Statement

Current Multimodal Large Language Models (MLLMs) face a fundamental bottleneck in long-video understanding due to bounded perception context budgets.

The Limitation: A single MLLM cannot process high-resolution, long-duration videos without aggressive downsampling (temporal/spatial), which leads to significant information loss (e.g., missing specific objects or temporal events).
Failure of Existing Agentic Methods: Existing multi-agent approaches attempt to mitigate this via rule-based preprocessing (e.g., sampling) or retrieval. However, these often rely on text-based communication between agents. Text is lossy for visual tasks, failing to preserve fine-grained details like motion, spatial relations, and textures. Furthermore, text-based pipelines are often expensive and rely on intermediate text quality, creating a disconnect from native visual representations.
The Core Challenge: How to distribute the perceptual workload across multiple agents under strict local budgets while enabling efficient, information-preserving global coordination without relying on lossy textual summaries.

2. Methodology: MACF Framework

The authors propose MACF (Multi-Agent Collaboration Framework), an end-to-end distributed system that decouples per-agent perception budgets from global video complexity.

A. Architecture

Distributed Local Agents: The input video is partitioned into disjoint temporal segments. Each segment is processed by a local agent ( $A_m$ ) operating under a strict perception budget ( $B_{per}$ ), defined by the maximum number of pixels (frames $\times$ resolution) it can ingest.
Central Coordinator: A coordinator agent ( $A_0$ ) aggregates information from all local agents to produce the final answer.
Agent-Native Latent Communication: Instead of exchanging text captions, local agents encode their observations into compact, continuous latent tokens within a shared embedding space.
- Each agent $A_m$ maps its visual input $X^{(m)}$ and query $q$ into a fixed-length sequence of communication tokens $c^{(m)} \in \mathbb{R}^{K \times d}$ .
- The coordinator consumes these tokens directly, bypassing the need for text generation and interpretation.

B. Optimization & Curriculum Training

Direct end-to-end training is unstable due to the "min-cut" bottleneck in information flow. The authors introduce a three-stage curriculum training strategy:

Semantic Alignment: Agents learn to map visual inputs to a unified latent space using caption supervision. This ensures the coordinator can decode the tokens into meaningful language, establishing a stable communication channel.
Evidence Summarization: Agents are fine-tuned on Image-QA datasets to learn how to compress task-relevant evidence into compact tokens conditioned on the query.
Cross-Agent Collaboration: The full system is trained on Video-QA datasets. The coordinator learns to fuse distributed tokens from multiple agents to perform global reasoning.

3. Key Contributions

Formalization of Budget-Constrained Multi-Agent Video Understanding: The paper defines a problem setting where both perception (pixels) and communication (tokens) are strictly bounded, proposing a framework that scales video length by increasing the number of agents rather than the input size per agent.
Agent-Native Latent Communication Mechanism: A novel protocol where agents exchange dense, continuous vectors instead of text. This preserves fine-grained visual semantics (e.g., specific colors, textures, spatial relations) that are often lost in textual summarization.
Curriculum Training Strategy: A progressive training approach that stabilizes the optimization of the shared latent space, ensuring semantic alignment and effective evidence aggregation across agents.

4. Experimental Results

The authors evaluated MACF on four long-video benchmarks: Video-MME, LongVideoBench, LVBench, and MLVU-Test.

Performance: MACF consistently outperforms state-of-the-art MLLMs (including Qwen3-VL-8B, LLaVA-OneVision, and GPT-4o) under identical perception budget constraints.
- Example: Using Qwen3-VL-8B as the backbone, MACF achieved gains of +4.5% to +7.7% over the base model across benchmarks.
- On MLVU-Test, MACF surpassed all compared open-source models, even those without input budget constraints.
Communication Efficiency:
- Compared to text-based communication (MapReduce), MACF improved performance by 9.7% to 20.3%.
- Compared to KV-cache sharing (LatentMAS), MACF improved performance by 10.3% to 14.1%, while achieving significantly lower latency and throughput costs.
Qualitative Analysis: In a case study involving a dog's rope color, text-based agents failed because one agent misidentified the dog due to text compression errors. MACF's latent tokens preserved the visual evidence, allowing the coordinator to correctly identify the rope color.
Scalability: Performance scales linearly with the number of agents (temporal capacity) and spatial resolution until hitting the communication bottleneck, demonstrating the framework's ability to handle longer videos by adding more agents.

5. Significance and Impact

Paradigm Shift: MACF moves away from text-centric multi-agent systems toward latent-native collaboration, proving that direct visual-semantic communication is superior for video understanding tasks.
Scalability: It offers a practical solution for long-video analysis on fixed hardware constraints (e.g., limited VRAM or context windows) by distributing the load rather than compressing the input.
Generalization: The framework is model-agnostic, working effectively with different backbone architectures (Qwen, LLaVA) via adapter modules, making it adaptable to future MLLM developments.
Efficiency: By avoiding autoregressive text generation for inter-agent communication, MACF reduces inference latency and computational overhead significantly compared to retrieval-based or text-based multi-agent systems.

In conclusion, MACF demonstrates that decoupling perception from complexity via compact latent collaboration is a highly effective strategy for scaling video understanding, preserving visual fidelity while adhering to strict computational budgets.