🤖 AI

Symphony: A Cognitively-Inspired Multi-Agent System for Long-Video Understanding

Symphony is a cognitively-inspired multi-agent system that enhances long-video understanding by decomposing tasks into fine-grained subtasks, employing reflection-based deep reasoning collaboration, and utilizing VLM-based grounding to achieve state-of-the-art performance on major benchmarks.

Original authors: Haiyang Yan, Hongyun Zhou, Peng Xu, Xiaoxue Feng, Mengyi Liu

Published 2026-03-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haiyang Yan, Hongyun Zhou, Peng Xu, Xiaoxue Feng, Mengyi Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are handed a 10-hour documentary about a complex mystery and asked a single, tricky question: "Why did the main character fail to enter the city, and did the mother and child bribe the guard?"

If you tried to read the whole thing, remember every detail, and solve the puzzle all by yourself, your brain would likely get overwhelmed. You might forget the beginning by the time you reach the end, or miss a tiny clue hidden in the middle.

This is exactly the problem computers face with Long-Video Understanding (LVU). Current AI models are smart, but when videos get too long and questions get too complex, they get "lost in the middle."

Enter Symphony. Think of Symphony not as a single super-brain, but as a highly organized detective agency working together to solve the mystery.

The Symphony Team: A Detective Agency

Instead of one AI trying to do everything, the authors created a team of specialized agents, each with a specific job, mimicking how humans think:

The Project Manager (Planning Agent):
- Role: This is the boss. When a question comes in, the Project Manager doesn't try to solve it alone. Instead, they break the big, scary question into small, manageable steps.
- Analogy: Imagine a chef who doesn't cook the whole meal at once. They say, "First, we need to chop the onions. Then, boil the water. Then, sear the steak." The Project Manager directs the team on what to do next based on what they've already learned.
The Spotlight Operator (Grounding Agent):
- Role: Long videos are like a dark room. The Spotlight Operator finds the specific 10-second clips where the answer might be hiding.
- The Trick: Old methods just searched for keywords (like "guard" or "bribe"). But what if the video shows a handshake or money without saying the word "bribe"? The Spotlight Operator uses a smart AI to understand the context. It asks, "Does this scene look like a bribe?" rather than just looking for the word "bribe." It shines a light on the right moments so the team doesn't waste time watching the whole movie.
The Eyes (Visual Perception Agent):
- Role: Once the Spotlight Operator points to a specific scene, this agent zooms in. It looks closely at the frames.
- Analogy: This is like a forensic expert examining a crime scene photo. They check: "Is that a gun? Is that a passport? Are they smiling or frowning?" They can compare two different scenes side-by-side to spot differences.
The Scribe (Subtitle Agent):
- Role: This agent reads the subtitles and listens to the dialogue.
- Analogy: Sometimes the answer is in what people say, not what they do. If a character says, "This pass is invalid," the Scribe catches that crucial detail immediately, so the team doesn't have to guess.
The Quality Control Inspector (Reflection Agent):
- Role: This is the most important new feature. Before the team gives a final answer, the Inspector reviews the whole process.
- The "Wait a Minute" Moment: If the team says, "They bribed the guard," the Inspector checks the evidence. "Hold on," the Inspector says. "I see the guard taking money, but the mother and child didn't. Did we mix up the characters? Let's go back and check."
- Why it matters: This prevents the AI from being overconfident and making silly mistakes. It forces the system to double-check its work, just like a human would.

How They Work Together (The "Symphony")

In the past, AI tried to solve these problems in a straight line: Watch video $\rightarrow$ Think $\rightarrow$ Answer. If it got stuck, it was stuck.

Symphony works like a jazz band or a roundtable discussion:

The Project Manager breaks the question down.
The Spotlight finds the relevant clips.
The Eyes and Scribe gather details.
The Inspector checks the logic.
If the Inspector finds a flaw, the team goes back, re-examines the evidence, and tries again.

This back-and-forth "reflection" is what makes them so good at hard questions. They don't just guess; they argue, verify, and refine until they are sure.

The Results: Why It Matters

The paper tested Symphony on four difficult video benchmarks (like LVBench and VideoMME).

The Result: Symphony beat the previous best AI by a significant margin (5% better on the hardest test).
The Takeaway: By splitting the work among specialized "agents" and adding a "quality control" step, the system can handle videos that are hours long and questions that require deep thinking.

In a Nutshell

Symphony is like upgrading from a lone detective trying to solve a 100-page case file in one sitting, to a team of experts (a researcher, a forensic analyst, a translator, and a supervisor) who divide the work, check each other's facts, and only deliver the final verdict when they are 100% sure they have the truth.

This approach allows AI to finally understand long, complex stories without getting lost in the details.

1. Problem Statement

Long-form Video Understanding (LVU) presents significant challenges due to high information density and extended temporal spans. While Multimodal Large Language Models (MLLMs) and single-agent systems have advanced, they face critical limitations:

Reasoning Bottlenecks: Single agents struggle with complex, multi-step reasoning tasks, often degrading in performance as input length and task complexity increase.
Retrieval Limitations: Existing Retrieval-Augmented Generation (RAG) approaches often rely on simple embedding-based retrieval (e.g., CLIP). This fails to capture abstract concepts, implicit intentions, or multi-hop reasoning requirements, leading to the loss of key information.
Context Overload: Directly processing long videos or reducing context via naive retrieval often results in "lost in the middle" phenomena or the omission of crucial temporal cues.
Collaboration Gaps: Existing multi-agent systems often use linear pipelines or modality-specific agents that lack deep inter-modal interaction and dynamic task adjustment.

2. Methodology: Symphony

Symphony is a centralized Multi-Agent System (MAS) designed to emulate human cognitive patterns (perception, attention, reasoning, language, and decision-making) to solve LVU tasks. It decomposes complex problems into fine-grained subtasks managed by specialized agents.

A. System Architecture

The system consists of five core agents orchestrated by a central Planning Agent:

Planning Agent: Acts as the central coordinator. It decomposes the user query into a sequence of subtasks, delegates them to specialized agents, and synthesizes evidence to generate the final answer. It maintains the reasoning trajectory.
Grounding Agent: Simulates "attention." It analyzes the query to identify relevant video segments. Crucially, it employs a hybrid strategy:
- CLIP-based Retrieval: For simple, entity-specific queries.
- VLM-based Scoring: For complex queries involving abstract concepts or multi-hop reasoning. It uses an LLM to expand/decompose the query and a Vision-Language Model (VLM) to score video segments based on semantic relevance, rather than simple keyword matching.
Subtitle Agent: Handles the "language" component. It processes video subtitles to perform entity recognition, sentiment analysis, and topic modeling, extracting textual clues without overloading the planning agent's context window.
Visual Perception Agent: Handles "perception." It utilizes tools like a Frame Inspector (for fine-grained analysis), Global Summary (for high-level context), and Multi-Segment Analysis (for cross-segment comparison) to analyze visual content.
Reflection Agent: Acts as a "critic" (inspired by the Actor-Critic framework). It evaluates the reasoning chain and the proposed answer. If the evidence is insufficient or the logic is flawed, it generates a critique to trigger a new round of reasoning, effectively expanding the exploration space.

B. Collaboration Mechanism

Symphony utilizes a Reflection-Enhanced Dynamic Collaboration mechanism:

Forward Reasoning: The Planning Agent iteratively selects actions (invoking specific agents) based on the current state (question + historical trajectory).
Verification & Iteration: After an initial solution is proposed, the Reflection Agent validates the logic. If the critique ( $C$ ) identifies gaps, the system state is updated, and the Planning Agent initiates a new round of subtask execution. This allows for iterative evidence accumulation and self-correction.

3. Key Contributions

Cognitively-Inspired Task Decomposition: Unlike modality-based partitioning, Symphony decouples tasks by functional capability (planning, grounding, perception, language, reflection). This reduces cognitive load on individual models and minimizes inter-agent coupling costs.
Advanced Grounding Strategy: The proposed Grounding Agent moves beyond simple text-image matching. By combining LLM-based query expansion with VLM-based relevance scoring, it achieves robust grounding for abstract concepts and implicit intentions in long videos.
Reflection-Enhanced Reasoning: The integration of a dedicated Reflection Agent allows the system to verify its own reasoning chains, significantly improving accuracy on complex, multi-hop reasoning tasks compared to single-pass or linear multi-agent approaches.
State-of-the-Art Performance: Symphony achieves superior results across multiple benchmarks, demonstrating that dynamic collaboration and reflection are more effective than fixed pipelines.

4. Experimental Results

The authors evaluated Symphony on four major LVU benchmarks: LVBench, LongVideoBench, Video-MME, and MLVU.

LVBench: Symphony achieved 71.8%, surpassing the previous state-of-the-art (DVD) by 5.0%. It showed particular strength in Reasoning (Rea) and Event Understanding (EU).
LongVideoBench: Achieved 77.1%, outperforming VideoDeepResearch by 6.5%.
Video-MME: Achieved 78.1%.
MLVU: Achieved 81.0%.
Ablation Studies:
- Removing the Reflection Agent caused a 2.5% drop, highlighting its value in preventing overconfidence.
- Removing the specialized Subtitle or Visual Perception agents led to performance drops due to context overload or reduced perception precision.
- The VLM-based scoring tool in the Grounding Agent proved essential for handling complex queries, outperforming pure CLIP retrieval.
Cost Efficiency: Despite using multiple agents, Symphony is more cost-effective than some SOTA methods (e.g., using OpenAI o3) by leveraging efficient open-source models (DeepSeek R1/V3, Seed 1.6 VL) and reducing token consumption through targeted retrieval.

5. Significance

Symphony represents a paradigm shift in long-video understanding from single-pass processing to iterative, cognitive-inspired collaboration.

Scalability: By decoupling reasoning from perception and grounding, the system scales better with video length and query complexity.
Robustness: The reflection mechanism ensures that errors in early reasoning steps are caught and corrected, a critical feature for long-context tasks where "hallucinations" are common.
Generalizability: The framework is model-agnostic regarding the underlying VLM/LLM, as demonstrated by its consistent performance across different base models.

In conclusion, Symphony demonstrates that mimicking human cognitive processes through specialized, collaborating agents with a built-in reflection loop is the most effective current approach for solving the complex, multi-step reasoning challenges inherent in long-form video understanding.