🤖 AI

Orchestra-o1: Omnimodal Agent Orchestration

This paper introduces Orchestra-o1, an omnimodal agent orchestration framework that enables efficient multi-agent collaboration across diverse inputs like text, images, audio, and video through a unified orchestration mechanism and decision-aligned group relative policy optimization (DA-GRPO), achieving state-of-the-art performance on the OmniGAIA benchmark.

Original authors: Fan Zhang, Vireo Zhang, Shengju Qian, Haoxuan Li, Hao Wu, Jinyang Wu, Donghao Zhou, Zhihong Zhu, Zheng Lian, Xin Wang, Pheng-Ann Heng

Published 2026-06-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Fan Zhang, Vireo Zhang, Shengju Qian, Haoxuan Li, Hao Wu, Jinyang Wu, Donghao Zhou, Zhihong Zhu, Zheng Lian, Xin Wang, Pheng-Ann Heng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a massive, complicated mystery. You have a clue written on a piece of paper, a photo of a suspect, a recording of a voice, and a video of a crime scene.

The Old Way (Native Agents):
In the past, researchers tried to solve this by hiring one "Super Detective" (a single AI model). They expected this one person to read the paper, analyze the photo, listen to the audio, watch the video, and then write the solution all by themselves. The problem? Even the smartest "Super Detective" gets overwhelmed. They might miss a detail in the audio because they were too busy looking at the photo, or they might get confused trying to do everything at once. It's like asking a single chef to cook a 10-course meal, wash the dishes, and manage the restaurant's finances simultaneously.

The New Way (Orchestra-o1):
The paper introduces Orchestra-o1, which changes the game. Instead of one Super Detective, it creates a conductor who leads a team of specialized experts.

Here is how it works, using simple analogies:

1. The Conductor (The Main Agent)

The "Main Agent" in Orchestra-o1 isn't trying to do the heavy lifting itself. Think of it as the Conductor of an Orchestra.

It doesn't play the violin or the drums.
Its job is to listen to the problem, look at all the clues (text, images, audio, video), and decide: "Who needs to handle this part?"
It breaks the big mystery into smaller, manageable tasks.

2. The Specialists (The Sub-Agents)

Once the Conductor figures out what needs to be done, it calls in the right experts:

The Audio Expert: "Hey, listen to this recording and tell me what time the event happened."
The Visual Expert: "Look at this photo and tell me what city this building is in."
The Web Researcher: "Go check the internet for the time zone of that city."
The Calculator: "Do the math to convert the local time to universal time."

3. The Superpower: Parallel Processing

This is where Orchestra-o1 gets really fast.

Old Way (Linear): The Conductor would ask the Audio Expert, wait for the answer, then ask the Visual Expert, wait for that answer, and so on. It's like a relay race where everyone waits for the previous runner to finish.
Orchestra-o1 Way (Parallel): The Conductor shouts out all the tasks at once! The Audio Expert, Visual Expert, and Web Researcher all work simultaneously.
The Result: The team finishes the job much faster because they aren't waiting on each other. The paper proves this mathematically: if you have 5 independent tasks, doing them one by one takes 5 hours, but doing them together takes roughly 1 hour (plus a tiny bit of time to gather the results).

4. The Memory (Context)

As the experts finish their tasks, they hand their findings back to the Conductor. The Conductor doesn't just throw them in a pile; it organizes them into a "Memory Notebook."

If the Audio Expert says "It's 7:49 AM" and the Visual Expert says "It's Prague," the Conductor writes down: "7:49 AM in Prague."
It keeps this notebook organized so it doesn't get confused or forget details while solving the next part of the puzzle.

5. Training the Conductor (DA-GRPO)

The paper also explains how they taught an open-source AI (a smaller, free model) to be this Conductor.

They didn't just tell the AI "Get the right answer."
Instead, they used a special training method called DA-GRPO. Think of this as a strict coach who watches every move the Conductor makes.
The coach gives points not just for the final answer, but for how the Conductor decided to split the work. Did they pick the right expert? Did they ask the right questions? Did they use the right tools?
By rewarding these good decisions, the small AI learned to become a brilliant Conductor, even though it wasn't originally designed to be one.

The Results

The paper tested this system on a difficult benchmark called OmniGAIA (which mixes text, images, audio, and video).

The Champion: When Orchestra-o1 used a top-tier "Conductor" (GPT-5), it beat the best single "Super Detective" (Gemini-3-Pro) by a significant margin (10.3% higher accuracy).
The Underdog: Even when they used their own trained, smaller AI (Orchestra-o1-8B) as the Conductor, it still beat all other open-source "Super Detectives" by a huge amount, going from a 20.8% success rate to 30.0%.

In Summary:
Orchestra-o1 proves that for complex, multi-sensory problems, it's better to have a smart manager who coordinates a team of specialists working in parallel, rather than one overworked genius trying to do everything alone. It makes AI faster, cheaper, and much more accurate at solving real-world puzzles.

Technical Summary: Orchestra-o1: Omnimodal Agent Orchestration

1. Problem Definition

Current Large Language Model (LLM)-based agent systems have largely shifted from single-agent workflows to multi-agent swarms to handle complex tasks. However, existing orchestration frameworks face significant limitations in omnimodal scenarios, where tasks require the unified understanding and coordination of heterogeneous inputs (text, image, audio, and video).

Two primary categories of existing omnimodal agents are identified as insufficient:

Native Omnimodal Agents: These employ a single omnimodal large language model (OLLM) to handle perception, reasoning, planning, and tool use simultaneously. They struggle with long-horizon reasoning, external information seeking, and fine-grained cross-modal understanding, often resulting in suboptimal performance on complex benchmarks (e.g., 62.5% accuracy for Gemini-3-Pro on OmniGAIA).
Orchestration-Based Agents: While these decouple high-level reasoning from low-level modality processing, existing open-source frameworks are often limited to text or vision-language tasks, rely on rigid linear workflows, and lack the flexibility to dynamically adapt agent roles or execute sub-tasks in parallel.

The core challenge is designing a framework that supports modality-aware task decomposition, online sub-agent specialization, and parallel execution to effectively tackle real-world tasks involving diverse information sources.

2. Methodology: The Orchestra-o1 Framework

Orchestra-o1 is proposed as an omnimodal agent orchestration framework that separates high-level orchestration from specialized perception and action execution.

Core Architecture

The system consists of a Main Agent (Orchestrator), a pool of Sub-agent Backends, and a Unified Tool Ecosystem.

Unified Tool Ecosystem: Includes perception tools (image, audio, video analysis) and action tools (web search, page visit, code execution).
Flexible Backends: Supports heterogeneous models (open-source and proprietary) for both the main agent and sub-agents. Each backend is characterized by a skill vector and a cost-latency profile.

Key Mechanisms

Modality-Aware Task Decomposition:
- The main agent induces a latent dependency graph over unsolved sub-goals.
- It predicts modality masks (which inputs are needed) and tool masks for each sub-goal.
- It selects a parallel batch of ready sub-tasks (those whose dependencies are met) to execute simultaneously, rather than following a linear sequence.
Backend and Tool Assignment:
- Model Selection: The main agent matches sub-task requirements against backend capabilities, optimizing for a cost-aware matching score (balancing capability match, unit cost, and latency).
- Tool Selection: Tools are assigned based on a sparse coverage objective to ensure necessary evidence is gathered without redundancy.
Parallel Sub-task Execution:
- Delegated sub-tasks are executed asynchronously by independent ReAct-style sub-agents.
- The framework theoretically demonstrates a latency advantage over linear orchestration, where the round-level latency is determined by the maximum execution time of parallel tasks plus synchronization overhead, rather than the sum of all execution times.
Context Memory and Iterative Refinement:
- A structured memory stores evidence from sub-agents.
- The main agent compresses this context to stay within token budgets, prioritizing information relevance and successful evidence.
- The process iterates until an evidence sufficiency score triggers the generation of the final answer.

Theoretical Justification

The paper provides an information-theoretic proposition stating that orchestration yields higher information gain ( $I(Y; E_{orch} | q) > I(Y; E_0 | q)$ ) compared to native agents. By assigning modality-aware sub-tasks to specialized backends, the system retains more relevant information for the final answer, leading to strictly smaller expected risk under log loss.

3. Training Recipe: DA-GRPO

To enable open-source models to serve as effective main agents, the authors propose Decision-Aligned Group Relative Policy Optimization (DA-GRPO).

Data Curation: A seed-based pipeline generates diverse training data from public datasets (e.g., FineVideo, LongVideoBench). It uses "anchor facts" to ensure rewritten questions preserve essential perceptual evidence while varying reasoning paths (e.g., pivot swapping, temporal shifting).
Reward Design: Unlike standard GRPO which focuses on final answer correctness, DA-GRPO evaluates step-level orchestration decisions using a multi-dimensional rubric:
1. Format Reward: Validity of the JSON output.
2. Action Reward: Validity of the action and parameters.
3. Tool Reward: Reasonableness of tool and sub-task assignments.
4. Decision Reward: Overall strategic quality of the orchestration.
Optimization: The method computes relative advantages within a group of sampled decisions and optimizes the policy to align with high-quality reference trajectories, providing dense feedback on delegation, tool selection, and stopping criteria.

4. Experimental Results

The framework was evaluated on the OmniGAIA benchmark, covering text, image, audio, and video inputs across various domains (Geo, Tech, Hist, etc.).

Performance with Proprietary Models

Orchestra-o1-GPT-5 achieved a state-of-the-art 72.8% overall accuracy.
This represents a 10.3% absolute improvement over the strongest native proprietary model (Gemini-3-Pro, 62.5%) and a 32.8% improvement over the orchestration baseline AOrchestra-GPT-5 (40.0%).
Gains were consistent across all categories and difficulty levels, with significant improvements on "Hard" tasks requiring multi-step reasoning.

Performance with Open-Source Models

Orchestra-o1-8B (trained on Qwen3-8B using DA-GRPO) achieved 30.0% accuracy.
This substantially outperforms the strongest open-source baseline, OmniAtlas-Qwen3-30B-A3B (20.8%), despite using a much smaller 8B backbone.
Ablation studies confirmed that the orchestration scaffold itself provides a strong inductive bias (improving accuracy from 12.5% to 26.3%), and DA-GRPO further enhances strategic decision-making compared to SFT or vanilla GRPO.

Efficiency

Orchestra-o1 demonstrated superior cost-effectiveness. It achieved 72.8% accuracy at a cost of 341.6, whereas AOrchestra required a cost of 565.7 to achieve only 40.0% accuracy.
This efficiency stems from parallel execution of independent sub-tasks and precise tool/backend selection, avoiding unnecessary expensive calls.

5. Significance and Contributions

The paper claims three primary contributions:

Omnimodal Agent Orchestration Framework: Orchestra-o1 provides a scalable, open-source framework that decouples high-level planning from specialized execution. By enabling modality-aware decomposition and parallel scheduling, it addresses the gap between current agent research and the heterogeneous nature of real-world tasks.
Efficient Training Recipe (DA-GRPO): The proposed algorithm enables open-source models to acquire strong orchestration capabilities. By aligning step-level decisions with high-quality trajectories via a multi-dimensional reward, it allows compact models (like 8B) to outperform larger native omnimodal models.
Multifaceted Validation: Extensive experiments validate that explicit orchestration is a viable and superior path for omnimodal intelligence. The results suggest that advancing agent intelligence does not solely rely on scaling native OLLMs but also on learning to coordinate specialized agents, tools, and evidence sources in a principled manner.

The authors conclude that while future work will explore practical scenarios like audio-video collaborative coding, the current work establishes a new paradigm for handling complex, heterogeneous agent tasks through structured orchestration.