OmniGAIA: Towards Native Omni-Modal AI Agents

This paper introduces OmniGAIA, a comprehensive benchmark for evaluating omni-modal agents on complex reasoning and tool-use tasks across video, audio, and image modalities, alongside OmniAtlas, a native omni-modal foundation agent trained with advanced strategies to bridge the gap between current bi-modal models and next-generation real-world AI assistants.

The Gist

Imagine you are trying to build a super-smart personal assistant, like a digital butler who can see, hear, and talk just like a human.

Right now, most AI assistants are a bit like bilingual people who only speak two languages at a time. They are great at looking at a picture and describing it (Vision + Language), or listening to a song and telling you the lyrics (Audio + Language). But they struggle when you ask them to do something complex that requires all three senses at once: watching a video, listening to the background noise, reading the subtitles, and then going out to the internet to find more information to solve a puzzle.

This paper introduces two big things to fix that problem: a giant test called OmniGAIA and a new super-agent called OmniAtlas.

Here is the breakdown in simple terms:

1. The Problem: The "Two-Channel" Limitation

Think of current AI like a detective who only has a camera and a microphone, but they can't use them together effectively. If you show them a video of a bridge and ask, "How old was this bridge when a famous movie was filmed here?", the AI might get confused. It might look at the bridge but ignore the audio clues, or it might guess based on what it thinks it knows instead of checking the facts.

2. The Solution Part A: The "OmniGAIA" Exam

The researchers created a new, very difficult test called OmniGAIA.

• The Analogy: Imagine a final exam for a detective agency. Instead of asking simple questions like "What color is the car?", the exam gives the detective a 10-minute video clip with background noise, a specific question about a historical event, and a rule: "You must use the internet to verify your answer before you write it down."
• What it tests:
  • Multi-Sense: Can the AI watch the video and listen to the audio at the same time?
  • Tool Use: Can the AI realize it doesn't know the answer, so it opens a web browser to search for facts?
  • Logic: Can it connect the dots? (e.g., "The video says the bridge is in Joliet. I need to search for 'Joliet bridges' to find the name, then search for the construction date, then do some math.")

The Result: The exam is brutally hard. Even the smartest AI (Gemini-3-Pro) only got about 62% right. A popular open-source AI (Qwen-3-Omni) only got 13% right. This shows we have a long way to go!

3. The Solution Part B: The "OmniAtlas" Agent

Since the test was so hard, the researchers built a new AI agent called OmniAtlas to try and pass it.

• The Analogy: Think of OmniAtlas as a detective who has learned a new trick: "Active Perception."
  • Old AI: Like a person staring at a blurry photo of a whole city, trying to guess what's in the distance.
  • OmniAtlas: Like a detective who says, "That part of the video is too blurry. Let me zoom in on that specific 5-second clip," or "The audio is quiet here; let me listen to just that one sentence again." It doesn't just swallow the whole video; it chooses what to look at and listen to.
• The Training: They taught OmniAtlas by showing it thousands of examples of how to solve these puzzles. They used a method called "Hindsight-Guided Tree Exploration."
  • Metaphor: Imagine a detective trying to solve a crime. They try a path, realize it's a dead end, and then go back and try a different path. OmniAtlas was trained to look at all the "dead ends" (mistakes), figure out exactly where it went wrong (did it mishear the audio? Did it search the wrong website?), and learn to fix that specific step.

4. The Results: A Big Leap Forward

When they put OmniAtlas to the test:

• It improved the open-source AI's score from 13% to 20%. That's a huge jump in the world of AI!
• The Key Lesson: The researchers found that just making the AI "bigger" (adding more brain power) didn't help much. The real secret sauce was teaching the AI how to use tools (like search engines) and how to check its own work.

Summary

• The World: We need AI that can see, hear, and think together to help us in the real world.
• The Test (OmniGAIA): A tough new exam that forces AI to use all its senses and go to the internet to find answers.
• The Hero (OmniAtlas): A new AI that learns to "zoom in" on details and use tools smartly, rather than just guessing.

The Bottom Line: We are moving from AI that just "looks and listens" to AI that can investigate, verify, and act like a true human assistant. This paper is a major step toward that future.

Technical Summary

1. Problem Statement

Current Multimodal Large Language Models (MLLMs) are primarily limited to bi-modal interactions (e.g., vision-language or audio-language) and focus heavily on perception tasks. They lack the unified cognitive capabilities required for general AI assistants that must:

• Seamlessly integrate omni-modal perception (vision, audio, and language).
• Perform long-horizon reasoning across complex, time-aligned media.
• Execute multi-turn tool use (e.g., web search, code execution) to verify facts and acquire external knowledge.
• Provide verifiable, open-form answers rather than simple multiple-choice selections.

Existing benchmarks (e.g., OmniBench, WorldSense) are largely perception-centric, short-duration, and bi-modal, failing to stress-test the agentic reasoning and tool-integration capabilities necessary for real-world scenarios.

2. Methodology

The paper introduces two core components: the OmniGAIA benchmark and the OmniAtlas agent.

A. OmniGAIA Benchmark Construction

OmniGAIA is a challenging benchmark comprising 360 tasks across 9 real-world domains (e.g., Geography, History, Science). It features two input settings: Video-with-Audio and Image+Audio.

• Construction Pipeline (Event-Graph Driven):
  1. Data Collection & Signal Mining: Uses strong models (Gemini-3-Flash) to extract fine-grained signals (OCR, ASR, speaker ID, events) from raw media.
  2. Event Graph Construction: Builds an initial graph connecting cross-modal entities and relations.
  3. Agentic Expansion: An exploration agent (DeepSeek-V3.2) uses external tools (web search, cross-modal retrieval) to expand the graph with missing evidence.
  4. Event Fuzzification: Key nodes/edges are "fuzzified" (masked or abstracted) to force multi-hop reasoning, preventing trivial fact retrieval.
  5. Quality Control: A pipeline of LLM screening and human verification ensures solvability, uniqueness, and difficulty.
• Task Characteristics: Tasks require multi-step planning, cross-modal association, and external tool usage to derive open-form answers.

B. OmniAtlas: Native Omni-Modal Agent

OmniAtlas is a foundation agent designed to unify perception and reasoning with active tool use.

• Active Omni-Modal Perception: Unlike models that downsample entire media, OmniAtlas supports selective perception. It can dynamically request specific video segments, audio clips, or image crops (read_video, read_audio, read_image) to inspect details "where needed," reducing token costs and hallucination.
• Tool-Integrated Reasoning (TIR): The agent interleaves internal reasoning with tool calls (web search, browser, code executor) in a single trajectory.
• Training Strategy:
  1. Trajectory Synthesis: Uses Hindsight-Guided Tree Exploration to generate high-quality tool-integrated trajectories. A verifier prunes incorrect branches, keeping only successful paths.
  2. Supervised Fine-Tuning (SFT): Applies masked supervision to train the model on agent thoughts and actions while ignoring noisy tool observations.
  3. OmniDPO (Fine-Grained Error Correction): A Direct Preference Optimization method that identifies the first error in a failed trajectory, generates a corrected prefix, and trains the model to prefer the corrected path over the erroneous one. This targets specific failure modes (perception, reasoning, tool use).

3. Key Contributions

1. OmniGAIA Benchmark: The first benchmark specifically designed for native omni-modal agents, featuring video/audio/image inputs, multi-domain coverage, multi-hop reasoning, and multi-turn tool use with open-form answers.
2. Event-Graph Construction Pipeline: A scalable method to synthesize hard yet solvable tasks from real-world data by structuring evidence and applying controllable fuzzification.
3. OmniAtlas Agent & Training Recipe: A practical framework for improving open-source models via active perception and a three-stage training process (Synthesis, SFT, OmniDPO) that significantly boosts agentic capabilities.
4. Comprehensive Analysis: Detailed evaluations revealing that current models struggle with tool planning and long-horizon reasoning, and that parameter scaling alone is insufficient.

4. Experimental Results

Benchmark Performance

• High Difficulty: The benchmark is extremely challenging. The strongest proprietary model, Gemini-3-Pro, achieves 62.5 Pass@1, while the strongest open-source baseline, Qwen3-Omni, only reaches 13.3.
• Proprietary vs. Open Gap: There is a massive performance gap (~4.7x) between proprietary and open models, highlighting the need for better open-source agentic reasoning.
• Scaling Limits: Increasing model size (e.g., LongCat-Flash-Omni at 560B) does not guarantee better performance if tool-use policies are weak.

OmniAtlas Improvements

• Significant Gains: Applying the OmniAtlas recipe to Qwen3-Omni improved performance from 13.3 to 20.8 (+7.5 absolute).
• Small Model Boost: The gains were even more pronounced on smaller backbones (e.g., Qwen2.5-Omni-7B improved from 3.6 to 13.3, a ~3.7x increase).
• Error Reduction: OmniAtlas effectively reduced ineffective tool-use (from 81.1% to 59.4%) and reasoning errors (from 79.7% to 64.4%) in the 30B model.

Key Findings from Analysis

• Tool Use is Indispensable: Models that do not use tools (or use them minimally) fail almost all tasks.
• Perception Bottleneck: While OmniAtlas improved tool policies, visual and audio perception errors remain high (~30-50%), indicating that fundamental perception capabilities are a persistent bottleneck.
• Native vs. Tool-Based Perception: Experiments showed that for capable agents, native omni-modal perception is superior to delegating perception to external tools. Native perception yields higher accuracy with fewer tool calls.
• Failure Modes: The primary failures are ineffective tool usage (wrong queries, confirmation bias) and reasoning collapse due to incorrect evidence acquisition.

5. Significance

This work marks a critical step toward next-generation native omni-modal AI assistants.

• Benchmarking Standard: OmniGAIA provides a rigorous standard for evaluating the complex interplay of perception, reasoning, and tool use, moving beyond simple perception tasks.
• Training Paradigm: The proposed training recipe (Tree Exploration + OmniDPO) offers a viable path to unlock agentic potential in open-source models without requiring massive proprietary datasets.
• Future Directions: The paper identifies Omni-modal Agentic RL, MCP Services, and Embodied Agents as key future research directions to bridge the gap between current models and general-purpose AI assistants capable of operating in the real world.

In summary, OmniGAIA demonstrates that while current models struggle with complex, multi-modal agentic tasks, targeted training strategies focusing on active perception and fine-grained error correction can significantly enhance open-source models' ability to reason and act in real-world scenarios.