🤖 AI

Omni-Decision: A Progressive Evidence-State Agent System for Omni-Modal QA

The paper introduces Omni-Decision, a training-free agent system that improves omni-modal QA accuracy by maintaining a structured, explicit evidence state to systematically track, validate, and close information gaps across diverse media sources, achieving significant performance gains over existing baselines.

Original authors: Ming Ma, Yi Zhu, Yiran Zhong, Feida Zhu, Weigao Sun, Junhan Shi, Lingrui Mei, Tianming Yang, Steven Hoi

Published 2026-07-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ming Ma, Yi Zhu, Yiran Zhong, Feida Zhu, Weigao Sun, Junhan Shi, Lingrui Mei, Tianming Yang, Steven Hoi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but the clues are scattered everywhere: a tiny detail in a video clip, a whisper in an audio track, a fact hidden on a website, and a number you have to calculate on a calculator. Most AI detectives today work like someone frantically scribbling notes on a messy napkin. They look at a clue, write it down, look at another, write it down, and hope that by the time they reach the end, they remember which note belongs to which part of the mystery. Often, they get lost in their own scribbles, forget what they've already found, or try to solve the puzzle before they have all the pieces.

The paper introduces a new detective system called Omni-Decision. Instead of a messy napkin, this system uses a structured, living checklist called an "evidence state." Think of this checklist as a magical whiteboard that the detective can't ignore. It doesn't just store notes; it actively tracks exactly what is confirmed, what is missing, what is conflicting, and what needs to be calculated next.

Here is how the magic works:

The Checklist: When a question is asked, the system immediately breaks it down into specific "needs" on the whiteboard. For example, "Find the watch brand in the video" or "Find the date Instagram launched."
The Loop: The detective picks a tool (like a web search or a video scanner) to fill one item on the list.
The Judge: Before the detective moves on, a strict "Critic" checks the new information. Did it solve the problem? Did it contradict something found earlier?
The Update: A special "Reducer" updates the whiteboard. If a clue is found, the "missing" box turns green. If two clues fight each other, a red "conflict" flag is raised. The detective never moves forward until the whiteboard says the current step is done.
The Stop: The system knows exactly when to stop. It won't guess an answer just because it's tired; it only answers when the whiteboard shows that every single required piece of evidence is locked in and no conflicts remain. If it runs out of tools and the board is still incomplete, it admits defeat rather than making up a fake answer.

What the paper argues against:
The authors explicitly argue that simply making AI models bigger or giving them longer "memory" (like a longer napkin) isn't the solution. They show that having a massive history of past actions doesn't help if the AI can't track what is actually grounded and what is still missing. They also rule out the idea that the system needs to be "trained" on new data to work this way; it works by organizing how existing tools are used, not by learning new tricks.

The Results:
The paper measured this system on two tough tests. On a test called OmniGAIA, which involves open-world searching across videos, audio, and the web, Omni-Decision scored 45.6% accuracy. This is a huge jump compared to the standard "base" agent, which only scored 18.33%. That's a gain of 27.3 percentage points.

On a second test called WorldSense, which focuses on integrating audio and video without external web searches, the system scored 58.3%, beating the baseline by 30.2 percentage points.

The authors suggest that this approach is particularly good at spotting when a detective is stuck. In their analysis of failed attempts, they found that many "wrong" answers actually had the detective making real progress—they just got blocked because a specific piece of evidence (like reading tiny text on a sign) was too hard for the tools to find. The system's strength isn't that it never fails, but that it clearly shows where it failed and why, rather than just guessing blindly.

In short, Omni-Decision proves that for complex, multi-step mysteries, having a clear, shared map of what is known and what is missing is far more powerful than just having a longer memory or a bigger brain. It turns the chaotic process of searching for clues into a disciplined, step-by-step mission.

Technical Summary: Omni-Decision

Problem Statement

Omni-modal evidence-seeking Question Answering (QA) requires agents to answer questions where the necessary evidence is sparsely distributed across heterogeneous sources, including videos, audio, images, web pages, and computational results. The authors identify three primary challenges in this setting:

Sparse and Distributed Evidence: Relevant clues may appear in short video segments, specific subtitle spans, audio events, image regions, or external web results, making it difficult to locate all necessary components.
Partial Observability: The system cannot know in advance which entity attributes, relations, or intermediate values are required until earlier evidence is grounded.
Answer Readiness as a Control Problem: The agent must decide not only what to acquire next but also whether a candidate answer is sufficiently supported, whether conflicts require repair, and when to stop if gaps are unfillable.

The paper argues that existing agentic multimodal systems often fail because they rely on implicit dialogue histories, scratchpads, or free-form tool trajectories. These approaches make it difficult to track grounded evidence, identify missing attributes, or determine when an evidence chain is complete. Consequently, the problem is framed not merely as a context-scaling issue but as a state-maintenance problem.

Methodology: Omni-Decision

The authors propose Omni-Decision, a training-free system that reframes omni-modal QA as a query-scoped evidence-closure process. The core innovation is the maintenance of an explicit, structured, and updateable Evidence State ( $S_t$ ) that serves as the central control object for the entire inference loop.

1. Query-Scoped Evidence State

Unlike general video memory or tool logs, the Evidence State is specific to the current query and contains four mutable components:

$E_t$ (Confirmed Evidence): Atoms of confirmed evidence with sources and temporal support.
$C_t$ (Unresolved Conflicts): Records of contradictions between evidence sources.
$F_t$ (Fact/Computation Dependencies): Tracks external facts, entity-attribute completions, and computation results.
$U_t$ (Open Evidence Needs): Lists unresolved questions and uncertainty.

The system initializes $S_0$ from the query before any tool calls. As the agent interacts with tools, the state is updated deterministically.

2. Inference Loop and Control

The system operates in a loop (Algorithm 1) where all decision-making modules read from a shared state view, while only one module writes to it:

Planner: Reads a bounded digest of the immutable context ( $R_0$ ) and the current state ( $S_t$ ) to select the next action (tool usage or finishing).
Tools: Execute actions (e.g., video grounding, web search, code execution) and return observations.
Critic: Validates whether new observations support, conflict with, or leave open the current evidence needs.
Reducer: The only component that writes to the state. It normalizes observations and critic verdicts into typed events and commits them to $S_{t+1}$ using deterministic field-update rules.
Finalizer: Drafts an answer only when the state indicates readiness.

3. Readiness and Stopping Conditions

Answer readiness is derived strictly from the state, not as an independent planner decision:
$\text{ready}(S_t) = (U_t = \emptyset) \land \text{complete}(F_t) \land (C_t = \emptyset)$
The system terminates with a "sufficient" answer only if all open needs are filled, all dependencies are resolved, and no conflicts exist. If no available action can plausibly reduce an open need or resolve a conflict, the system stops with an "insufficient" status rather than forcing an unsupported answer.

Key Contributions

Formulation: The authors formulate omni-modal evidence-seeking QA as a query-scoped evidence closure problem, requiring the agent to explicitly maintain grounded entities, open attributes, cross-entity relations, and external dependencies rather than relying on implicit history.
System Design: They propose Omni-Decision, a training-free evidence-state system. A key architectural distinction is that planning, verification, repair, and stopping are all conditioned on the same state view, while the Reducer is the sole writer to the state. This makes trajectories inspectable, replayable, and ablatable.
Evaluation: The system is evaluated on OmniGAIA (open-world, tool-augmented) and WorldSense (self-contained audio-video MCQ), demonstrating that explicit state control improves performance across different task formats.

Experimental Results

OmniGAIA (Open-World Evidence Collection)

Performance: Omni-Decision achieved 45.6% accuracy on the OmniGAIA benchmark.
Improvement: This represents a +27.3 percentage point improvement over the official OmniGAIA base agent (18.33%) and a +30.2 percentage point improvement over the WorldSense baseline (58.3% vs. 28.1% for OmniAgent).
Ablation: Removing the evidence state (No-State ablation) caused a significant drop in accuracy (e.g., -9.17 points in the default configuration), confirming that the gain is separable from the choice of planner or perception backend.
Failure Analysis: Audits of 40 Medium/Hard cases revealed that many "incorrect" runs actually made partial progress along the evidence chain but were blocked by unclosed perception or retrieval slots, rather than failing to start.

WorldSense (Self-Contained Audio-Video)

Performance: Omni-Decision achieved 58.3% accuracy on WorldSense.
Transferability: Despite the absence of external tools (web search, browsing) in this benchmark, the evidence-state system remained effective, organizing internal media evidence for multiple-choice read-out. This supports the claim that the state interface is a system-level control mechanism independent of specific dataset tooling.

Significance and Claims

The paper claims that Omni-Decision demonstrates that long-horizon multimodal control should not be left to implicit dialogue history. By treating the query-scoped evidence state as the central control object, the system can:

Target Evidence Acquisition: Directly identify and pursue specific missing evidence needs.
Preserve Sparse Cues: Maintain cross-modal cues that might otherwise be lost in long free-form trajectories.
Enable Inspectable Control: Provide a clear mechanism for repair, conflict resolution, and stopping based on the state of evidence closure.

The authors position Omni-Decision as a system-level control interface rather than a fixed multi-role implementation. They emphasize that the contribution lies in the read/write contract (modules consume a shared state, only normalized events are committed) rather than the specific number of roles or tools. The paper concludes that while the system significantly improves decision-making regarding what is missing and when to continue, it does not replace the need for low-level perception capabilities or calibrated uncertainty in the underlying models.