💻 computer science

OmniReasoner: Thinking with Long Audio-Video via Native Tool Use

OmniReasoner is a tool-use post-training framework that enables omnimodal LLMs to efficiently reason over long audio-video streams by dynamically deciding when to invoke a "zoom-in" tool for high-fidelity inspection of sparse evidence, supported by a novel TimeAnchor mechanism for temporal consistency and a Temporal Augmented Data Engine for scalable training.

Original authors: Yu Chen, Caorui Li, Ziyu Xiong, Yidong Wang, Mingqi Gao, Shuman Liu, Biao Liu, Chunfeng Yang, Anxiang Zeng, Haibo Zhang, Chaofan Chen

Published 2026-07-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yu Chen, Caorui Li, Ziyu Xiong, Yidong Wang, Mingqi Gao, Shuman Liu, Biao Liu, Chunfeng Yang, Anxiang Zeng, Haibo Zhang, Chaofan Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery in a library that never ends. The library is filled with thousands of books, but the clues you need to solve the case are tiny: maybe a single word whispered on page 4,002, or a specific drawing hidden in the margin of page 12,500. If you try to read every single page at the same speed, your brain gets tired, and you might miss the tiny details because you were rushing through the whole thing. This is the exact problem computers face when they try to understand long videos and audio. They are "omni-modal" models, meaning they can see and hear, but when a video is an hour long, the most important clue might be a split-second sound or a brief visual action. If the computer tries to process the whole hour in high detail, it runs out of memory. If it speeds through everything, it misses the clues. Scientists have been trying to teach computers how to be better detectives: how to skim the whole story quickly, spot the suspicious part, and then slow down to look closely only where it matters.

Enter OmniReasoner, a new "thinking" framework that teaches these AI detectives how to use a special tool: a zoom-in button. Instead of staring at the whole hour-long video at once, OmniReasoner first takes a quick, low-quality "glance" at the entire timeline, kind of like flipping through a book to get the gist. If it thinks the answer is right there, it gives the answer. But if it senses a clue is hiding in a specific moment—like a character saying a specific phrase or a fish appearing in a bowl—it pauses and asks for a zoom-in. It requests a high-definition, high-speed clip of just that few seconds. The AI then looks at this "zoomed" evidence alongside its original quick glance to solve the puzzle.

The magic trick here is how the AI knows where to zoom. In the past, computers struggled because they counted "frames" (like counting pages), but if you speed up or slow down the video, the page numbers change. OmniReasoner uses a clever system called TimeAnchor. Think of it like a GPS coordinate based on the clock time (e.g., "look at 2 minutes and 49 seconds") rather than a page number. This way, whether the AI is looking at the blurry, fast-forwarded version or the crystal-clear zoomed-in version, "2:49" always points to the exact same moment. This ensures the AI doesn't get lost when switching between the big picture and the close-up.

To teach the AI this skill, the researchers didn't just ask humans to label every video; that would take forever. Instead, they built a Temporal Augmented Data Engine. Imagine a video editor that takes short clips, stitches them together into a long movie, and secretly hides a "clue" in one of the new seams. The AI is then trained to find that hidden clue. Sometimes the AI is told to just guess from the whole movie, and sometimes it's forced to use the zoom tool. Through a process of trial and error (reinforcement learning), the AI learns that using the zoom tool leads to better scores, but only when it's actually necessary.

The results show that this approach works. When tested on various video and audio challenges, OmniReasoner got significantly better at finding answers, especially in very long videos where clues are rare. It didn't just get smarter at answering; it learned to be efficient, spending its "brain power" only on the moments that mattered. The paper suggests that by teaching AI to decide when to look closer, rather than forcing it to look at everything equally, we can solve complex mysteries in long audio-video streams that were previously too difficult to crack. It's a step toward AI that doesn't just watch videos, but actually thinks about them, knowing exactly when to pause and pay attention.

Technical Summary: OmniReasoner

Problem Statement

Long audio-video reasoning presents a fundamental challenge for omnimodal Large Language Models (LLMs): the decisive evidence required to answer a question is often sparse, distributed across time, and cross-modal (involving both visual and auditory cues). Preserving every frame and audio segment at high fidelity is computationally prohibitive due to context window constraints and token costs. Conversely, uniform downsampling to fit long streams into the context often discards the fine-grained evidence necessary for accurate reasoning. Existing approaches either rely on fixed preprocessing, search for relevant segments using visual cues alone, or optimize reasoning after evidence selection, failing to address the core decision of when and where to allocate high-fidelity computation in a long audio-video stream.

Methodology

The authors introduce OmniReasoner, a tool-use post-training framework designed to teach omnimodal LLMs to "think with long audio-video" by adaptively deciding whether to call a zoom-in tool.

1. Inference Workflow

OmniReasoner operates in a two-stage process:

Global Preview: The model first consumes a low-cost, sparse global preview of the full audio-video stream (low-frame-rate video and full audio).
Tool Decision: Based on this preview and the user question, the policy decides between two actions:
- Direct Answer: If the global evidence is sufficient, the model answers immediately.
- Zoom-in: If more evidence is needed, the model calls a zoom-in tool, specifying a temporal interval $[s, e]$ .
Local Inspection: The system retrieves a high-fidelity local clip corresponding to the requested interval. The model then performs comprehensive reasoning using both the global context and the detailed local evidence to generate the final answer.

2. TimeAnchor

A critical technical contribution is TimeAnchor, which solves the problem of temporal grounding across different sampling granularities.

The Challenge: Standard frame indices or chunk IDs are relative to a specific sampling rate. A frame index valid in a sparse global preview does not correspond to the same physical moment in a dense local clip.
The Solution: TimeAnchor prepends a plain-text absolute time marker (e.g., <32 seconds>) to every two-second chunk of the interleaved audio-video token stream. This ensures that a temporal argument (e.g., "inspect 32–38 seconds") remains valid and round-trip consistent regardless of whether the model is viewing the sparse global stream or the dense local clip. The argument is tied to wall-clock time, not frame indices.

3. Temporal Augmented Data Engine

To train the model without expensive manual interval annotations, the authors built a Temporal Augmented Data Engine that synthesizes tool-use trajectories:

Task Construction: It creates long-form reasoning tasks via Multi-segment Composition (stitching independent clips) and Anomaly Insertion (replacing audio, video, or both in a window). The editing process inherently provides the ground-truth evidence intervals.
Trajectory Synthesis: Using a "MediaSandbox," the engine simulates the inference environment. It generates two types of trajectories:
- Tool-use trajectories: The model observes the global view, requests a zoom interval, receives the local clip, and answers.
- Direct-answer trajectories: The model answers directly from the global view.
Training Regime: The framework employs Supervised Fine-Tuning (SFT) on these synthesized trajectories to establish the tool-use protocol, followed by Group Relative Policy Optimization (GRPO) reinforcement learning. The RL reward is based on answer accuracy and format compliance, optimizing the policy to call the tool only when it improves the final answer.

Key Contributions

OmniReasoner Framework: A post-training approach enabling omnimodal models to learn adaptive tool use (zoom-in) for long audio-video reasoning via SFT and RL.
TimeAnchor: A mechanism that maintains temporal argument validity across fidelity boundaries (sparse vs. dense sampling) by using absolute time markers rather than frame indices.
Temporal Augmented Data Engine: A scalable data generation pipeline that synthesizes tool-use trajectories with known evidence intervals through video editing, eliminating the need for manual span annotation.

Experimental Results

Experiments were conducted on omnimodal and video reasoning benchmarks, including OmniVideoBench, LVOmniBench, Daily-Omni, WorldSense, VideoMME, and VideoHolmes.

Performance Gains: OmniReasoner (based on Qwen2.5-Omni-7B) significantly outperformed the base model. Notable improvements include:
- OmniVideoBench: +5.5 points (29.3 $\to$ 34.8).
- LVOmniBench: +3.4 points (32.0 $\to$ 35.4).
- VideoHolmes: +15.6 points (24.4 $\to$ 40.0), attributed to the ability to retrieve more frames via zoom-in compared to the baseline's fixed 32-frame limit.
Duration Sensitivity: The performance gap widened with video length. On OmniVideoBench, gains increased from +3.2 points for 0–5 minute videos to +9.9 points for 10–30 minute videos, demonstrating the efficacy of adaptive zoom-in for long-context sparse evidence.
Ablation Studies:
- Removing the self-curated temporal data caused significant drops, confirming the necessity of constructed evidence intervals.
- Removing the audio input or the tool-returned clip degraded performance, validating the reliance on cross-modal evidence and active local acquisition.
- TimeAnchor ablation showed that while the training recipe alone improves grounding, TimeAnchor provides substantial additional gains (e.g., +6.1 points on Charades-STA mIoU), proving its role in aligning temporal arguments across granularities.
- RL-only training without SFT warm-up failed, highlighting the need for a supervised cold-start to establish the tool-use format.

Significance

The paper argues that long audio-video reasoning requires a shift from uniform perception to adaptive evidence acquisition. OmniReasoner demonstrates that omnimodal models can be trained to act as agents that decide when to look and where to listen again, rather than passively processing a fixed context. By introducing TimeAnchor and a scalable data engine, the work addresses the specific technical hurdles of temporal grounding in multi-fidelity streams, offering a practical post-training recipe for improving reasoning accuracy in long-form audio-video tasks without requiring massive manual annotation. The authors note limitations regarding the current two-step tool scope and the need for more mature RL infrastructure for multi-turn agentic interactions in omnimodal settings.