💬 NLP

Sandboxed Coding Agents are Competitive Omni-modal Task Solvers

This paper demonstrates that text-only coding agents equipped with sandboxed tools can effectively solve complex audio-video tasks by converting multimodal problems into information-processing workflows, often outperforming native omnimodal models, while also introducing the Code-X training recipe and TerminalBench-O benchmark to advance open-source many-modality processing.

Original authors: Dongping Chen, Xuanao Huang, Zhihan Hu, Qingyuan Shi, Dianqi Li, Tianyi Zhou

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dongping Chen, Xuanao Huang, Zhihan Hu, Qingyuan Shi, Dianqi Li, Tianyi Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Smart Intern" vs. The "Super-Genius"

Imagine you have a massive, complex puzzle made of video clips, audio recordings, and documents. You need to solve it.

For a long time, the tech world assumed you needed a "Super-Genius" (a native omnimodal AI) to solve this. This Super-Genius is a model that has "eaten" the entire video and audio file whole, trying to understand every single frame and sound wave all at once inside its brain.

This paper argues that you don't actually need a Super-Genius for this. Instead, you can use a "Smart Intern" (a coding agent).

The Smart Intern doesn't try to swallow the whole video. Instead, it has a toolbox. It looks at the file, picks up a specific tool (like a video editor or a speech-to-text machine), extracts just the tiny piece of information it needs (like a transcript or a specific frame), and then solves the puzzle using that small piece of evidence.

The paper's main claim: The Smart Intern, using tools, is actually faster, cheaper, and often better at solving these puzzles than the Super-Genius who tries to memorize everything at once.

How the "Smart Intern" Works

Think of the Smart Intern as a detective in a library.

The Old Way (Native Models): The Super-Genius walks into the library and tries to read every single book, listen to every audio tape, and watch every movie on the shelves simultaneously. It gets overwhelmed, uses a lot of energy (tokens), and sometimes misses details because it's trying to process too much at once.
The New Way (Sandboxed Agents): The Smart Intern walks into the library but only takes the specific books or tapes it needs.
- If it needs to know what someone said, it uses a Speech-to-Text tool to turn the audio into a written transcript.
- If it needs to know what's in a video, it uses a Frame-Extraction tool to pull out a few key pictures.
- It then reads that transcript or looks at those pictures to answer the question.

The Result: The Smart Intern uses way less energy (fewer "tokens") because it isn't carrying the whole library in its head. It only carries the specific evidence it needs.

The Evidence: Who Wins the Race?

The researchers tested this on four different "puzzle competitions" (benchmarks) involving videos and audio.

The Results: The Smart Interns (using models like GPT-5.4 and Claude Opus) beat the best Super-Genius models (like Gemini 3.1 Pro) in several categories.
The Analogy: It's like a race between a person trying to memorize a whole encyclopedia to answer a trivia question versus a person who knows exactly which page to look up in a library index. The person with the index wins because they are efficient and precise.

Why Do They Win? (The "Tool" Advantage)

The paper found that the Smart Interns win because they treat the video/audio as raw materials to be processed, not as memories to be stored.

Selective Retrieval: Instead of watching a 90-minute soccer game to find one goal, the Intern uses a tool to scan the video for "goal" events and only looks at those 30 seconds.
Error Correction: If a tool fails (e.g., the speech-to-text machine gets confused by noise), the Intern can try a different tool or write a small script to fix it. A Super-Genius often just gets stuck on the noise.

What Goes Wrong? (The Failure Taxonomy)

Even the Smart Interns make mistakes. The researchers created a "failure menu" to explain why:

Bad Hearing: The speech-to-text tool misheard the audio.
Bad Eyes: The tool picked the wrong video frame.
Giving Up Too Soon: The agent stopped looking for clues before finding the answer.
Wrong Facts: It found the right video but looked up the wrong information in a database.
Bad Math: It had the right facts but did the calculation wrong.
Broken Tools: The computer didn't have the right software installed to run the tool.

Can We Teach the Intern to Be Better? (Skill Injection)

The researchers asked: "Can we make the Intern even smarter without rebuilding its brain?"

They tried three methods:

Human Coaching: Giving the Intern a manual written by experts.
Self-Practice: Letting the Intern try, fail, and adjust its own rules based on a simple "Right/Wrong" check.
Learning from Logs: Having a second AI watch the Intern's work logs, find patterns in what worked and what didn't, and write a new "Best Practices" guide for the Intern.

The Winner: The Log-driven Self-Distillation (Learning from Logs) worked best. It was like having a coach review the Intern's game tape and say, "You always forget to check the timestamp before guessing the time." This improved the Intern's accuracy significantly.

The Future: From "Understanding" to "Doing"

The paper argues that we are moving from an era where AI just understands media (answering questions about a video) to an era where AI processes media (editing the video, cutting the audio, creating a new file).

They introduced a new test called TerminalBench-O to measure this.

The Task: Instead of just asking "Who is in this video?", the AI must create a highlight reel, write a caption, and save the file.
The Challenge: This is much harder. Even the best AI only passed about 24% of these tasks. It requires long planning and reliable tool use, which is the next frontier for these "Smart Interns."

Summary

You don't need a giant, expensive brain that memorizes every video and sound to solve complex problems. You need a smart, efficient agent that knows how to use tools to extract the specific evidence it needs. This approach is cheaper, faster, and often more accurate than trying to "understand" everything at once.

Technical Summary: Sandboxed Coding Agents are Competitive Omni-modal Task Solvers

1. Problem Statement

As multimodal Large Language Models (LLMs) increasingly target video and audio, the prevailing assumption is that native omnimodal models—which directly ingest audio and video streams into their context—are necessary for effective task solving. However, deploying such models requires substantial engineering effort and modeling overhead. Conversely, many existing agentic systems (e.g., OpenClaw, Hermes) remain text-only or text+image systems.

This paper investigates a critical gap: Can coding agents, which lack native video/audio perception but possess access to a sandboxed terminal environment, solve omnimodal tasks by orchestrating external tools (e.g., ffmpeg, whisper, tesseract) to extract evidence, rather than relying on end-to-end perception? The authors challenge the necessity of native omnimodal architectures for tasks that can be decomposed into retrieval and information-processing problems.

2. Methodology

2.1. Sandboxed Agent Framework

The authors evaluate coding agents within isolated workspaces (Ubuntu 22.04) where raw media files (video, audio, images, documents) are staged as filesystem objects rather than packed into the model's context.

Input: The agent receives a natural language instruction and file paths.
Process: The agent interacts with the environment via a terminal interface. It writes and executes scripts to:
- Extract transcripts (using whisper/ASR).
- Sample frames or crop regions (using ffmpeg, ffprobe, OpenCV).
- Perform OCR (tesseract).
- Conduct web searches and metadata analysis.
- Synthesize intermediate artifacts (tables, logs, crops) to form a reasoning trajectory.
Output: A final answer parsed from the trajectory, typically formatted within a specific tag (e.g., <answer>).

2.2. Evaluation Benchmarks

The study utilizes four benchmarks with varying emphases:

OmniGAIA: Open-ended, tool-mediated factual and multi-hop omnimodal reasoning.
SocialOmni: Audio-visual social perception and identity recognition.
LVOmniBench: Long-form audio-video understanding.
VideoZeroBench: Challenging video-centric reasoning requiring spatio-temporal evidence.

The authors compare Sandboxed Coding Agents (using Codex and Claude Code harnesses with models like GPT-5.4 and Claude Opus 4.6) against Native Omnimodal LLMs (Gemini 3.1 Pro, Qwen 3.5-Omni, MiMo-V2-Omni) and Predefined Multimodal Agent Scaffolds (OmniAgent, Agent-Omni).

2.3. Failure Analysis and Skill Injection

To understand limitations, the authors constructed a Omni Terminal Failure Taxonomy based on 200 annotated trajectories, categorizing errors into:

Audio Perception and Extraction
Video Perception and Extraction
Insufficient Exploration of Modal Content
Knowledge Retrieval and Factual Errors
Logical Reasoning and Calculation
Tool and Environment Infrastructure Failures

They further explored Skill Injection to improve performance without retraining base models:

Calibration-set self-iteration: Agents revise skills based on binary correctness feedback.
Log-driven self-distillation: A separate agent mines reusable skills from raw execution traces.
Human-in-the-loop Skills: Manually written workflows encoding expert strategies.

2.4. Open-Source Elicitation (Code-X)

To demonstrate reproducibility on open-source models, the authors introduced Code-X, a training recipe involving:

OmniCoding Dataset: A 6,035-example trajectory dataset spanning video, audio, image, and cross-modal tasks.
Training Pipeline: Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) using Group Sequence Policy Optimization (GSPO).
Reward Design: A process-aware verifiable reward that penalizes modality-skipping, malformed tool calls, and answer mismatches, while rewarding correct format and tool engagement.

2.5. TerminalBench-O

Recognizing that understanding is distinct from processing, the authors introduced TerminalBench-O, the first process-level benchmark for coding agents on omnimodal processing tasks (e.g., editing videos, auditing databases). It evaluates agents on real-world tasks requiring long-horizon planning, tool chaining, and strict output compliance.

3. Key Results

3.1. Performance Competitiveness

Outperforming Native Models: On OmniGAIA, the GPT-5.4 xHigh coding agent achieved 75.0% accuracy, surpassing the strongest native omnimodal model, Gemini 3.1 Pro (66.1%), by over 8%. Claude Opus 4.6 also exceeded Gemini 3.1 Pro by 2.5%.
Efficiency: Coding agents consumed significantly fewer tokens than native omnimodal models. By selectively retrieving evidence (transcripts, frames) rather than ingesting entire media streams, they achieved competitive performance with a much smaller multimedia footprint.
Reasoning Depth: Increasing reasoning effort (e.g., from "Low" to "xHigh" in GPT-5.4) yielded clear performance gains, suggesting that agentic computation depth is as critical as perception capacity.

3.2. Tool-Use Dynamics

Staged Pipelines: Successful agents employed a staged pipeline: media extraction $\rightarrow$ transcription $\rightarrow$ external search $\rightarrow$ reasoning.
Key Tools: ffmpeg, ffprobe, whisper, and web_search provided the most consistent gains. python3 (self-authored tools) was frequently invoked but showed mixed reliability.
Text-Only Ablation: Even text-only agents (without direct image access) remained competitive on many benchmarks, indicating that many "omnimodal" tasks can be solved via text or shallow perception (OCR) rather than deep visual understanding.

3.3. Impact of Skill Injection

Skill injection substantially improved performance. The Log-driven self-distillation setting raised average OmniGAIA accuracy from 61.4% (No Skills) to 76.7%, outperforming both calibration-set self-iteration and human-written skills.
This suggests that execution traces provide richer supervision for refining procedural knowledge than binary feedback or static human guides.

3.4. Open-Source Results (Code-X)

The Code-X recipe enabled open-source models (Qwen-3.6-27B) to achieve state-of-the-art results among open-source models.
The 27B model reached 43.3% on OmniGAIA and 60.0% on LVOmniBench, surpassing all existing open-source coding agents and doubling performance on VideoZeroBench.

3.5. TerminalBench-O Challenges

Despite strong understanding results, TerminalBench-O proved difficult for current agents. Even the strongest GPT-5.5 Codex achieved only 24% pass rate.
Tasks requiring long-horizon planning, reliable tool chaining, and strict output adherence remain a significant bottleneck, highlighting the gap between understanding and active processing.

4. Significance and Claims

The paper argues that sandboxed coding agents represent a highly competitive, and often superior, paradigm for offline omnimodal task solving compared to native omnimodal models.

Paradigm Shift: The authors claim that for many tasks, the bottleneck is not the model's ability to "see" or "hear" raw data, but its ability to orchestrate tools to convert raw media into compact, verifiable evidence that fits a general-purpose reasoning interface.
Efficiency: This approach offers a more token-efficient pathway, avoiding the massive context costs of packing raw video/audio streams.
Future Frontier: The paper posits that the next frontier is Many-Modality Processing (editing, auditing, generating) rather than just understanding. The introduction of TerminalBench-O aims to shift evaluation focus toward these active, process-level capabilities where coding agents hold a structural advantage over end-to-end models.
Reproducibility: By releasing Code-X, OmniCoding, and TerminalBench-O, the authors provide a foundation for open-source research into multimodal agentic intelligence, moving beyond reliance on proprietary native omnimodal APIs.

In conclusion, the study demonstrates that "native" multimodal perception is not strictly necessary for high-performance omnimodal agents; instead, tool-mediated evidence extraction within a sandboxed environment can yield superior results in terms of both accuracy and efficiency for a wide range of video and audio tasks.