Technical Summary: MMTB: Evaluating Terminal Agents on Multimedia-File Tasks

Problem Statement

While terminal interfaces provide powerful environments for AI agents to automate complex workflows using diverse tools, existing benchmarks for terminal agents (e.g., Terminal-Bench, InterCode) predominantly focus on tasks grounded in text, code, and structured files. However, many real-world practitioner workflows—ranging from media production and compliance auditing to educational content creation—require direct interaction with audio and video files.

Current terminal agents face a significant limitation in these scenarios: they lack native perception of multimedia content. Instead of directly inspecting audio or video, agents must rely on command-line tools to convert media into intermediate representations (e.g., spectrograms, extracted frames, or transcripts) before reasoning. This conversion process introduces inefficiencies, potential information loss, and additional points of failure. Furthermore, existing benchmarks either lack multimedia-file tasks entirely or focus on multimedia understanding (answering questions about media) rather than multimedia action (executing terminal workflows to transform or analyze media files).

Methodology

Benchmark Construction: MultiMedia-TerminalBench (MMTB)

The authors introduce MMTB, a benchmark comprising 105 realistic tasks designed to evaluate terminal agents on multimedia-file workflows.

• Scope: Tasks are grounded in public sources reflecting paid practitioner workflows (e.g., Upwork, Fiverr) and cover five meta-categories: Media Production, Performance & Coaching, Enterprise & Compliance, Personal & Education, and Operations & Research.
• Format: Tasks follow the Harbor format used by Terminal-Bench, consisting of an instruction, a persistent containerized workspace with multimedia assets, an allowed tool interface, an output schema, and an artifact evaluator.
• Content: The benchmark includes 536 media files totaling nearly 7 hours of audio-visual content. Tasks require agents to ground decisions in auditory and visual evidence across files and produce verifiable output artifacts (e.g., edited clips, JSON logs, timestamps).
• Validation: The construction pipeline involved automated checks for task structure, Docker build integrity, and oracle solvability, followed by manual review to eliminate trivial shortcuts and unrealistic setups.

Evaluation Harness: Terminus-MM

To evaluate agents under controlled conditions, the authors propose Terminus-MM, a multimedia harness extending the Terminus-KIRA framework.

• Native Perception: Unlike standard terminal agents that rely on shell commands to extract features, Terminus-MM exposes native perception tools (view_image, listen_audio, watch_video) directly to the agent.
• Workspace-Aware Routing: A key design feature is dynamic tool masking. The harness scans the workspace at runtime and exposes only the perception tools relevant to the files present (e.g., hiding listen_audio if no audio files exist). This prevents agents from wasting resources on irrelevant modalities.
• Comparative Variants: The study evaluates a family of controlled harnesses with varying modality access (Text-only, Text+Image, Text+Audio, Text+Video, and Full Multimodal) to isolate the impact of specific perception capabilities.
• Baselines: The benchmark also evaluates off-the-shelf agents, specifically Codex CLI (with GPT-5.2) and Claude Code (with Sonnet-4.6), which operate through their own agent loops and tool interfaces without the controlled Terminus harness.

Metrics

Performance is measured using:

1. Success Rates: Binary success (passing the verifier threshold) and partial success (average correctness score).
2. Efficiency: Mean API cost per task and mean execution time.
3. Failure Analysis: Categorization of failure modes (e.g., tool setup timeouts, tool execution failures, model reasoning errors, incorrect output formats).

Key Results

1. Native Multimedia Access Significantly Improves Performance

The study demonstrates that text-only and text-image access are insufficient for many multimedia-file tasks.

• On the Gemini-3.1-Pro backbone, the text-only Terminus-2 achieved a binary success rate of 12.4%.
• Adding native audio and video access via Terminus-MM (T+I+A+V) increased the binary success rate to 37.1% and partial success to 46.9%.
• Similar trends were observed with Gemini-2.5-Flash, where full multimodal access yielded a 22.9% binary success rate compared to 6.7% for text-only.
• Cost Efficiency: Terminus-MM was also more cost-effective, often requiring fewer API calls and less time than agents forced to use command-line conversion tools to derive evidence.

2. Command-Line Conversions Introduce Significant Overhead

When native perception is unavailable, agents must use command-line tools (e.g., ffmpeg, tesseract, whisper) to inspect media.

• Table 5 shows that for tasks where both partial-modality harnesses and Terminus-MM succeeded, the partial harnesses incurred significantly higher costs. API cost ratios ranged from 1.63× to 7.72× higher than Terminus-MM, with worst-case scenarios reaching 42.49× when native audio was missing.
• This overhead stems from longer evidence-acquisition paths: selecting intermediate representations, running tools, interpreting lossy results, and retrying.

3. Distinct Failure Modes and Solver Regimes

• Terminus-MM failures were primarily driven by model reasoning (47% of failures), where the agent misinterpreted available evidence. Tool-operation failures (setup, execution, crashes) accounted for a smaller portion (24%).
• Codex CLI (relying on command-line conversions) showed a much higher failure rate in tool-mediated workflows. Setup and execution timeouts alone accounted for 39% of failures, with tool-operation issues comprising nearly half of all failures.
• Solver Regimes: Analysis of task overlap revealed that Terminus-MM and Codex CLI solve different subsets of tasks.
  • MM-only tasks (28 tasks): Often require native modality understanding (e.g., aligning audio cues with visual events).
  • Codex-only tasks (6 tasks): Often involve workflows where media can be reliably converted to text/numeric signals via standard pipelines (e.g., OCR, transcription).
  • Both-fail tasks (60 tasks): Represent the current frontier, requiring both precise media grounding and robust terminal artifact construction.

Significance and Claims

The paper claims that MMTB fills a critical gap in the evaluation of AI agents by shifting the focus from "multimedia understanding" (answering questions) to "multimedia action" (executing workflows). The authors argue that:

1. Native Access is Essential: For terminal agents to effectively handle real-world multimedia workflows, they require direct access to audio and video content, not just the ability to run conversion scripts.
2. Efficiency Gains: Native perception shortens the evidence-acquisition path, reducing both the cost and the likelihood of failure compared to command-line conversion pipelines.
3. Need for Hybrid Capabilities: Effective multimedia terminal agents must combine native perception with reliable tool-use abilities. The "MM-only" and "Codex-only" task sets suggest that future systems need to be versatile enough to handle both grounding-limited and pipeline-limited workflows.
4. Benchmark Hygiene: MMTB treats command-line workarounds for missing modalities as evidence of workflow complexity rather than contamination, providing a more realistic assessment of agent capabilities in persistent terminal environments.

The authors conclude that while progress has been made in terminal agents, the integration of native audio-visual grounding with reliable shell execution is necessary to advance agents capable of automating complex, file-based multimedia workflows.