🤖 AI

JarvisBench: Always-on Intelligence Between Humans and Agents

This paper introduces JarvisBench, a benchmark and framework designed to evaluate an always-on attention-coordination layer that mediates bidirectional interactions between intermittent human users and continuously executing long-horizon agents across diverse multi-agent tasks.

Original authors: Chen Chen, Zhehuai Chen

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Chen Chen, Zhehuai Chen

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, a new kind of machine has emerged: the autonomous agent. These are not simple tools that wait for a command to perform a single action, but rather digital workers capable of undertaking long, complex projects on their own. They can write code, analyze data, or manage schedules for hours without stopping. However, a fundamental mismatch exists between these tireless machines and the humans they are meant to serve. Human attention is a finite resource; we cannot watch a screen every second, nor can we constantly interrupt a machine to ask questions. Conversely, machines cannot read our minds to know when we need to step in or when they have made a mistake that requires our judgment. This creates a difficult coordination problem: if a human stops paying attention, the machine might drift off course, but if the machine constantly demands attention, it becomes inefficient and frustrating. The challenge for researchers is to build a system that bridges this gap, allowing a human to remain in the loop without being overwhelmed by the constant hum of digital work.

To solve this, a team of researchers at NVIDIA has introduced a new framework called JarvisBench, named after the fictional assistant in the Iron Man films, designed to test a specific type of "always-on" coordination layer. Imagine a busy office where a team of workers is building a complex structure. In the past, a manager might have to either stare at the workers constantly or trust them blindly until the project was finished. The new system acts as a dedicated supervisor who watches the workers, answers the manager's questions instantly, and knows exactly when to tap the manager on the shoulder for a critical decision. This supervisor does not do the building itself; it simply manages the flow of information and attention between the human and the working agents. The researchers created a rigorous testing ground to see if such a supervisor could actually improve the quality of the work and keep the human engaged without causing chaos.

The researchers built a benchmark consisting of 45 distinct scenarios, ranging from single tasks performed by one agent to complex projects involving ten different agents working together. These tasks spanned 19 different domains, from software development to creative writing, and were carefully selected from thousands of public candidates to ensure they were realistic. Crucially, the need for human help in these tests was not forced by leaving out instructions at the start. Instead, the need for human judgment arose naturally as the work progressed. For example, an agent might be writing a report and suddenly encounter a situation where it must choose between two valid but very different paths, neither of which was explicitly forbidden or mandated by the initial instructions. At that precise moment, the system had to recognize that a human decision was required, pause the work, ask the human for guidance, and then seamlessly resume the task with that new direction.

The study measured two main things to see if this coordination layer worked. First, they tested whether the system could successfully identify the right moments to ask for human help and whether doing so improved the final result. Second, they tested whether the system could answer a human's questions about the ongoing work without interrupting the workers. The results showed that adding this coordination layer consistently improved the performance of the agents. When the system was active, the agents completed their tasks with significantly higher scores, with improvements ranging from roughly 5 to 25 points depending on the complexity of the task and the specific agent involved. The system proved particularly effective in multi-agent projects, where the coordination between different workers and the human supervisor led to gains of up to 28 points.

However, the researchers found that not all "supervisors" were created equal. The performance of the system depended heavily on the specific artificial intelligence model used to run the coordination layer. One particular model, GPT-5.6-Sol, emerged as the most effective, delivering the highest quality answers to human questions and the largest improvements in task outcomes. Other models were also tested, and while they all showed some improvement over having no supervisor at all, they varied in how efficiently they used human attention. Some models asked for help more frequently but gained less in return, while others were more selective. The study also measured how quickly the system could respond to a human speaking, finding that the speed of the response depended on the specific technology used to process the voice and generate the reply, with some systems responding in under two seconds.

The researchers were careful to note that this system does not replace the worker agents or change how they think. Instead, it acts as an external layer that observes the work and manages the conversation. This design allows the same coordination system to be used with different types of agents without needing to rebuild the agents themselves. The study also highlighted a trade-off: asking for more human input can lead to better results, but it also consumes more of the human's time and attention. The system they built allows for this balance to be adjusted, letting users decide how much they want to be involved. While the current prototype serves as a reference point rather than a final product, it demonstrates that separating the task of "doing the work" from the task of "managing attention" is a viable and powerful strategy. By creating a dedicated interface that is always available to the human but invisible to the worker's internal logic, it is possible to keep complex, long-term projects aligned with human needs without requiring the human to stare at a screen the entire time.

Technical Summary: JARVISBENCH: ALWAYS-ON INTELLIGENCE BETWEEN HUMANS AND AGENTS

Problem Statement

The paper addresses a fundamental "bidirectional coordination problem" arising from the mismatch between rapidly advancing long-horizon agent capabilities and the intermittent, scarce nature of human attention.

User-to-Agent Direction: Users often need to query or guide agents while work is ongoing. Current systems often require interrupting execution, which reduces efficiency and pollutes the agent's working context with transient conversation.
Agent-to-User Direction: Agents may encounter consequential decisions requiring human judgment after the user has stopped monitoring. Without a mechanism to ensure the user is watching, agents are forced to guess, potentially drifting from user needs.
Proactivity Limitations: While "proactivity" (agents pausing to seek help) is a known capability, recent studies indicate current agents are poorly calibrated for it, as they are primarily trained for task completion rather than timing human intervention. Furthermore, proactivity alone does not provide a continuous interface for user-initiated queries.

The authors posit that an always-on attention-coordination layer is required to mediate this interface, allocating scarce human attention across one or more working agents without modifying their underlying execution loops.

Methodology: JarvisBench

To evaluate this coordination layer, the authors introduce JarvisBench, a benchmark designed to test an intermediary (Jarvis) in two complementary directions.

Benchmark Design

Architecture: JarvisBench separates three roles: the User (simulated by an LLM with a frozen profile), Jarvis (the intermediary), and Working Agents (executing tasks). Jarvis operates as an external sidecar, observing bounded task events and communicating via narrow guidance interfaces, leaving the worker's runtime and tools unchanged.
Task Suite: The benchmark contains 45 agentic task instances:
- 20 Single-Agent Tasks: Spanning 15 domains, these test whether Jarvis can recognize when a single agent's trajectory reaches a user-owned decision point.
- 25 Multi-Agent Workstreams: Organized into 10 projects (5 with two workstreams, 5 with three), these evaluate whether Jarvis can identify shared, project-level decision points that emerge as work unfolds.
Natural Emergence: Crucially, the need for user attention is not artificially created by withholding constraints in the initial prompt. Instead, consequential decision points arise naturally during execution.

Evaluation Tracks

JarvisBench defines two distinct evaluation tracks:

Agent-Collaboration Track (Agent-to-User): Evaluates whether Jarvis can identify uncertainty, intervene at the right moment, and solicit user judgment to improve task outcomes.
- Metrics: Task Outcome Score (0–100), Mean number of attention requests ( $\bar{N}_{req}$ ), and Attention Efficiency (score gain per requested user turn).
User-Interaction Track (User-to-Agent): Evaluates whether Jarvis remains useful and accurate when the user initiates contact.
- Protocol: Causal replay of worker trajectories. At 25% and 75% of execution, a "General" question is asked, followed by a "Follow-up" question generated based on Jarvis's previous answer.
- Metrics: Response quality (0–2 scale converted to 0–100) and Latency (time to first audio).

Experimental Setup

Reference Implementation: A full-duplex speech interface using Qwen3-ASR, Silero VAD, and Kokoro-82M TTS.
Models Tested: Various proprietary and open-weight models (e.g., GPT-5.6-Sol, Claude Opus 4.8, DeepSeek V4-Pro) were tested as both Workers and Jarvis brains.
Intervention Policy: A lightweight policy permits intervention when useful but discourages unnecessary requests. When intervention occurs, the worker pauses at an action boundary, the pending action is cancelled, and scoped soft guidance is injected before resuming.

Key Results

The paper reports experiments using a prototype implementation of Jarvis:

Universal Improvement: Adding Jarvis as an external layer improved every completed worker-task configuration without modifying the worker's harness.
- Single-Agent Gains: 4.9 to 24.7 points.
- Multi-Agent Gains: 12.5 to 28.2 points.
Jarvis LLM Performance:
- GPT-5.6-Sol achieved the highest task outcome gains and the highest user-interaction scores (96.3 Overall).
- DeepSeek V4-Pro demonstrated the highest Attention Efficiency on multi-agent tasks (0.41), achieving significant gains with fewer requests (0.30 per task).
- Claude Opus 4.8 and GPT-OSS-120B showed varying trade-offs between score gains and efficiency.
User Interaction: GPT-5.6-Sol provided the most accurate and grounded answers. Latency measurements (ranging from 1.3s to 2.1s) were noted as dependent on deployment conditions (API vs. local) and TTS strategies rather than intrinsic model speed.

Significance and Claims

The paper claims that JarvisBench provides a stable evaluation target for a critical, unsolved problem in agent systems: the bidirectional coordination of human attention.

Separation of Concerns: By decoupling agent execution from attention coordination, the benchmark allows for the evaluation of the intermediary layer independently of worker capabilities. This enables the comparison of different intervention policies, interaction models, and speech interfaces against a fixed set of tasks.
Beyond Proactivity: The work argues that proactivity is insufficient; a dedicated, always-on intermediary is necessary to handle both agent-initiated requests for help and user-initiated queries without disrupting execution.
Trade-off Exposure: The results demonstrate that there is no single "optimal" configuration. The choice of Jarvis LLM involves trade-offs between task outcome quality, attention efficiency (minimizing user interruption), and response latency.
Reference Control System: The authors position their implementation not as a final design, but as a reference control system to guide future research in agent runtimes and human-agent interfaces.

The paper concludes that attention coordination is both highly useful for improving agent outcomes and technically demanding, requiring the intermediary to understand unfolding work, recognize when human judgment matters, and translate that judgment into effective, scoped interventions.