Technical Summary: GPTNT – Benchmarking Real-Time Collaboration Between Multimodal Agents

Problem Statement

Multimodal Large Language Models (MLLMs) are increasingly deployed for collaborative tasks with humans and other agents. While existing benchmarks demonstrate fundamental capabilities in vision and language, they typically fail to evaluate the specific conditions that define real-time collaboration: time pressure, information asymmetry, and the necessity of efficient communication across a dynamic visual environment. Current benchmarks often isolate these factors, operate in static environments, or rely on turn-based protocols that do not penalize generation latency. Consequently, there is no unified environment to test whether models can coordinate effectively when one agent possesses visual information but no instructions, and the other possesses instructions but no visual access, all while a countdown timer runs.

Methodology

The authors introduce GPTNT, a benchmark built upon the cooperative video game Keep Talking and Nobody Explodes (ktane). The framework wraps the game to create a zero-shot evaluation environment for multimodal agents.

The Environment

• Asymmetric Roles: Two agents coordinate to defuse a procedurally generated bomb.
  • The Defuser: Can see the bomb (a 3D environment with various puzzle modules) but lacks the instruction manual. They must manipulate the bomb (rotate, zoom, click) based on instructions.
  • The Expert: Has access to the instruction manual (text and images) but cannot see the bomb. They must identify modules and provide instructions based solely on the Defuser's descriptions.
• Real-Time Constraints: The benchmark operates in asynchronous mode, where the game clock advances in wall-clock time regardless of model generation speed. This penalizes verbose reasoning and slow inference, making latency a core part of the task difficulty. A synchronous mode is also available for controlled experiments where the clock pauses during generation.
• Interaction Mechanism: Agents communicate via natural language. The Defuser uses a "Set-of-Marks" (SoM) approach to interact with UI elements, overlaying segmentation masks to avoid coordinate grounding errors.
• Task Complexity: Missions involve solving multiple independent puzzle modules (e.g., Wires, Simon Says, Morse Code) before the timer expires or the team accumulates three strikes (errors).

Evaluation Protocol

• Models Evaluated: Five state-of-the-art MLLMs were tested: GPT-5.2, Claude Sonnet 4.6, Gemini 3 Flash, Qwen3.5 (27B), and InternVL 3.5 (38B).
• Baselines: A human baseline consisting of ten pairs of players (including non-experts) was established.
• Metrics: Success is measured by mission completion (solving all modules), partial success (modules solved), and failure modes (timeout vs. strikeout).
• Diagnostic Ablations:
  • Single-Module Missions: To isolate specific module difficulties.
  • Offline VQA: "Manual VQA" tests the Expert's ability to reason over the manual; "Simulator VQA" tests the Defuser's perception and counting abilities.
  • Single-Agent Ablations: Models were tested with access to both the bomb and manual (removing communication) or without the manual (testing parametric knowledge/contamination).

Key Contributions

1. Unified Benchmark for Real-Time Collaboration: GPTNT is the first benchmark to simultaneously require coordination under asymmetric information, real-time asynchronous action, visual grounding in a dynamic environment, and sustained multi-turn communication.
2. Comprehensive Evaluation of Frontier Models: The paper provides a systematic breakdown of where current models fail, moving beyond aggregate scores to analyze performance across different roles, partner combinations, and task complexities.
3. Diagnostic Tools for Collaboration vs. Memorization: The framework includes specific evaluations to distinguish between genuine collaborative capability and reliance on memorized game solutions (data contamination), as well as tools to isolate low-level perceptual failures from high-level reasoning errors.

Results

Main Findings

• Total Failure in Full Setting: In the primary asynchronous, multi-module setting, none of the five tested models successfully defused a single bomb. In contrast, 9 out of 10 human pairs solved at least one mission, with the best pair solving six.
• Time Pressure as a Bottleneck: The dominant failure mode for models was timeout (running out of time), occurring in 50–90% of missions. Even in synchronous mode (where time pressure is removed), models struggled significantly, solving only a fraction of missions, indicating that the bottleneck is not just speed but the core capability to collaborate.
• Role Asymmetry: Performance varies significantly by role. The Defuser role has a larger impact on overall success than the Expert role. Models like GPT-5.2 perform better as Defusers, while others like Gemini 3 Flash perform better as Experts.
• Partner Compatibility: In "other-play" (different models paired together), communication rates increased, but success did not necessarily improve. Compatibility between specific model pairs was a stronger predictor of success than the individual capability of the partners.
• Low-Level Weaknesses: Offline evaluations revealed critical deficits:
  • State Tracking: Models struggle to maintain state across turns (e.g., remembering wire colors in Wire Sequence or sequences in Simon Says).
  • Counting: Defusers performed poorly on counting tasks (e.g., number of batteries), a known weakness in large models.
  • Hallucination & Ambiguity: Models frequently fail to detect when a partner provides ambiguous or hallucinated information, often proceeding with incorrect assumptions rather than requesting clarification.
• Contamination: Models demonstrated parametric knowledge of the game (performing better than random baselines without the manual), but this knowledge was insufficient to solve missions without the manual's specific conditional logic. Visual context appeared to scaffold recall for the Defuser but not for the Expert's procedural reasoning.

Significance and Claims

The paper claims that GPTNT exposes a critical gap in current MLLM capabilities: while models possess strong individual vision and language skills, they lack the integrated collaborative competence required for real-world, time-sensitive, asymmetric tasks.

• Ecological Validity: Unlike static benchmarks, GPTNT simulates the "messy" conditions of real-world deployment where information is incomplete, communication is imperfect, and time is limited.
• Dynamic Challenge: Because the benchmark relies on the procedurally generated nature of ktane and its active modding community, it is designed to remain challenging as models improve, avoiding the saturation issues common in static benchmarks.
• Diagnostic Value: The paper argues that current models fail not just due to a lack of knowledge, but due to fundamental weaknesses in state tracking, error recovery, and the ability to recognize and resolve ambiguity in real-time dialogue.

The authors conclude that GPTNT serves as a necessary step toward more holistic evaluation, revealing that the path to effective multimodal collaboration requires addressing these specific interactive and temporal limitations rather than simply scaling model size.