Technical Summary: VideoGameBench

Problem Statement

While Vision-Language Models (VLMs) have demonstrated proficiency in complex human-challenging tasks such as advanced mathematics and coding, their ability to perform tasks that rely on innate human inductive biases—specifically perception, spatial navigation, and memory management—remains understudied. Existing benchmarks often rely on simplified grid-worlds, text-only environments, or game-specific scaffolding (e.g., internal state access, visual overlays, or tailored tools) that do not reflect the constraints of real-world interaction. The paper posits that real video games, particularly those from the 1990s (the 32-bit era), are ideal testbeds because they are crafted to be intuitive for humans, leveraging these same inductive biases. The core problem addressed is evaluating whether frontier VLMs can complete entire video games using only raw visual inputs and high-level objective descriptions, without access to game-specific hints, internal state data, or human intervention.

Methodology

The authors introduce VideoGameBench, a benchmark comprising 10 popular video games from the 1990s (Game Boy, Game Boy Color, and MS-DOS platforms) and a development set of 13 games. The benchmark is designed with three novel features:

1. Realism and Complexity: It utilizes 2D and 3D environments requiring planning, puzzle-solving, and strategic thinking, moving beyond the simplified environments of prior work.
2. Generalization Testing: Three of the test games are kept secret to prevent overfitting and to test an agent's ability to adapt to unseen environments.
3. Minimal Scaffolding: Agents receive only raw game frames and a high-level description of controls and objectives. No visual overlays, internal RAM access, or game-specific tools are permitted.

Agent Framework (VG-Agent)

The evaluation employs a VG-Agent based on the ReAct (Reasoning, Acting, Observing) framework. The agent:

• Receives raw game frames as input.
• Outputs actions in natural language (e.g., press_key "A", click [x,y]).
• Maintains a textual scratchpad for memory and planning.
• Operates under strict constraints: no human intervention, no access to game state beyond the screen, and a limit on runtime to prevent infinite loops.

VideoGameBench Lite

To disentangle reasoning capabilities from inference latency (a major bottleneck in real-time settings), the authors introduce VideoGameBench Lite. In this variant, the game emulator pauses while the agent processes information and decides on an action, effectively converting real-time games into turn-based interactions.

Evaluation Metrics

Progress is tracked automatically using perceptual image hashing. The system scrapes checkpoint images from online walkthroughs (e.g., YouTube) and compares them to the agent's current game frame using Hamming distance. A score is calculated based on the percentage of the game completed relative to the furthest checkpoint reached.

Key Contributions

1. VideoGameBench Benchmark: A standardized, challenging benchmark for evaluating VLMs in real-time, complex, and diverse game environments without auxiliary information.
2. VideoGameBench Lite: A latency-controlled variant to isolate reasoning performance from reaction time constraints.
3. Automated Progress Tracking: A novel method for granular scoring using perceptual hashing against scraped walkthroughs, avoiding binary pass/fail metrics.
4. Empirical Analysis: A comprehensive evaluation of frontier models (GPT-4o, Claude 3.7 Sonnet, Gemini 2.5 Pro, Llama 4, etc.) revealing significant limitations in current VLM capabilities.

Results

The experiments reveal that frontier VLMs struggle significantly to progress beyond the initial stages of any game:

• VideoGameBench Performance: The best-performing models, Gemini 2.5 Pro and Claude 3.7 Sonnet, achieved an overall score of only 0.48%. Most models scored 0% on the majority of games, failing to reach even the first major checkpoint.
• VideoGameBench Lite Performance: Even when latency is removed, performance remains low. The best models achieved only 1.6% on the Lite subset. While performance improved quantitatively compared to the real-time setting, agents still failed to complete games or reach significant milestones.
• Practice Games: In simpler practice tasks (location clicking, mouse dragging, 2D navigation), models showed mixed results. While some models could complete basic clicking tasks, they struggled with dragging and navigation, suggesting fundamental weaknesses in spatial control and execution.

Failure Modes

The paper identifies three primary failure modes:

1. Knowing-Doing Gap: Models often correctly identify the goal (e.g., "move down to the door") but execute incorrect actions (e.g., pressing "down" repeatedly without aligning the character with the door), leading to stagnation.
2. Visual Processing Errors: Models frequently misinterpret visual inputs, such as believing an enemy is alive when it is dead, or hallucinating interactions with NPCs that did not occur.
3. Memory and Planning Deficits: Agents fail to track long-horizon objectives or game states. They often overwrite their memory, forget primary goals (e.g., delivering a package in Pokémon Crystal), or get stuck in repetitive loops (e.g., walking back and forth between zones in Doom II).

Significance and Claims

The paper claims that VideoGameBench formalizes the evaluation of human-like inductive biases (perception, spatial reasoning, memory) in AI agents. The results suggest that despite strong performance in coding and math, current frontier VLMs lack the robustness required for real-time, multi-modal interaction in dynamic environments.

The authors emphasize that the benchmark serves as a rigorous test for generalization and autonomy. By removing game-specific scaffolding, the benchmark highlights that current agents rely heavily on external aids rather than intrinsic reasoning capabilities. The authors hope that this formalization will motivate progress in developing agents that can operate in new, unseen environments, with potential implications for real-world embodied AI applications such as robotics and drone control. The paper maintains a modest tone, noting that while the benchmark is limited to 1990s games, it provides a necessary foundation for understanding the gaps between current AI capabilities and human-level task completion.