💻 computer science

PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models

This paper introduces PokeGym, a scalable, visually-driven long-horizon benchmark within the 3D open-world game *Pokémon Legends: Z-A* that evaluates Vision-Language Models on complex embodied tasks and reveals that physical deadlock recovery, rather than high-level planning, is the primary bottleneck, with distinct failure modes observed across different model capabilities.

Original authors: Ruizhi Zhang, Ye Huang, Yuangang Pan, Chuanfu Shen, Zhilin Liu, Ting Xie, Wen Li, Lixin Duan

Published 2026-04-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ruizhi Zhang, Ye Huang, Yuangang Pan, Chuanfu Shen, Zhilin Liu, Ting Xie, Wen Li, Lixin Duan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to play a video game. But this isn't just any game; it's a massive, 3D open-world adventure like Pokémon, where the robot can't see the game's code, it can't ask the computer for its coordinates, and it can't just "read" the map. It has to learn entirely by looking at the screen, just like a human does.

This paper introduces PokeGym, a new "gym" (a testing ground) designed to see how good these AI robots are at playing such games.

Here is the breakdown in simple terms:

1. The Problem: The "Cheating" Robot

Previously, when scientists tested AI on video games, they often let the AI "cheat."

The Old Way: They gave the AI the game's internal data (like "You are at X, Y, Z coordinates" or "The door is 5 meters away"). It was like giving a student the answer key before the test. The AI didn't actually have to see or understand the world; it just processed numbers.
The New Way (PokeGym): The AI is blind to the code. It only sees raw pixels on a screen. It has to figure out, "Oh, that looks like a door," and "If I walk forward, I might hit a wall." This is much harder and much more realistic.

2. The Test: A Pokémon Adventure

The researchers built this test inside the game Pokémon Legends: Z-A.

The Mission: The AI has to complete long, complex quests (like "Go find the hotel owner and talk to him").
The Challenge: These quests take hundreds of steps. The AI has to navigate crowded streets, avoid getting stuck behind fences, and figure out which buttons to press.
The Twist: The researchers tested the AI with three different levels of help:
1. Visual-Guided: "Go to the red door, then turn left to the blue house." (Easy, lots of clues).
2. Step-Guided: "Go to the house, then find the owner." (Medium, no visual clues, just instructions).
3. Goal-Only: "Find the hotel owner." (Hard, the AI has to figure out the whole plan itself).

3. The Big Discovery: The "Stuck" Problem

The researchers found something surprising. They thought the AI would fail because it couldn't plan big, complex strategies. They were wrong.

The AI actually failed mostly because it got physically stuck.

The "Deadlock" Analogy: Imagine you are walking down a hallway and you get stuck between a wall and a potted plant. You keep trying to walk forward, but you just bump into the plant.
- Weak AI: It doesn't even realize it's stuck. It keeps saying, "I'm making progress!" while it just spins in circles. This is called an "Unaware Deadlock."
- Strong AI: It realizes, "Hey, I'm stuck!" But then it panics. It tries random things (jumping, spinning, hitting the wall) and still can't get free. This is called an "Aware Deadlock."

The Lesson: The biggest bottleneck isn't "thinking"; it's "moving." The AI needs better "spatial intuition"—the ability to understand 3D space and how to wiggle out of tight spots.

4. The Scoreboard

The paper tested many famous AI models (like GPT-5, Gemini, Claude, and Qwen).

The Winners: The big, expensive, closed-source models (like Gemini and GPT) did the best. They were better at realizing they were stuck and trying to fix it.
The Losers: Smaller or open-source models often got stuck and didn't even know it.
The Cost: Running these tests is expensive. Some models "ate" hundreds of thousands of words (tokens) just to try to solve one simple quest, costing the researchers money every time they ran a test.

5. Why This Matters

This paper is a wake-up call for AI researchers.

Current AI is good at "talking" but bad at "walking." It can write a poem or answer a math question, but if you put it in a 3D world, it trips over its own feet.
The Future: To build a truly helpful robot (one that can clean your house or drive a car), we need to teach it how to recover when it gets stuck, not just how to plan the perfect route.

In a nutshell: PokeGym is a rigorous obstacle course for AI. It proved that while our AI is getting smarter at understanding language, it's still very clumsy at navigating the physical world, often getting stuck in corners and not knowing how to get out.

1. Problem Statement

Despite significant progress in static visual understanding, Vision-Language Models (VLMs) struggle when deployed in complex, 3D embodied environments. Existing benchmarks suffer from four critical deficiencies that prevent a faithful evaluation of embodied intelligence:

Passive Perception: Most tasks (e.g., VQA) are single-turn and static, bypassing the need for interactive dynamics and persistent planning.
Simplified Visuals: Many interactive benchmarks use 2D grids or simplified worlds that fail to test depth perception and 3D spatial reasoning.
Privileged State Leakage: Realistic 3D simulators often expose internal states (coordinates, block IDs) to the agent, allowing it to bypass genuine visual processing.
Scalability Issues: Benchmarks relying on pure visual inputs often require expensive, subjective human evaluation, limiting reproducibility and scale.

The core challenge is creating a benchmark that enforces pure visual decision-making (raw pixels only) while enabling automated, scalable, and objective evaluation in a complex 3D open world.

2. Methodology: The PokeGym Benchmark

The authors introduce PokeGym, a benchmark instantiated within the 3D open-world RPG Pokémon Legends: Z-A.

A. Environment & Isolation

Game Engine: Built on Pokémon Legends: Z-A via the Ryujinx emulator (C#).
Pure Visual Input: Agents operate solely on raw RGB observations. No privileged state (coordinates, inventory lists) is exposed to the agent.
Observation Space: Agents receive a configurable set of images:
- Current Front View (default).
- Previous Frame (for temporal feedback).
- Left/Right Views (for expanded spatial awareness).
- Note: Images are extracted directly from GPU textures to minimize latency and avoid window occlusion.

B. Automated Evaluation Pipeline

To solve the scalability problem, PokeGym uses an independent evaluator that does not rely on the agent's output:

AOB (Array of Bytes) Memory Scanning: The evaluator scans the emulator's memory at runtime using signature patterns to locate critical variables (Map ID, X/Y/Z coordinates, Quest Flags).
Objective Verification: Success is verified by checking if memory variables meet specific thresholds (e.g., coordinates within a bounding box). This ensures zero human bias and allows for automated, large-scale testing.

C. Task Design

The benchmark comprises 30 long-horizon tasks (30–220 steps) derived from 10 distinct quests, categorized into:

Navigation: Moving to specific locations.
Interaction: Interacting with objects/NPCs.
Mixed: Combining navigation, interaction, and combat.

Instruction Granularities: To diagnose specific cognitive bottlenecks, tasks are presented in three formats:

Visual-Guided: Provides procedural steps with visual anchors (e.g., "Enter the door behind the red pillar"). Tests visual grounding.
Step-Guided: Provides procedural steps but removes visual anchors (e.g., "Enter the door"). Tests semantic reasoning.
Goal-Only: Provides only the ultimate objective (e.g., "Talk to the hotel owner"). Tests autonomous exploration and long-horizon planning.

D. System Architecture

Decision Module: VLMs generate actions based on visual context. An optional Self-Reflection module allows the model to summarize history and update short-term memory every $k$ steps.
Action Interface: Supports two paradigms:
- High-level Actions: Discrete commands (e.g., MoveForward, RotateLeft).
- Parametric Control: Continuous joystick values and durations.
Adaptive Pause: For time-sensitive tasks (like combat), the environment pauses during reasoning to prevent inference latency from affecting performance.

3. Key Contributions

PokeGym Benchmark: The first scalable, automated benchmark for VLMs in a complex 3D open world that strictly enforces pure-pixel inputs.
Rigorous Evaluation Pipeline: A novel method using AOB memory scanning to verify success objectively, eliminating the need for human evaluation.
Diagnostic Framework: A controlled setup using three instruction granularities to disentangle visual grounding, semantic reasoning, and autonomous exploration.
Comprehensive Failure Analysis: A deep dive into why models fail, identifying physical deadlocks as the primary bottleneck rather than high-level planning.

4. Key Results & Findings

The authors evaluated 8 VLMs (including GPT-5.2, Gemini-3-Pro, Qwen3.5, and Claude-Sonnet-4.6).

A. Performance Overview

Proprietary vs. Open-Source: Proprietary models (Gemini-3-Pro, GPT-5.2) generally outperform open-weight models, particularly in semantic reasoning and handling sparse instructions.
Instruction Sensitivity:
- Gemini-3-Pro improved significantly when visual anchors were removed (Step-Guided), suggesting dense visual cues can act as distractors for it.
- Qwen models degraded significantly without visual anchors, indicating a strong dependence on explicit visual grounding.

B. The Primary Bottleneck: Physical Deadlocks

Contrary to the assumption that high-level planning is the main failure point, the study found that physical deadlock recovery is the primary bottleneck.

Correlation: There is a strong negative correlation ( $r \approx -0.6$ ) between Ineffective Moves (collisions resulting in zero displacement) and Task Success.
Recovery Behavior: Successful episodes show "bump-and-recover" behavior (low entropy, deterministic recovery). Failed episodes show "flailing" (high entropy, erratic actions).

C. Metacognitive Divergence

The study uncovered a distinct difference in how models handle being trapped:

Unaware Deadlocks (Weaker Models): The agent is physically stuck but hallucinates progress, believing the path is clear. It fails to recognize the collision.
Aware Deadlocks (Stronger Models): The agent correctly identifies it is stuck but fails to execute a valid recovery maneuver. It recognizes the problem but lacks the spatial intuition to solve it.

D. Impact of Visual Context & Actions

Temporal vs. Spatial: Adding previous frames (temporal reflection) consistently improved success rates. Adding Left/Right views (spatial) helped Interaction tasks but hurt Navigation tasks (likely due to distraction).
Action Granularity: High-level actions with multi-step planning (predicting 3 actions at once) performed better than single-step parametric control, which suffered from error accumulation.

5. Significance

Paradigm Shift: PokeGym moves VLM evaluation from static recognition to dynamic, embodied interaction in realistic 3D environments.
Diagnostic Value: By isolating "Unaware" vs. "Aware" deadlocks, the benchmark provides actionable insights for model architecture improvements (e.g., integrating explicit spatial intuition vs. improving recovery strategies).
Scalability: The automated evaluation pipeline sets a new standard for evaluating embodied agents, removing the cost and bias of human annotation.
Future Directions: The findings suggest that future VLMs need better spatial intuition and error-recovery mechanisms rather than just better planning or larger context windows. The authors plan to release the benchmark for reinforcement learning training in the future.

Conclusion: PokeGym reveals that current VLMs are often "blind" to physical constraints or lack the micro-level control to escape them. Success in complex 3D worlds requires not just understanding the world, but robustly interacting with its physical laws.