🤖 AI

AGI Maze as a Benchmark Framework for World-Modeling Agents

The paper introduces AGI Maze, a lightweight grid-based benchmark framework designed to expose the inability of current large language models to construct persistent, manipulable world-state representations required for solving partially observable, stateful tasks, even when provided with working memory mechanisms.

Original authors: Alexey Potapov

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alexey Potapov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Why "Smart" Chatbots Get Lost in a Maze

Imagine you have a very smart friend who can write poetry, solve math problems, and tell jokes. This friend is like a Large Language Model (LLM). They are amazing at predicting what word comes next in a sentence.

However, the paper argues that if you put this friend in a dark, twisting maze where they can only see the wall right in front of them, they get lost. They might guess the next step, but they can't build a mental map of the whole maze in their head. They are great at finishing a story, but bad at simulating a world.

To prove this, the authors created a new test called AGI Maze.

What is AGI Maze? (The "Text-Based Video Game")

Think of AGI Maze as a video game you play entirely through text messages.

The Setup: You are in a grid (like a checkerboard). You have a start point, a treasure chest, a key, and an exit.
The Catch: You cannot see the map. You don't know where the walls are. You only know where you started.
The Gameplay: You type "Go Right." The game replies, "You hit a wall." Then you type "Go Down." The game says, "You found a key!"
The Goal: You have to figure out the layout of the maze, find the key, open the chest, and escape—all while remembering where you've been and where the walls are, without ever seeing a picture.

Why is this hard?
The maze has "tricks" to make it even harder:

Rivers: If you step on a river, you get swept away downstream automatically, but the game doesn't tell you which way the water flows. You have to guess.
Pits: If you fall in a hole, you might pop up in a different part of the maze entirely.
Time Limit: You only have a certain number of steps to finish. If you wander aimlessly, you run out of time.

The Experiment: Can AI Solve It?

The authors tested several popular AI models (like GPT-4o and Gemini) on these mazes. They treated the AI like a human player who only has the text descriptions to go on.

The Results were surprising:

The "Vanilla" AI: When the AI just tried to guess the next move based on the chat history, it failed miserably. On small mazes, it often did worse than a computer program that just picks a random direction (a "random-walk").
The Problem: The AI wasn't building a map in its "mind." It was just guessing the next word based on the last few sentences. It couldn't hold a stable picture of "where I am" and "where the walls are."

The "Notebook" Trick

The authors realized that humans can't solve a complex maze without a pencil and paper. We draw the map as we go. So, they gave the AI a "notebook" (by letting it write notes in its own chat history before making a move).

The Setup: Before the AI says "Go Right," it first writes a note: "I am at the bottom left. I went up and hit a wall. I think the key is to the right."
The Result: This helped the stronger AI models significantly. They went from failing to solving about 60-70% of the mazes.
The Catch: Even with the notebook, the AI still struggled. It didn't naturally build a perfect, structured map. It just wrote messy notes. And for the smaller, less powerful AI models, the notebook trick didn't help at all—they still got lost.

What Does This Tell Us?

The paper concludes with a few key takeaways:

Prediction vs. Understanding: Current AI is great at predicting the next word in a sentence (like finishing a story), but it is bad at simulating a changing world (like navigating a maze). It doesn't naturally keep a "mental model" of reality.
Memory is Hard: Just giving the AI a long list of past messages (its "memory") isn't enough. It needs to actively organize that information into a map or a plan, which it currently struggles to do on its own.
The Test is a Tool, Not a Final Score: The authors aren't saying "AI is useless." They are saying, "Here is a specific tool (AGI Maze) to show us exactly where AI is failing." It highlights that to build truly smart agents, we need to teach them how to build and use internal maps of the world, not just how to chat.

The Analogy Summary

Current LLMs are like a tour guide who has memorized a script. If you ask them to describe a city they've never visited, they can guess the words, but if you ask them to navigate a blindfolded maze, they stumble because they don't have a map.
AGI Maze is the blindfolded maze test.
The "Notebook" is giving the tour guide a pen and paper. It helps them a little, but they still aren't natural cartographers.

The paper argues that for AI to become truly intelligent (AGI), it needs to learn how to draw its own maps and update them as it explores, rather than just guessing the next sentence.

Technical Summary: AGI Maze as a Benchmark Framework for World-Modeling Agents

1. Problem Statement

Current Large Language Models (LLMs) function primarily as static next-token predictors rather than persistent world simulators. While effective in language tasks where the "world" is implicitly described by the text, LLMs struggle in interactive environments that require:

Persistent State Representation: Maintaining a manipulable, structured representation of an external world that evolves over time.
Partial Observability: Reasoning about hidden states (latent variables) when only a subset of the environment is observable.
Memory Efficiency: Tracking state across long horizons without relying on brittle, unstructured text history.
Belief Updating: Maintaining and updating beliefs under uncertainty, rather than simply inferring local rules from immediate observations.

Existing benchmarks like AGI-ARC-3 often fail to test these specific capabilities because they are effectively fully observable or do not require agents to construct a queryable, persistent world state. Furthermore, extensions of LLMs to embodied agents (e.g., Vision-Language-Action models) often fail to solve Simultaneous Localization and Mapping (SLAM) problems, relying instead on visual feature recall rather than constructing a coherent world model.

2. Methodology: The AGI Maze Framework

The authors introduce AGI Maze, a lightweight, grid-based framework designed to benchmark world-modeling capabilities in the language domain without the complexity of high-dimensional sensory inputs (pixels).

2.1 Environment Design

Core Mechanics: Agents navigate a grid (3x3 to 8x8) with walls, a "monolith" border with a single exit, a treasure chest, and a key.
Observations: The map is hidden. Agents receive textual feedback (e.g., "You tried to go right, but a monolith blocks the way") and must infer the layout, their location, and the positions of items.
Dynamic Uncertainty: To force genuine state inference, the framework includes mechanics that break naive exploration:
- Rivers: Trigger forced downstream movement without revealing the flow direction.
- Pit Cycles: Teleport agents to other pits in a cycle, creating ambiguity where the same observation can correspond to multiple locations.
Constraints: Agents operate under a strict step budget. Crucially, some mazes require specific action sequences (e.g., jumping into a river to reach a key) that cannot be solved by random exploration.

2.2 Framework Structure and Task Groups

The framework is organized into five task groups to prevent overfitting and test generalization:

TUTORIAL: Open-map mode for learning mechanics (not scored).
TRAINING: Small mazes with generous step budgets for calibration and baseline testing.
CLASSIC: Larger mazes under core rules, calibrated against human performance.
EXTENDED: Introduces new mechanics (e.g., boats, grenades, flashlights) to test the ability to infer new rules and strategies.
HIDDEN: A private hold-out set with unknown extensions to validate that performance is not due to dataset fitting.

2.3 API and Agent Interaction

The framework provides a stable HTTP API allowing agents to:

Retrieve game descriptions and parameters.
Submit actions (Up, Down, Left, Right, etc.).
Receive textual observations and structured state updates.
Agents are evaluated based on their ability to solve mazes within the step budget.

3. Experimental Setup and Baseline Results

The paper evaluates "vanilla" LLM agents that rely solely on the prompt history (game rules + action-observation pairs) without external tools or explicit state construction.

3.1 Vanilla LLM Performance

Setup: Several models (GPT-4o Mini, GPT-5 Mini, Gemini 3.1 Flash Light, MiniMax M2.7) were tested on TRAINING mazes.
Findings:
- None of the tested models reliably solved even simple mazes.
- GPT-4o Mini performed systematically worse than a random-walk baseline.
- While some models (e.g., Gemini 3.1 Flash) outperformed random on small mazes, all failed on larger grids (5x4 and above) and mazes with dynamic uncertainty (rivers/pits).
- Even stronger models (GPT-5.5, Gemini 3.5 Flash) achieved only ~30-40% success on difficult mazes with doubled step budgets, far below human performance.

3.2 Prompt-as-Working-Memory (Planning Agent)

To test if explicit state tracking helps, a "planning agent" was introduced. This agent uses a two-stage process:

Note-taking: It generates notes in the prompt to track coordinates, exploration status, and plans (acting as working memory).
Action: It selects an action based on these notes.

Results:
- Stronger Models: Performance improved significantly (e.g., GPT-5.5: 30% $\to$ 60%; Gemini 3.5 Flash: 40% $\to$ 70%).
- Lighter Models: Showed zero improvement on difficult tasks.
- Limitation: The improvement suggests that vanilla LLMs do not internally construct explicit latent representations of the world state during inference; they require externalizing this state into the text prompt to utilize it effectively. However, even with notes, the "memory-as-text" approach remains brittle and insufficient for reliable solving.

4. Key Contributions

AGI Maze Framework: A novel, low-bandwidth benchmark specifically designed to isolate and test world-modeling capabilities (state representation, belief tracking, long-horizon planning) without the confounding factor of visual perception.
Empirical Evidence of LLM Limitations: Demonstrates that vanilla LLMs fail to build persistent, manipulable world representations, often performing worse than random baselines in partially observable, stateful environments.
Differentiation of Memory Types: Highlights the distinction between internal latent state (which LLMs fail to maintain) and externalized text memory (which helps but is inefficient and brittle).
Extensibility: A framework that supports dynamic extensions (new items, mechanics) to test generalization and the ability to infer new rules from interaction rather than hardcoding.

5. Significance and Claims

The paper posits that AGI Maze is not an ultimate AGI benchmark but a diagnostic tool to identify specific gaps in current agent architectures.

Core Claim: The failure of LLMs in AGI Maze supports the hypothesis that next-token prediction does not inherently lead to the construction of explicit, coherent world-state models.
Implication: Robust world modeling requires mechanisms beyond standard inference, such as explicit external world models (maps, graphs, simulators) and better memory management strategies.
Future Direction: The framework is intended to guide research into agents that can construct and query external artifacts (e.g., programmatic simulators) and handle uncertainty and memory more effectively than current "text-as-memory" approaches.

The authors conclude that while AGI Maze isolates representation and reasoning as the primary bottleneck, solving these tasks will likely require architectural shifts beyond ad-hoc tooling and prompt engineering.