💬 NLP

A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents

This paper proposes a framework that combines behavioral testing with interpretability analysis to evaluate goal-directedness in language model agents, demonstrating through a grid-world case study that while agents exhibit robust performance, their internal representations of spatial maps and action plans undergo non-linear reorganization during reasoning, highlighting the necessity of introspective examination beyond mere behavioral observation.

Original authors: Raghu Arghal, Fade Chen, Niall Dalton, Evgenii Kortukov, Calum McNamara, Angelos Nalmpantis, Moksh Nirvaan, Gabriele Sarti, Mario Giulianelli

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Raghu Arghal, Fade Chen, Niall Dalton, Evgenii Kortukov, Calum McNamara, Angelos Nalmpantis, Moksh Nirvaan, Gabriele Sarti, Mario Giulianelli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot friend who loves solving mazes. You tell it, "Find the green flag!" and it starts moving. But here's the big question: Does the robot actually know where the flag is and plan to get there, or is it just guessing and getting lucky?

This paper is like a detective story where the authors try to figure out exactly how this robot (a Large Language Model agent) thinks while it solves a maze. They didn't just watch what the robot did; they also peeked inside its "brain" to see what it was thinking.

Here is the breakdown of their investigation using simple analogies:

1. The Setup: The Robot in the Maze

The researchers put a robot (called GPT-OSS-20B) in a digital grid world, like a giant chessboard.

The Goal: The robot has to walk from a starting point to a goal.
The Obstacles: There are walls it can't walk through.
The Test: They made the mazes harder by making them bigger or filling them with more walls.

2. Part One: Watching the Robot's Feet (Behavioral Evaluation)

First, the researchers just watched the robot move. They compared the robot's path to the "perfect path" a human mathematician would calculate.

The Finding: The robot was pretty good. When the maze was easy, it walked straight to the goal. When the maze got harder (bigger or more walls), it made more mistakes, but it didn't just give up or wander randomly. It still seemed to be trying to reach the goal.
The "Mirror" Test: They took a maze and flipped it like a mirror image or rotated it. The robot still solved it just as well. This proved the robot wasn't just memorizing specific patterns; it actually understood the concept of the maze.
The "Distraction" Test: They put a shiny "Key" in the maze.
- Scenario A: The Key was needed to open a locked door to reach the goal. The robot figured this out perfectly.
- Scenario B: The Key was useless (no door). The robot still got distracted by it! It often walked toward the key even though it didn't need it. This suggests the robot has a "habit" of thinking keys are important, even when they aren't.

3. Part Two: Peeking Inside the Brain (Representational Evaluation)

Watching the feet isn't enough. The robot might be walking toward the goal by accident. So, the researchers used a special tool called a "probe" to read the robot's internal thoughts (its digital brain signals) without asking it to speak.

They looked at two different times:

Before the robot starts thinking: What does it see when it first looks at the map?
After the robot thinks: What does it see right before it decides to move?

The "Cognitive Map" (The Mental Picture)
They found that the robot builds a rough, fuzzy mental map of the room.

It knows roughly where it is and where the goal is.
It's not a perfect, high-definition photo; it's more like a sketch. It knows the goal is "somewhere over there," but it might be slightly off on the exact coordinates.
Crucial Discovery: When the robot starts "reasoning" (thinking hard), this fuzzy map gets blurrier. The robot stops focusing on the whole map and starts focusing intensely on the very next step. It's like a driver who stops looking at the whole highway and focuses entirely on the car right in front of them before making a turn.

The "Plan" (The To-Do List)
They also checked if the robot was holding a plan for the next few steps in its head.

Before thinking: The robot's brain held a long-term plan (a rough idea of the whole path).
After thinking: The long-term plan faded away, and the brain became very clear about the immediate next move.
The Takeaway: The robot doesn't just react; it actually holds a plan. But the act of "thinking" changes its brain from a "map reader" to a "step-by-step executor."

4. The Big Conclusion

The authors conclude that to truly understand if a robot is "goal-directed" (has a purpose), you can't just watch what it does. You have to look inside its brain, too.

The Good News: The robot does have a goal. It builds mental maps and plans steps to reach that goal.
The Bad News: Its mental map is a bit fuzzy, and sometimes it gets distracted by things that look like goals (like a key) even when they aren't.
The Lesson: If we want to trust AI agents in the future, we need to check both their actions and their internal thoughts to make sure they are actually trying to do what we want, and not just faking it or getting confused by distractions.

In short: The robot isn't just a mindless walker. It has a map, it has a plan, and it knows where it's going—but its map is a little blurry, and it sometimes gets sidetracked by shiny objects.

Technical Summary: A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents

Problem Statement

Understanding an agent's goals is essential for explaining and predicting its behavior, yet there is no established methodology for reliably attributing goals to agentic systems, particularly Large Language Model (LLM) agents. Purely behavioural evaluations face fundamental limitations: consistent failure may reflect capability constraints rather than a lack of goal-directedness, and aligned behavior can emerge from misaligned internal objectives (instrumental convergence). Conversely, purely internal analyses without behavioural grounding may fail to capture how agents actually interact with their environment. This paper addresses the need for a holistic framework that integrates behavioural assessment with interpretability-based analyses of internal representations to characterize how agents represent and pursue objectives.

Methodology

The authors propose a "white-box" framework combining behavioural evaluation with activation probing, tested on a GPT-OSS-20B agent navigating a fully observable 2D grid world (MiniGrid environment).

1. Experimental Setup

Environment: The agent operates in $n \times n$ grids ( $n \in \{7, 9, 11, 13, 15\}$ ) with varying obstacle densities ( $d \in [0, 1]$ ). The task is to navigate from a start position to a goal state.
Observability: The environment is fully observable to eliminate confounds related to memory, belief updating under uncertainty, and exploration-exploitation trade-offs.
Input Representation: Grids are converted into text-based representations where each cell corresponds to a single token to mitigate tokenization issues.
Baseline: An optimal policy ( $\pi^*$ ) is derived using A* search with Manhattan distance.

2. Behavioural Evaluation

The authors evaluate the agent's policy ( $\pi_\theta$ ) against the optimal policy across several dimensions:

Task Difficulty: Metrics include per-action accuracy, policy entropy, and Jensen-Shannon Divergence (JSD) from the optimal policy.
Iso-difficulty Transformations: To test robustness to incidental grid properties, the authors apply transformations that preserve task difficulty (grid size, obstacle density, optimal path length) but alter configuration: reflection, rotation, start/goal swapping, and transposition.
Complex Goal Structures:
- Instrumental Goals: Tasks requiring intermediate subgoals (e.g., collecting a key to unlock a door).
- Implicit Goals: Tasks containing goal-like objects that are functionally irrelevant (e.g., a key with no door).

3. Representational Evaluation (Probing)

The authors extract residual-stream activations from intermediate layers (7, 15, 23) at specific points: pre-reasoning (before the model generates reasoning tokens) and post-reasoning (after reasoning, before action selection).

Cognitive Maps: Linear and MLP probes are trained to decode cell types (wall, open, agent, goal) from activations, reconstructing the agent's internal "belief" of the grid state.
Plan Decoding: A Transformer decoder is trained to predict multi-step action sequences ( $T=10$ ) from a fixed set of activations using a one-shot decoding approach to ensure the plan structure is encoded in the base model, not the probe.
Intervention: Preliminary activation patching experiments were conducted to test causal links between representations and actions.

Key Results

Behavioural Findings

Sensitivity to Difficulty: Per-action accuracy decreases monotonically with grid size and obstacle density, while entropy and JSD increase. Performance is robust to distance-to-goal only in larger grids.
Robustness to Transformations: The agent's performance shows no statistically significant difference across iso-difficulty transformations (reflection, rotation, etc.), indicating behavior is driven by task-relevant information rather than grid configuration biases.
Goal Structures:
- The agent successfully handles instrumental goals (100% success in KeyDoorEnv).
- The agent is susceptible to implicit goals: In KeyNoDoorEnv (where the key is useless), the agent picks up the key in 17% of trajectories, and 75% of non-optimal actions move toward the key. In 2PathKeyEnv, the agent biases toward the path containing a vestigial key, slightly reducing overall success rates.

Representational Findings

Cognitive Maps: The agent non-linearly encodes a coarse spatial map. MLP probes achieve ~70% accuracy in decoding cell identities, with high recall for agent and goal positions (83–99%) but lower precision due to "blurring" (assigning labels to neighboring cells). Wall information is less detailed.
Reasoning Reorganization:
- Pre-reasoning: Activations preserve broader spatial cues and longer-horizon plans.
- Post-reasoning: Activations shift focus toward immediate action selection. Cognitive map accuracy drops significantly (from ~75% to ~60%) after reasoning, suggesting a reorganization of representations away from decodable spatial structure toward action-specific encoding.
Plan Decoding:
- One-shot decoding reveals that pre-reasoning activations contain recoverable long-horizon trajectory structure (best for prefixes $N \ge 5$ ).
- Post-reasoning activations are superior for immediate action selection (best for $N=1$ ).
- A 2-layer decoder performs best overall, suggesting immediate action information is accessible to low-capacity readouts, while multi-step structure requires more capacity.
Consistency: A substantial fraction of actions suboptimal in the ground-truth grid are optimal relative to the decoded cognitive map (Recovery rate: 37.4%–88.4%). This suggests many failures stem from inaccurate internal world models rather than a lack of goal-directedness.
Intervention Limits: Single-layer activation patching failed to alter actions. Only simultaneous patching across all layers successfully changed the model's predicted action, indicating that task-relevant state information is distributed rather than localized.

Contributions

Framework: Proposes a unified framework combining behavioural assessment and representation probing for evaluating goal-directedness.
Experimental Design: Introduces controlled environment perturbations (iso-difficulty transforms) and multi-goal structures to measure bias and robustness.
Insight into Mechanisms: Demonstrates that LLM agents encode cognitive maps and multi-step plans, and that reasoning reorganizes these representations from spatial/long-horizon cues to immediate action selection.
Coherence Analysis: Shows that behavioural deviations from optimality often align with the agent's internal (imperfect) beliefs, necessitating introspective examination beyond surface-level performance metrics.

Significance and Claims

The paper argues that introspective examination is required beyond behavioural evaluations to fully characterize goal-directedness. The authors claim that:

Goal-directedness is a rich property arising from the interaction of beliefs, planning, and action selection.
Purely behavioural metrics can be confounded by capability limitations or misaligned internal goals that produce aligned behavior instrumentally.
The observed "bias" toward goal-like but irrelevant objects (keys) may reflect a conflict between the explicit task prompt and semantic associations acquired during pre-training (e.g., keys imply progress), which are not consistently suppressed.
The framework provides a foundation for developing rigorous approaches to goal attribution, which is a prerequisite for safety claims regarding autonomous AI systems.

The authors remain modest, noting that their controlled setup abstracts away real-world complexity and that establishing causal links via activation patching remains an open challenge. They do not claim to have solved the alignment problem but rather provided a methodological step toward understanding the internal mechanics of goal pursuit in LLM agents.