🤖 machine learning

Interaction Locality in Hierarchical Recursive Reasoning

This paper introduces "interaction locality," a task-geometry-aware framework that uses activation patching and feature ablation to demonstrate that hierarchical and recursive reasoning models (HRM and TRM) solve complex spatial tasks by accumulating local information writes into global structures, a pattern that contrasts with the boundary-concentrated locality observed in large-scale embodied 3D models.

Original authors: Yosuke Miyanishi, Tetsuro Morimura

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yosuke Miyanishi, Tetsuro Morimura

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: How Do AI "Think" About Space?

Imagine you are trying to solve a giant maze. To do it, you need two things working together:

Local moves: "I can't go right because there's a wall here."
Global plans: "I need to get to the exit, which is far away, so I should head north."

This paper asks a specific question about how AI models (specifically "recursive" ones that think in loops) handle these two things. Do they keep local details and global plans separate? Or do they get mixed up?

The authors created a new tool called "Interaction Locality." Think of this as a spatial heat map for information flow. It measures: If I poke a specific part of the AI's brain (like a single cell in a maze), does the reaction stay right there, or does it ripple out to change the whole picture?

The Tools: How They Tested It

To see how these AI models work, the researchers used three different "probes" (ways of poking the AI):

The "Sparse Feature" Check (SAE): Imagine the AI has a library of thousands of tiny, specific ideas (like "this is a red wall" or "this is a turn"). They turned these ideas off one by one to see which parts of the maze or puzzle they affected.
The "Noise Injection" Test (Activation Patching): This is the main test. They took a tiny bit of "static" or noise and injected it into one specific spot in the AI's memory. Then, they watched to see where that noise traveled. Did it stay in that one room? Or did it spread to the whole house?
The "Blueprint" Check (Jacobian/Attention): They looked at the mathematical wiring diagrams to see how the AI could theoretically connect things, even if it didn't always do so in practice.

The Experiments: Puzzles and 3D Worlds

They tested this on three types of puzzles and one real-world simulation:

Maze-Hard: Navigating a complex path.
Sudoku Extreme: Filling a grid with numbers based on rules.
ARC-AGI: Solving visual pattern puzzles (like "if this shape turns blue, the next one turns red").
MTU3D: A large-scale 3D robot navigating a real indoor room (like a ScanNet dataset).

What They Found

1. The "Local Writer" vs. The "Global Messenger"

In the puzzle models (HRM and TRM), they found a fascinating pattern:

The "High-Level" State (H): This is the part of the AI that holds the "big picture" plan. Surprisingly, when this part writes information, it tends to be very local. It writes notes to its immediate neighbors (e.g., "This specific cell in the Sudoku is a 5").
The "Low-Level" State (L): This is the part that does the nitty-gritty work.
The Magic of Loops: The key is that the AI repeats this process over and over. The "High-Level" part writes a local note, then passes it to the next loop, which passes it to the next. Over time, these tiny, local notes accumulate to build the global solution.

Analogy: Imagine a team of people passing a message down a long line.

Person A (High-Level) whispers a secret to Person B (Local).
Person B whispers it to Person C.
Eventually, the message reaches the end of the line.
The paper found that the "whisper" itself is very quiet and local, but the chain of whispers creates a loud, global announcement.

2. The 3D Robot Surprise (MTU3D)

When they tested this on the 3D robot navigating a real room, the pattern changed.

In the puzzles, the "local-to-global" handoff happened inside the thinking loops.
In the 3D robot, the "local" behavior only happened at the boundary where the robot's eyes (visual encoder) handed data to its brain (grounding module).
Inside the robot's visual brain, the information didn't stay local; it was mixed up everywhere.

Analogy:

Puzzle AI: Like a group of neighbors passing a bucket of water down a line to put out a fire. Each person only touches the bucket next to them, but the water moves across the whole street.
3D Robot: Like a camera taking a photo. The camera sees the whole room at once (global). The "local" part only happens when the camera hands the photo to the person deciding what to do next. The camera itself doesn't keep the "local" details separate; it blends them all together.

The Main Conclusion

The paper concludes that "Local" and "Global" are not fixed labels for different parts of an AI. Instead, they are relationships that depend on:

The Task: Is it a grid puzzle or a 3D room?
The Timing: Is it the first step of thinking or the tenth?
The Architecture: Does the AI use separate modules or one shared module?

The "Interaction Locality" framework proves that for these puzzle-solving AIs, the "Global Plan" is actually built by stacking many tiny, local updates on top of each other, rather than having a separate "Global Brain" that looks at everything at once.

In short: The paper gives us a new way to measure where an AI's thoughts stay and where they spread, showing that for recursive models, the "big picture" is just a pile of very careful, local notes passed around in a loop.

Technical Summary: Interaction Locality in Hierarchical Recursive Reasoning

Problem Statement
Spatial reasoning in artificial agents requires a dual capability: the ability to perform location-bound computations (e.g., moving a specific cell, avoiding a specific obstacle) while simultaneously maintaining location-invariant structures (e.g., global routes, constraint houses, or object-level plans). While compact recursive architectures like the Hierarchical Reasoning Model (HRM) and the Tiny Recursive Model (TRM) have demonstrated success on tasks like Maze-Hard, Sudoku Extreme, and ARC-AGI, the mechanistic understanding of how these models separate local execution from global planning remains largely hypothetical. Existing interpretability often relies on informal labels (e.g., "High-level state = global planning") which do not constitute mechanistic measurements of spatial reach. There is a need for a framework that can empirically measure whether information flow stays within nearby cells/semantic segments or crosses them, applicable across both symbolic puzzles and embodied 3D environments.

Methodology: Interaction Locality Framework
The authors propose Interaction Locality, a task-geometry-aware framework designed to measure the spatial reach of information flow within neural computations. The framework defines "locality" relative to the external geometry of the task (e.g., maze corridors, Sudoku 3x3 houses, or ARC-AGI object neighborhoods).

The methodology employs a three-tiered probe suite to instantiate this framework:

Sparse Autoencoder (SAE) Feature Ablations: Used to identify human-readable features and measure their impact within specific task segments (e.g., a specific corridor or Sudoku box). This provides semantic hypotheses but does not fully establish causal reach.
Finite-Noise Activation Patching: The primary causal test. The authors perturb a single activation vector at a source site $v$ with finite noise ( $\sigma \epsilon$ ) and measure the resulting activation difference at target sites $u$ . Locality is quantified as the fraction of the activation-change mass that remains within the task-defined neighborhood of the source. This method tests causal reach under non-infinitesimal changes.
Structural Checks (Appendix): Jacobian and attention analyses are used as structural diagnostics to report linearized topology and attention bias, serving as context for the finite-perturbation results rather than the primary causal evidence.

The framework is applied to:

HRM and TRM: Analyzed on Maze-Hard, Sudoku Extreme, and ARC-AGI. Critical analysis windows are selected based on state-change convergence (e.g., early reasoning steps).
MTU3D: A large-scale embodied 3D scene-grounding model tested on ScanNet indoor scenes. Here, neighborhoods are defined by metric object distance in 3D space.

Key Results

Local-to-Global Handoff in Recursive Models: Across Maze, Sudoku, and ARC-AGI, finite-noise patching reveals a consistent pattern: High-level (H) recurrent states tend to write information within nearby cells or same-segment units. While H-level writes are often more spatially concentrated than Low-level (L) writes, cross-cycle propagation allows these local summaries to be carried broadly through the recursive state.
- In TRM/Maze, the contrast is stark: within-H locality is high (0.580), while within-L is near baseline (0.032), suggesting the shared recursive operator concentrates local writes in the H-state.
- In HRM, H-level writes are also local, but the distinction is less pronounced in Sudoku due to the larger size of the "house" constraint unit.
Dissociation in Embodied 3D Models: When applied to MTU3D, the framework reveals a different dynamic. Causal spatial locality appears primarily at the transition where visual scene features are handed to the downstream grounding module (Stage 1–2 boundary). However, inside the unified visual encoder, causal recovery patching remains near the random baseline, despite structural Jacobian and attention scores showing spatial bias. This suggests that in embodied 3D models, causal spatial structure is concentrated at module boundaries rather than being uniformly distributed or emerging from recursive dynamics as seen in HRM/TRM.
Triangulation of Evidence: The paper demonstrates that SAE features provide readable segment-level hypotheses (e.g., features acting on specific corridors), while finite-noise patching confirms that these localities survive non-infinitesimal interventions. Structural Jacobians provide the linearized topology context.

Significance and Claims
The paper claims to shift the measurement paradigm for spatial reasoning interpretability from qualitative visualization to a reproducible, intervention-based framework.

Mechanistic Revision: The results revise the informal hypothesis that "H is global and L is local." Instead, the authors demonstrate that H-level states often perform local writes within a cycle, which are then propagated broadly across recursive time. Local and global computation are not static properties of a named state but are relations among the state, task geometry, perturbation scale, and recursive time edges.
Framework Scope: Interaction locality is presented as a diagnostic tool to determine whether local actions, object-level relations, and global plans are causally connected, rather than merely co-visible in a representation.
Limitations and Modesty: The authors explicitly state that the analysis compares released checkpoints rather than an accuracy-matched training sweep. They note that the framework is geometry-dependent (relying on defined neighborhoods) and that the dissociation observed in MTU3D (structural bias without causal locality) highlights that architectural spatial attention does not automatically guarantee causal local recovery. The paper does not claim to solve spatial reasoning but offers a method to audit how existing models handle spatial scale.

Conclusion
Interaction locality provides a unified coordinate system for comparing local movement, constraint propagation, and route-level planning across different architectures and domains. By grounding interpretability in finite interventions and task geometry, the framework offers a more rigorous way to understand how recursive and embodied models manage the separation of local execution and global planning.