Technical Summary: Grounding Spatial Relations in a Compact World Model

1. Problem Statement

The paper investigates a critical failure mode in compact, goal-conditioned world models designed for embodied control. These models, often based on Joint-Embedding Predictive Architectures (JEPA), aim to represent spatial relations (e.g., "put the red block left of the blue block") using a combination of a global latent state and explicit metric reference anchors.

The central problem identified is Instruction Leakage. The authors demonstrate that when a world model's transition dynamics are conditioned on a language goal that explicitly names the relation being evaluated (e.g., the instruction "move X to the left of Y" contains the word "left"), the model does not learn to perceive the spatial relation from the scene. Instead, it learns to transcribe the instruction directly into its predicted anchor coordinates.

This creates a deceptive illusion of grounding:

• High Readout Accuracy: Models achieve high accuracy (e.g., 0.90) on relation readouts.
• Fragility: This accuracy collapses to chance levels (e.g., 0.27) when the goal is withheld.
• Counterfactual Failure: When presented with a counterfactual instruction (e.g., "move X to the right" when the scene shows "left"), the model's predicted anchors follow the false instruction 94.5% of the time, ignoring the actual scene.

The paper argues that this leakage occurs because the predictor can exploit the instruction as a shortcut, rendering the non-instruction inputs (state, action) irrelevant for the specific task of relation readout.

2. Methodology

Experimental Setup

The authors utilize a controlled 2D relational tabletop environment (inspired by CLEVR) where:

• Scenes: Contain $N=4$ colored shapes.
• Task: A target object must be moved relative to an anchor object (relations: left, right, above, below).
• Ambiguity: The number of duplicate distractors ($a \in \{0, \dots, 4\}$) is tunable to test referential ambiguity.
• Model Architecture (PrismWM): A JEPA-based model factoring state into:
  • A global latent $z$ (encoded by ViT-tiny).
  • Sparse metric point anchors $p$ (read from patch features).
  • A goal embedding $g$.
  • An action-conditioned transformer predictor that rolls $(z, p)$ forward.

Detection Protocol

To distinguish between genuine perception and instruction transcription, the authors employ two specific controls on the training render distribution:

1. Goal-Withheld Control: The goal tokens are zeroed out during the readout phase. A genuine perceptual model should maintain accuracy; a transcribing model should collapse to chance.
2. Counterfactual Control: The model is fed an instruction naming a different relation than the true scene. A genuine model follows the scene; a transcribing model follows the false instruction.

Model Variants

The authors train and compare several variants to isolate the cause:

• GoalDyn: Goal is fed into the dynamics (predictor).
• GoalDyn+Read: Goal in dynamics, plus supervised read path for observed anchors.
• GoalFree (The Fix): The goal is removed from the dynamics (the predictor never sees the goal) and is only used in the planner's cost function. The read path is supervised.
• NoGoal: Baseline with no goal conditioning.
• External Benchmarks: The phenomenon is validated on BabyAI (MiniGrid) and Language-Table (Google), varying whether instructions name the relation or just the referents.

3. Key Contributions

1. Characterization of Instruction Leakage: The paper establishes that leakage is governed by transcribability. It occurs when the instruction names the scored quantity (the relation) and is essentially independent of the predictive strength of non-instruction inputs (state/action). This was verified across three settings (Tabletop, BabyAI, Language-Table) and an action-ablation dose-response.
2. Validated Detection Protocol: The authors propose a rigorous testing protocol (Goal-Withheld and Counterfactual probes) that cleanly separates perception from transcription. They show that a goal-conditioned model's high accuracy is a "pure transcription" artifact.
3. Architectural Fix (Goal-Free Dynamics): The paper prescribes and validates a fix: remove the goal from the transition dynamics. By forcing the predictor to infer relations solely from the scene (without seeing the goal) and supervising the anchor read path, the model achieves genuine, instruction-independent grounding.
4. Control Analysis: The study reveals that goal-conditioning in dynamics actively degrades control performance. Removing the goal from dynamics recovers control performance to the level of the simplest "no-goal" baseline, confirming that the goal was the source of the degradation, not a lack of features.

4. Results

Representation and Grounding

• Leakage: The standard goal-conditioned model (GoalDyn) achieves ~0.90 predicted-anchor accuracy with the goal present, but this drops to ~0.27 (chance) when the goal is withheld. Under counterfactual instructions, it follows the false instruction 94.5% of the time.
• Genuine Grounding: The GoalFree model (goal removed from dynamics) achieves high accuracy (~0.88) that is identical whether the goal is present or withheld. This proves the anchors encode the relation based on perception, not instruction transcription.
• Ambiguity Boundary: Genuine grounding exists at low and medium ambiguity but collapses to chance at maximal ambiguity ($a=4$), indicating the learned grounding is narrowly tuned to training distribution cues rather than being a general relational perceiver.
• External Validation:
  • BabyAI: Relation-naming instructions cause leakage (counterfactual score 1.00).
  • Language-Table: Instructions naming only referents (no relation name) show no leakage. Augmenting these instructions to name the direction induces leakage.

Control Performance

• Monotonic Degradation: Control success rate decreases monotonically as the goal gains more influence over the dynamics:
  • NoGoal / GoalFree: ~0.30
  • GoalDyn (goal as predictor): ~0.14
  • GoalDyn (goal in rollout): ~0.03 (random floor)
• The Fix's Payoff: The GoalFree model ties the "NoGoal" baseline in control performance. The paper explicitly states it does not claim a control gain over the baseline; rather, the value of the fix is the recovery of grounded perception (which the NoGoal baseline lacks) and the removal of the active harm caused by goal-conditioning.
• Residual Gap: All variants perform significantly below the Oracle (true dynamics), which the authors attribute to dynamics fidelity (multi-step rollout drift) rather than the representation or grounding mechanism.

5. Significance and Claims

The paper's significance lies in identifying a subtle but pervasive "shortcut" in goal-conditioned world models where the instruction acts as a spurious signal, bypassing the need for visual perception.

• Methodological Lesson: High relation-readout accuracy in goal-conditioned models is not evidence of grounding if the instruction names the relation. It must be validated with goal-withheld and counterfactual controls.
• Architectural Lesson: The goal belongs in the planner's cost function, not in the transition dynamics. Conditioning the dynamics on the instruction creates a path for the model to copy the answer rather than predict the scene.
• Scope: The authors are modest about their claims. They do not claim to have solved 3D grounding or real-world robotics. They demonstrate that for instruction-naming tasks, the current "recipe" of conditioning dynamics on language is unsafe. The fix (Goal-Free dynamics) recovers genuine perception but does not inherently improve control beyond the simplest baseline; the remaining bottleneck is the fidelity of the learned dynamics.

The paper concludes that this leakage is a structural issue in designs that condition predictors on language when the language names the evaluated quantity, and that the proposed detection and fix are applicable to any such goal-conditioned world model.