Technical Summary: When to Re-Plan: Subgoal Persistence in Hierarchical Latent Reasoning

Problem Statement

Long-horizon reasoning requires a system to balance stability (committing to a medium-horizon intent) with adaptivity (revising that intent when necessary). In externalized reasoning (e.g., Chain-of-Thought), each token implicitly commits the system to a specific path. However, in latent reasoning architectures (where computation occurs within hidden states rather than token traces), no such explicit commitment is forced. A latent reasoner can theoretically revise its medium-horizon intent at every micro-step or never.

The core problem addressed is determining the optimal temporal scale for planning in latent spaces: How long should a latent reasoner commit to an intent before revising it? Existing Hierarchical Reasoning Models (HRM) couple slow and fast loops but lack an explicit signal for medium-horizon intent, relying solely on emergent recurrent dynamics. This paper investigates whether introducing a "feudal-style" manager-worker interface with a controllable subgoal persistence period ($P$) can resolve the stability–adaptivity tradeoff.

Methodology

The authors propose Subgoal-Augmented HRM, an extension of the Hierarchical Reasoning Model (HRM) that operationalizes persistent latent intent through a manager-worker architecture.

Architecture and Mechanism

1. Manager-Worker Interface:

   • Manager (High-Level): A slow module that updates every $T$ low-level steps. It emits a normalized directional subgoal vector $g_k$ every $P$ low-level steps.
   • Worker (Low-Level): A fast module that performs micro-steps. Its hidden state updates are biased by the active subgoal $g(t)$ via a learned projection $V_L$.
   • Persistence Period ($P$): The central hyperparameter. It defines the number of low-level steps a subgoal remains active before being revised.
     • $P=1$: The subgoal is overwritten every step (no temporal coherence).
     • $P>1$: The subgoal persists, allowing the worker to compose multiple updates around a single intent.

2. Subgoal Emission:

   • The manager projects its state $z^H$ to a directional vector $\tilde{g}_k$, which is normalized to unit length to encode "where to move" in latent space without forcing a specific absolute target (which would be brittle in non-stationary dynamics).
   • A scalar commitment gate $\alpha_k$ optionally modulates the influence strength based on manager uncertainty.

3. Training Objective:

   • The total loss combines the baseline HRM objective ($L_{HRM}$) with an intrinsic alignment loss ($L_{align}$):
     $$L = L_{HRM} + \lambda \sum L^{(k)}_{align}$$
   • Alignment Loss: A cosine-based loss that rewards the net displacement of the worker's hidden state ($\Delta z^L_k$) over the commitment window $P$ for aligning with the emitted subgoal $g_k$.
   • Hyperparameter $\lambda$: The weight of the alignment loss, controlling the tradeoff between the alignment prior and the task gradient.

Experimental Setup

• Datasets: ARC-AGI and ConceptARC (specifically arc-aug-1000 and ConceptARC-mini).
• Metrics: Primary metric is Token-level LM Loss (cross-entropy); secondary metrics include accuracy and alignment loss.
• Sweeps:
  • Persistence Sweep: Varying $P$ (from 1 to 8) with fixed $\lambda=0.05$.
  • Alignment Weight Sweep: Varying $\lambda$ (from 0.01 to 1.0) with fixed $P=4$.
  • Ablation Study: A controlled three-cell experiment at $\lambda=0.10$ (past the optimum) to isolate the source of interference.

Key Results

1. Subgoal Persistence is the Central Knob

The study finds that persistence is necessary, not optional.

• Under-Commitment ($P=1$): When subgoals are revised every step, performance drops significantly (LM Loss 1.674) compared to the no-subgoal baseline (1.640). The injection mechanism alone, without temporal coherence, harms performance.
• Sweet Spot ($P \in [3, 6]$): Performance improves sharply at $P=3$ (LM Loss 1.544, a drop of 0.13 from baseline) and remains robust up to $P=6$.
• Over-Commitment ($P=8$): Performance decays only gradually as $P$ increases beyond the optimum. This indicates that staleness is a softer failure mode than instability; a slightly stale subgoal still provides directional consistency, whereas an instantaneously revised subgoal provides none.
• Replication: The $P=3$ configuration was replicated across 5 seeds, yielding a mean LM loss of 1.595 (std 0.045), confirming the result is not a single-seed artifact.

2. Narrow Optimum for Alignment Weight ($\lambda$)

• The alignment weight exhibits a narrow optimum at $\lambda \approx 0.05$.
• At $\lambda = 0.10$, performance approaches the baseline, and at $\lambda \ge 0.20$, it degrades below the baseline.
• This suggests the alignment loss functions best as a lightweight planning prior that gently shapes worker dynamics, rather than a dominant geometric constraint that competes with the task gradient.

3. Source of Interference (Ablation Study)

At $\lambda=0.10$ (past the optimum), the full mechanism underperforms the baseline. A controlled ablation isolated the cause:

• $A_{full}$ (Learned Directions): Underperforms baseline (+0.100 LM loss).
• $E_{random}$ (Random Directions): Performs nearly identically to the baseline (+0.003 LM loss).
• Conclusion: The interference is specifically driven by the learned directional structure, not the architectural capacity or the auxiliary loss term itself. When the alignment weight is too high, the learned directions capture representational capacity that the worker would otherwise use for the task, pulling the worker toward directions that serve alignment but not the LM objective.

Significance and Claims

The paper claims to identify a fundamental design principle for compositional planning in latent reasoning systems: medium-horizon intent must be coherent across enough computational steps for compositional structure to form.

• Design Principle: A subgoal mechanism without persistence ($P=1$) possesses the architectural form of compositionality (manager, worker, intent vector) but lacks its function. It cannot produce the temporally extended structure required to distinguish "planning a sequence of steps" from "taking a single step with extra signal."
• Stability-Adaptivity Tradeoff: The findings quantify the tradeoff, showing that latent planners require a minimum coherence horizon ($P \ge 3$) to organize computation, but that staleness (long $P$) is more tolerable than instability.
• Role of Alignment: The alignment signal acts as a soft prior. It is most effective when it guides the worker without overriding the task-specific gradients, evidenced by the narrow optimum at $\lambda \approx 0.05$.

The authors frame this as a contribution to compositional learning, demonstrating that explicit medium-horizon intent signals, when persisted correctly, enable the assembly of subgoal-conditioned hidden-state updates into longer-horizon computation. The work is empirically modest, noting that gains are relative to the baseline and restricted to ARC-style tasks, but the identified tradeoff is presented as a generalizable constraint for hierarchical latent reasoning.