What's the plan? Metrics for implicit planning in LLMs and their application to rhyme generation and question answering

Technical Summary: "What's the Plan? Metrics for Implicit Planning in LLMs and Their Application to Rhyme Generation and Question Answering"

Problem Statement
While Large Language Models (LLMs) are trained on next-token prediction, there is a growing debate regarding whether they exhibit "implicit planning"—the ability to represent future goals in hidden activations to condition intermediate generation. Prior evidence, such as the study by Lindsey et al. (2025) on Claude 3.5 Haiku using cross-layer transcoders, suggested that models might plan ahead for rhyming words. However, such findings were limited to a single closed-source model and relied on computationally expensive, difficult-to-reproduce methods. Furthermore, skepticism remains regarding the general planning capabilities of LLMs, with some arguments suggesting that constant-time token generation precludes principled reasoning. This paper aims to establish whether implicit planning is a universal mechanism present across diverse model families and sizes, and to develop scalable, quantitative methods to assess it.

Methodology
The authors propose a suite of simpler, scalable techniques to detect and quantify implicit forward and backward planning, avoiding the need for complex dictionary learning or cross-layer transcoders. The methodology relies on mean activation difference steering (Arditi et al., 2024) applied to specific positions in the input sequence.

1. Definitions of Planning:

   • Forward Planning: The creation of representations at earlier positions encoding properties of future goal tokens.
   • Backward Planning: The use of those representations to condition the generation of intermediate tokens leading up to the goal.
   • Implicit Planning: Representations exist within hidden activations rather than explicit outputs (e.g., Chain of Thought).

2. Experimental Tasks:

   • Rhyme Generation: Models are prompted to complete a couplet. The goal is to predict the rhyme family of the second line based on the first.
   • Question Answering (QA): Models answer questions where the target noun determines the grammatical article ("a" vs. "an"). The goal is to predict the noun answer.

3. Steering Procedure:

   • Vector Estimation: Steering vectors are calculated as the mean difference in hidden activations between two categories (e.g., lines ending in "-ight" vs. "-ake") at specific positions (e.g., the last word of the first line, the newline token, or the question mark).
   • Intervention: During generation, the steering vector is added to the residual stream at the chosen position and layer.
   • Metrics:
     • Forward Planning: Measured by the "Fraction of Correct Rhyme Family/Answer" in steered generations.
     • Backward Planning: Measured by "Regeneration Metrics." The model is asked to regenerate the final word of a generated line without the original context. If the model successfully planned backward, the regenerated word should still align with the intended rhyme/answer family.
     • Probability Metrics: Analysis of KL divergence and top-1 token shifts in intermediate steps to detect changes in probability distributions caused by steering.

4. Datasets and Models:

   • Datasets: 1,050 lines for 10 rhyme families and 500 questions for 20 noun pairs (distinguishing between vowel/consonant-starting nouns).
   • Models: 23 open-weight models across four families (Gemma2, Gemma3, Qwen3, Llama3.1/3.2) ranging from 1B to 32B parameters, including both base and instruction-tuned versions.

Key Results

• Universality of Implicit Planning: Evidence of successful forward and backward planning was found across all 23 models, including those as small as 1B parameters. This suggests planning is a universal mechanism, not limited to massive models.
• Steering Effectiveness:
  • Steering at the end of the first line (last word or newline token) successfully altered the rhyme family generated in the second line.
  • Steering successfully shifted article choice ("a" vs. "an") in QA tasks before the noun answer was generated, confirming backward planning.
  • Instruction-tuned models generally exhibited stronger planning behaviors and higher steering effectiveness than their base counterparts.
• Backward Planning Validation: Regeneration metrics showed that lines generated under steering were more likely to be completed with the target rhyme/answer even when the original context was removed. This indicates that the planning representation influenced the intermediate tokens, not just the final output.
• Circuit Analysis (Gemma2 9B): The authors identified specific attention heads (L30H3 and L31H15) that read from the steering direction and attend to the first line's last word and newline token. Patching these heads with steered activations recovered 59–93% of the steering effect, suggesting a dedicated circuit for rhyme planning.
• Model Differences: While the mechanism is universal, the specific implementation varies. For instance, Gemma2 9B and Gemma3 27B utilize the newline token for planning, whereas Llama models rely primarily on the last word token.

Significance and Claims
The paper claims to provide the first broad, quantitative evidence that implicit planning is a pervasive feature in LLMs, present even in models with 1B parameters. By introducing a methodology that scales easily across many models without expensive training, the authors offer a direct way to study these abilities.

The authors emphasize that their findings support the view that LLMs engage in heuristic planning similar to humans, rather than purely reactive token prediction. They argue that understanding these planning circuits is critical for AI safety and control, as implicit planning allows models to manipulate intermediate steps to achieve specific, potentially hidden, goals. The study concludes that while the specific circuits differ between models and tasks (rhyming vs. QA), the underlying capability to represent and act on future goals is a general property of the architecture.

Limitations and Future Directions
The authors note that specific word steering (e.g., forcing a specific word like "rabbit" rather than a rhyme family) is less effective than family-level steering, suggesting that specific word planning may require larger models. They also acknowledge that the exact nature of the planning circuits in models other than Gemma2 9B remains an open question. The methodology is proposed as a foundation for further analysis of planning in other domains, such as instruction following and Chain of Thought reasoning.