Technical Summary: Efficient Agentic Reasoning Through Self-Regulated Simulative Planning

Problem Statement

Current agentic Large Language Models (LLMs) predominantly operate as reactive policies with adaptive computation, relying on unconstrained chain-of-thought (CoT) reasoning where planning capabilities are expected to emerge implicitly from end-to-end training. The authors argue this approach suffers from a fundamental lack of control over the presence, structure, or horizon of planning. Consequently, these systems often exhibit dramatic increases in reasoning length (token consumption) during training without reliable gains in accuracy. Furthermore, the absence of an explicit planning structure prevents the analysis, regulation, or independent improvement of the planning process. Existing alternatives, such as effort-adaptive approaches or mode-routing, address only subsets of the problem (e.g., controlling reasoning amount or selecting execution modes at task onset) but fail to combine explicit simulative planning with autonomous, per-turn self-regulation.

Methodology: SR2AM

To address these inefficiencies, the paper proposes SR2AM (Self-Regulated Simulative Reasoning Agentic LLM), which decomposes agent deliberation into three interacting systems:

1. Reactive Execution (System I): Handles fine-grained reasoning and direct action execution.
2. Simulative Reasoning (System II): A "Simulative Planner" that grounds deliberation in future-state prediction using a world model. Unlike unconstrained CoT, this system proposes candidate actions, predicts their consequences (belief states), and selects sequences that maximize expected long-term progress. In SR2AM, the LLM itself serves as the world model in language space.
3. Self-Regulation (System III): A learned "Configurator" that autonomously decides when and how deeply to plan at each turn. It assesses the current belief state and determines whether to invoke a new plan, continue an existing one, or skip planning entirely to act directly.

Instantiations and Training

The authors explore two instantiations of this framework, both trained via Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL):

• SR2AM-v0.1 (8B): Demonstrates feasibility by recording decisions from a multi-module prompted system (using o4-mini as a teacher). The configurator and planner are implemented as separate prompted LLMs with additional modules for belief formation and reflection.
• SR2AM-v1.0 (30B): A scalable approach that reconstructs structured plans from the traces of a pretrained reasoning LLM (DeepSeek-V3.2). An annotator LLM infers configurator decisions and simulative plan content (state-action-future-state sequences) from existing CoT traces, preserving the original free-form reasoning while augmenting it with structured planning.

The models are trained on interactive reasoning tasks involving math, science, tabular data analysis, and web information seeking, utilizing tools such as web search, web browsing, and a Python sandbox. The RL objective (based on Group Relative Policy Optimization) prioritizes answer correctness while providing gradient signals for structural compliance.

Key Contributions

1. Architectural Decomposition: The paper formalizes a three-system decomposition (System I+II+III) that separates self-regulation, simulative planning, and reactive execution. This allows for explicit control over planning horizons and structures without per-domain engineering.
2. Efficient Agentic Reasoning: By decoupling the decision to plan from the act of planning, SR2AM avoids the inefficiency of unregulated deliberation (over-thinking) and the rigidity of always-on planning.
3. Empirical Validation of Self-Regulation: The study demonstrates that a learned configurator can effectively modulate planning depth, leading to significant token savings without sacrificing accuracy.
4. RL Dynamics: The paper analyzes how RL refines the interaction between systems, showing that improvements come from increasing planning quality (horizon) rather than planning frequency.

Results

The authors evaluated SR2AM across 11 benchmarks (math, science, tabular, web) and compared it against reference systems, unregulated deliberation models, and partially-regulated baselines.

• Performance vs. Scale:
  • SR2AM-v0.1-8B achieved an overall Pass@1 of 57.0, outperforming other 7–8B models and competing with unregulated agentic LLMs at 30–32B and pretrained LLMs with tools at 120–355B.
  • SR2AM-v1.0-30B achieved an overall Pass@1 of 71.3, competitive with DeepSeek-V3.2 (685B) and Kimi-K2.5 (1.0T) in the same tool harness, and exceeding GPT-5.4-xhigh as a text-only reasoning model.
• Reasoning Efficiency:
  • SR2AM-v1.0-30B consumed 25.8–95.3% fewer reasoning tokens than competitive agentic LLMs of similar scale (30–32B).
  • Compared to MiroThinker-v1.5-30B, SR2AM-v1.0-30B achieved similar accuracy while using 51.2% fewer tokens (5,518 vs. 11,295).
• RL Impact: Analysis revealed that RL increased the average planning horizon by 22.8% while planning frequency grew by only 2.0%. This indicates the model learns to plan further ahead when necessary, rather than planning more often.
• Ablation Studies: Removing any of the three systems (free-form reasoning, simulative planning, or selective planning) degraded performance or efficiency, confirming their complementary roles. Specifically, removing selective planning (System III) caused the largest increase in token consumption.

Significance and Claims

The paper claims that efficient agentic reasoning benefits from decomposing deliberation into simulative planning governed by learned self-regulation, operating alongside reactive execution. The authors argue that the need for ever-longer reasoning traces in current models is a consequence of building agents as undifferentiated reactive policies lacking internal agency over their own planning.

The significance of this work lies in:

1. General-Purpose Planning: The simulative planning framework achieves strong performance across diverse task categories without per-domain procedures, supporting the view of simulative reasoning as a general mechanism.
2. Autonomous Regulation: The configurator instantiates a principle of autonomous regulation of reasoning processes. The authors posit this principle may extend beyond inference-time planning to how agents govern their own learning and adaptation.
3. Scalability: The v1.0 instantiation demonstrates that structured planning and self-regulation can be reconstructed from existing reasoning traces, offering a scalable path to training efficient agentic models.

The authors remain modest regarding limitations, noting that their world model is currently limited to language space and that the configurator's calibration of when not to plan (stopping criteria) remains an area for improvement, as evidenced by occasional over-verification in specific tasks. They suggest future work should explore embodied settings, multimodal world models, and the extension of self-regulation to multiple interacting processes.