Technical Summary: Adaptive Latent Agentic Reasoning (ALAR)

1. Problem Statement

Large Reasoning Models (LRMs) have demonstrated that extended Chain-of-Thought (CoT) reasoning significantly improves performance on complex tasks. However, when applied to LLM agents operating in multi-turn interactive environments (e.g., search, tool use), this behavior introduces substantial inefficiency. Current agents often generate verbose textual reasoning at every decision step and allocate reasoning effort uniformly, regardless of the actual complexity of the turn.

This "overthinking" leads to:

• Token Inefficiency: Reasoning tokens accumulate in the context window across turns, increasing latency and cost.
• Structural Mismatch: Many agent turns only require lightweight internal computation to select the next action based on external observations, yet they still produce full explicit CoT traces.
• Limitations of Existing Solutions: Current token compression methods (e.g., pruning, length budgets) operate within the explicit CoT interface. They shorten the text but do not eliminate the need for textual reasoning tokens, failing to address the structural inefficiency of generating any text for routine decisions.

2. Methodology: Adaptive Latent Agentic Reasoning (ALAR)

The authors propose ALAR, a dual-mode framework that allows agents to use compact latent reasoning for routine turns and selectively escalate to explicit CoT only when deeper deliberation is required.

2.1 Core Architecture

• Dual-Mode Policy: At each turn $t$, the agent samples a mode $m_t \in \{<LAT>, <THINK>\}$ directly from its policy.
  • Latent Mode ($$): The agent generates a fixed-length sequence of $K$ continuous thoughts ($z_t$) in the hidden-state space instead of discrete tokens. These thoughts are processed autoregressively via a projection layer $f_\phi$ before the agent selects an action.
  • Explicit Mode ($$): The agent generates a standard variable-length textual CoT ($c_t$).
• Action-Oriented Reasoning: Unlike single-pass reasoning where latent thoughts lead to a final answer, ALAR's latent reasoning is designed to support intermediate action selection in an environment-coupled trajectory.

2.2 Training Pipeline

The framework employs a two-stage optimization procedure:

Stage 1: Action-Anchored Self-Distillation (AASD)

Training latent reasoning is challenging because continuous hidden states cannot be directly supervised by variable-length textual CoT. AASD addresses this by:

• Teacher-Student Setup: The base LRM acts as a teacher in explicit mode and a student in latent mode.
• Action Anchoring: Instead of aligning latent thoughts with textual CoT, the student is trained to reproduce the teacher's subsequent action. The teacher's CoT spans are replaced with latent blocks in the student's trajectory.
• Objective: The loss function maximizes the log-likelihood of the teacher's anchor actions ($a_t$) given the state ($s_t$) and the latent block ($z_t$). This allows the projector $f_\phi$ to discover the optimal trajectory in hidden-state space to produce the correct action without being constrained by the structure of the teacher's text.

Stage 2: Adaptive Reasoning GRPO (AR-GRPO)

To learn when to use latent versus explicit reasoning, the authors introduce a reinforcement learning method:

• Mode Warmup: A brief Supervised Fine-Tuning (SFT) initializes the policy to explore both modes, as AASD initially biases the model toward latent reasoning.
• Asymmetric Reward: The trajectory reward $R(\tau)$ encourages latent reasoning only when task success is preserved:
  • If the trajectory is correct ($EM=1$), the reward increases with the fraction of latent turns ($f(\tau)$).
  • If the trajectory is incorrect, the reward penalizes overuse of latent reasoning.
• Diversity and Length Control: A diversity bonus encourages exploration of different mode mixtures early in training, while a length-scaling factor penalizes excessively long explicit reasoning segments.

3. Key Contributions

1. ALAR Framework: A dual-mode reasoning architecture that combines latent reasoning with adaptive mode selection, enabling agents to bypass unnecessary textual reasoning while retaining explicit deliberation for complex steps.
2. Action-Anchored Self-Distillation (AASD): A novel self-distillation technique that trains latent agentic reasoning without direct latent-state supervision, using environment-facing actions as supervision anchors.
3. AR-GRPO: A reinforcement learning method that optimizes per-turn mode selection, rewarding latent reasoning efficiency when it does not compromise task success.

4. Experimental Results

The authors evaluated ALAR on agentic search (6 open-domain QA benchmarks) and tool-use (BFCL benchmark) domains, comparing against baselines like O1-Pruner, ThinkPrune, and ShorterBetter.

• Search Domain (Search-R1 3B/7B):
  • ALAR achieved comparable or slightly improved Exact Match (EM) accuracy while reducing generated tokens by 22.1% to 43.6% (Stage 2).
  • Stage 1 (pure latent) achieved up to 52.5% token reduction with competitive accuracy.
  • Text-based compression baselines showed limited gains (4–10% reduction) because the base models already produced relatively compact CoT.
• Tool-Use Domain (Qwen3-4B-Thinking):
  • ALAR demonstrated superior performance, improving average accuracy from 86.1% to 89.2% while reducing tokens by 84.6%.
  • This highlights that the base model was overly verbose, and ALAR successfully replaced unnecessary text with latent computation.
• Adaptive Behavior Analysis:
  • Task Difficulty: The model used explicit reasoning more frequently on harder benchmarks (e.g., Bamboogle, 2Wiki) and relied heavily on latent reasoning for easier datasets (e.g., NQ, TriviaQA).
  • Turn Position: Explicit reasoning was concentrated in the first turn (initial planning), while subsequent turns were predominantly latent.

5. Significance and Claims

The paper claims that ALAR improves the accuracy-efficiency trade-off of LLM agents by fundamentally changing the reasoning substrate rather than merely compressing text.

• Heterogeneity of Reasoning: The results support the hypothesis that agentic reasoning demands are heterogeneous; routine turns often require only sufficient internal computation (latent) to select an action, while only specific steps (e.g., initial planning or complex decomposition) require explicit CoT.
• Beyond Compression: By replacing the reasoning interface itself, ALAR achieves token reductions (up to 84.6% in tool use) that are unattainable by methods that merely shorten textual traces.
• Practicality: The approach demonstrates that LLM agents can be made significantly more efficient without sacrificing performance, provided they can dynamically allocate reasoning resources between latent and explicit modes.

6. Limitations

The authors acknowledge several limitations:

• Scope: The work focuses on agentic tasks involving external environments (search, tools) and does not evaluate single-pass domains like math or coding where additional reasoning tokens might directly correlate with final answer accuracy.
• Teacher Bias: The latent policy inherits the coverage and biases of the successful teacher trajectories used in AASD.
• Interpretability: Latent reasoning reduces the transparency of intermediate steps, which may be undesirable in high-stakes settings where action justification is required.
• Fixed Granularity: The current implementation uses a fixed latent block length ($K=4$) and a discrete mode choice, leaving finer-grained control for future work.