Technical Summary: ASTER (Agentic Scaling with Tool-integrated Extended Reasoning)

1. Problem Statement

Reinforcement Learning (RL) has become a dominant paradigm for eliciting long-horizon reasoning in Large Language Models (LLMs). However, scaling Tool-Integrated Reasoning (TIR) via RL faces a critical failure mode termed interaction collapse. In this pathological state, models fail to sustain multi-turn tool usage. Instead, they degenerate into extensive internal reasoning followed by trivial, post-hoc code verification, failing to leverage external tools for intermediate computation and verification.

Existing approaches to TIR initialization suffer from specific limitations:

• ZeroTIR (RL from scratch): Suffers from unstable training due to non-stationary feedback and distributional drift, often failing to instill new capabilities beyond the base model's representational limits.
• Supervised Fine-Tuning (SFT) Cold-Starts: Existing methods often introduce strong inductive biases that lead to interaction collapse.
  • Rewriting text-only CoT to code (e.g., ReTool): Preserves text-only structures, limiting genuine tool-driven planning.
  • Teacher-generated trajectories (e.g., DemyAgent): Often feature short, reactive reasoning that contradicts the deliberate "System-2" behavior required for complex tasks.

2. Methodology: The ASTER Framework

The authors introduce ASTER (Agentic Scaling with Tool-integrated Extended Reasoning), a two-stage framework designed to circumvent interaction collapse by prioritizing interaction density in the cold-start phase.

Core Hypothesis

The paper posits that the behavioral prior established during cold-start SFT dictates downstream RL outcomes. Specifically, the interaction density (the number of tool invocations per trajectory) is the key property. High-density, long-horizon trajectories induce a high-entropy behavioral prior that preserves exploration capacity during RL, preventing premature convergence to shallow policies.

Technical Pipeline

1. Data Construction:

   • Sources: Skywork-OR1-RL-Data and 93K problems from AoPS forums.
   • Synthesis: Tool-augmented solutions generated by GPT-OSS-20B-high.
   • Filtering: Only trajectories with 100% tool execution success and correct final answers are retained.
   • Key Curation: The authors construct a small, expert cold-start dataset of 4K trajectories, specifically selecting those with more than nine tool-interaction turns.

2. Stage 1: Targeted Cold-Start SFT:

   • The model is fine-tuned on the 4K interaction-dense trajectories.
   • Unlike traditional approaches that optimize for immediate post-SFT accuracy, this stage prioritizes establishing a behavioral prior for sustained multi-turn tool use, even if it results in lower initial accuracy compared to reasoning-centric initialization.

3. Stage 2: Multi-Stage RL Training:

   • Algorithm: Group Relative Policy Optimization (GRPO).
   • Curriculum: A two-stage process with increasing context lengths (18K tokens in Stage 1, 32K tokens in Stage 2).
   • Interaction Budget: The maximum number of tool invocations per trajectory is capped at 50 during training to encourage long-horizon planning.
   • Filtering: Before transitioning to Stage 2, prompts that consistently yield correct outcomes are filtered out to focus training on challenging instances.

3. Key Contributions & Findings

Research Question 1: The Role of Cold-Start SFT

The study demonstrates that cold-start SFT is not universally beneficial; its efficacy depends entirely on the interaction patterns encoded in the demonstrations.

• Sparse/Short-Horizon Priors: Datasets like ReTool and DemyAgent (dominated by 1–2 tool calls) induce a prior that biases learning toward shallow verification behaviors, often performing worse than ZeroTIR (RL from scratch).
• Dense/Long-Horizon Priors: The ASTER approach, using interaction-dense trajectories, induces a robust prior that enables sustained multi-turn tool use and superior downstream performance.

Research Question 2: Effect of Interaction Density

Stratifying the cold-start dataset by tool-call count reveals a clear ordering in performance:

• ≥9 Tool Calls (4K subset): Yields the highest final accuracy and maintains the highest training entropy throughout RL, indicating broader exploration.
• ≤1 Tool Call: Results in the poorest performance and rapid entropy collapse.
• Conclusion: Interaction-dense cold-starts are essential for shaping RL dynamics toward persistent tool use.

Research Question 3: Effect of RL Interaction Budget

There is a critical budget-alignment effect between training-time and inference-time interaction budgets.

• Models trained with generous interaction budgets (50 calls) achieve significantly higher peak performance when inference budgets are large (e.g., 256 calls).
• Conversely, models trained with constrained budgets perform better under tight inference constraints.
• Implication: Mismatched budgets degrade performance; alignment is necessary for optimal scaling.

4. Experimental Results

The authors evaluate ASTER-4B (based on Qwen3-4B) against state-of-the-art baselines on competitive mathematical benchmarks (AIME 2024/2025, HMMT 2025, BeyondAIME).

• Performance: ASTER-4B achieves 90.0% on AIME 2025 (with a 90K inference budget) and 85.0% (30K budget).
• Comparison: It outperforms significantly larger models, including DeepSeek-V3.2-Exp (671B parameters) and Qwen3-235B-A22B-Thinking.
• Scaling: The model exhibits strong test-time scaling, with performance improving from 85.0% to 90.0% as the inference budget increases from 30K to 90K.
• Efficiency: ASTER-4B surpasses the best agentic baseline (rStar2-Agent-14B) by significant margins (e.g., +20.2% on AIME 2025) despite being trained on a much smaller parameter count.

5. Significance and Claims

The paper claims that ASTER successfully mitigates interaction collapse in tool-integrated reasoning by redefining the cold-start strategy. The primary significance lies in the finding that cold-start SFT should prioritize behavioral diversity and exploration (via interaction density) over short-term accuracy metrics.

By establishing a high-entropy prior through a small set of expert, interaction-dense trajectories, ASTER enables compact models (4B) to rival or exceed the performance of giant baselines (200B+ parameters) in complex, long-horizon reasoning tasks. This suggests that agentic scaling can be achieved through better initialization and interaction design rather than solely relying on parameter scaling.

The authors emphasize that while these capabilities improve the efficiency of complex reasoning workflows (e.g., mathematical problem solving), they also introduce safety considerations regarding automated tool use, advocating for sandboxed execution and human oversight in deployment.