Technical Summary: ATLAS: Agentic Test-time Learning-to-Allocate Scaling

Problem Statement

Scaling inference-time compute has emerged as a primary mechanism for enhancing the reasoning capabilities of Large Language Models (LLMs). However, the orchestration of this compute remains largely "designer-engineered." Existing approaches rely on fixed sample budgets, static refinement loops, predetermined scoring rules, or rigid search policies (e.g., Best-of-N, Self-Refine, Tree of Thoughts) to decide how compute is allocated. In these frameworks, the model is tasked with solving the problem, but a human-defined rule or external controller dictates when to stop, how many samples to generate, and how to aggregate results. This separation leaves the model without agency over the allocation process, potentially leading to suboptimal compute usage, poor coverage, or reward hacking. The central challenge addressed is shifting the control of test-time scaling from external, static rules to the model itself, allowing it to dynamically decide how much compute to spend and when to terminate based on the specific problem context.

Methodology

The paper introduces ATLAS (Agentic Test-time Learning-to-Allocate Scaling), a framework where an LLM orchestrator owns the control loop end-to-end. The system operates through a sequential, stateful process:

1. The Orchestrator Loop: An LLM orchestrator manages a growing pool of candidate solutions. It does not solve the problem directly. Instead, it observes the original problem and the accumulated history of candidate solutions (the "trajectory").
2. The explore Action: The orchestrator's primary tool is a single action called explore. When invoked, this action dispatches a fresh, independent solver to the original problem. Crucially, the solver receives only the problem statement, ensuring that each candidate is conditionally independent. The solver returns a structured observation containing the answer, reasoning, approach, and confidence.
3. The stop Action: Based on its assessment of the accumulated evidence, the orchestrator can choose to stop. Upon stopping, it synthesizes the final answer directly from the entire history of candidates and reasoning traces. This "stateful evidence management" allows the orchestrator to make synthesis decisions based on the full context of the search process, rather than just a final pool of answers.
4. Extensible Action Space: While the minimal instantiation uses a parameterless explore[] (deciding only when to stop), the framework supports an extensible action space. Variants include:
   • ATLAS-MM: Allows the orchestrator to select which solver model to dispatch (e.g., choosing between different capability tiers like Haiku, Sonnet, or Opus).
   • ATLAS-MI: Allows the orchestrator to pass specific instructions to the solver to focus on particular sub-steps or reasoning strategies.
5. Explore Effort: The framework includes a hyperparameter, "explore effort," which sets the orchestrator's stopping threshold (Low, Medium, High). This determines how much evidence is required before the orchestrator deems the problem "solved," balancing cost against expected information gain.

The decision logic is encoded in the orchestrator's prompt as declarative principles (e.g., "genuine convergence means independent solvers arriving at the same answer through different methods") rather than prescriptive rules, allowing the model to adapt its strategy to the specific problem.

Key Contributions

1. Agentic Orchestration: The paper introduces ATLAS, a framework where the LLM itself decides the compute allocation and stopping criteria in context, replacing hand-specified loops and scoring functions.
2. Stateful Evidence Management: The authors demonstrate that having the orchestrator synthesize the final answer directly from the full trajectory (Thought/Action/Observation history) is a key driver of performance. Ablation studies show that replacing this direct synthesis with a separate integrator degrades or fails to improve accuracy on three of four benchmarks.
3. Scalable Action Space: The framework supports extending the action space to include solver selection and prompting strategies. The multi-model extension (ATLAS-MM) shows that the orchestrator's value scales with the richness of its decision space.
4. Benchmark-Agnostic Transfer: The orchestrator's prompt is designed to be benchmark-agnostic, allowing the same controller to transfer across text-only and image-conditioned tasks without retraining.

Experimental Results

The authors evaluated ATLAS under a Claude Sonnet 4.6 backbone across four diverse benchmarks: HLE-Verified (scientific QA), LiveCodeBench (code generation), GPQA-Diamond (graduate-level science), and BabyVision (multimodal reasoning).

• Performance: ATLAS achieved competitive accuracy while using significantly fewer API calls than fixed-workflow baselines (e.g., Self-Refine, Budget Forcing, Majority Voting).
  • HLE-Verified: 56.00% (vs. 53.00% for Self-Refine).
  • LiveCodeBench: 82.29% (matching Self-Refine no early stop, but with lower cost).
  • GPQA-Diamond: 85.86% (vs. 85.35% for Self-Refine no early stop).
  • BabyVision: 23.71% (vs. 21.39% for Self-Refine).
• Multi-Model Extension (ATLAS-MM): By adding solver choice to the action space, ATLAS-MM further improved performance:
  • HLE-Verified: 60.00%.
  • LiveCodeBench: 85.63%.
  • GPQA-Diamond: 88.38%.
  • BabyVision: 23.97%.
• Cost-Efficiency: ATLAS strictly dominates baselines on the cost-accuracy Pareto frontier, achieving higher mean accuracy at substantially lower per-question costs. The orchestrator dynamically adjusts compute, stopping early for easy problems and exploring further for difficult ones, whereas baselines often follow fixed budgets or stop prematurely.
• Ablations:
  • Orchestrator Capability: Accuracy on GPQA-Diamond increased with the capability tier of the orchestrator model (Haiku < Sonnet < Opus).
  • Separate Integrator: Adding a separate integrator model for final synthesis increased cost without improving accuracy, confirming the value of the orchestrator's direct synthesis.
  • Explore Effort: Higher effort levels led to non-decreasing accuracy across all benchmarks, with the cost rising most significantly on free-form benchmarks (HLE, LiveCodeBench, BabyVision) compared to the multiple-choice GPQA-Diamond.

Significance and Claims

The paper positions ATLAS as a shift in the design surface for test-time scaling. Rather than focusing on the controller logic (the "rules" of the game), the framework exposes the action space to the agent. The authors claim that any decision an agent should make in context—whether to stop, which model to use, or how to prompt—can be exposed as an argument to the explore action without requiring retraining.

The significance lies in demonstrating that an LLM orchestrator, when given the agency to manage its own compute budget through a stateful loop, can outperform rigid, designer-engineered scaling strategies. The results suggest that the "orchestrator" role is distinct from the "solver" role and that the value of the orchestrator scales with the richness of the decision space it is allowed to navigate. The paper concludes that stateful evidence management is a critical component of these gains, enabling the model to synthesize answers based on the full context of its search history rather than just the final set of candidates.