Technical Summary: TMAS – Scaling Test-Time Compute via Multi-Agent Synergy

1. Problem Statement

Test-time scaling (TTS) has emerged as a paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs) by allocating additional computation during inference. While early approaches focused on extending single generation trajectories (e.g., Chain-of-Thought) or aggregating parallel independent solutions (e.g., Self-Consistency), recent structured methods attempt to organize reasoning across multiple trajectories and refinement rounds.

However, existing structured TTS methods face significant limitations in balancing exploration (trying diverse solution paths) and exploitation (refining promising directions):

• Weak Coordination: Trajectory-aggregation methods often rely on noisy historical information without explicitly filtering what to retain, leading to suboptimal signal reuse.
• Insufficient Collaboration: Verify-refine systems introduce explicit feedback but often leave different trajectories weakly coupled, failing to share reusable intermediate findings or local feedback across attempts.
• Memory Granularity: Current approaches fail to distinguish between low-level reliable intermediate conclusions (local signals) and high-level strategic insights (global patterns), limiting effective information sharing.

The core challenge is to design a framework that explicitly coordinates multi-agent collaboration, manages hierarchical memory to separate local experience from global strategy, and aligns the model to effectively utilize this structure without becoming trapped in suboptimal patterns.

2. Methodology: The TMAS Framework

The authors propose TMAS (Test-time compute via Multi-Agent Synergy), a framework that organizes inference as a collaborative process among specialized agents. The system operates iteratively, balancing parallel exploration with sequential exploitation.

2.1. Multi-Agent Architecture

TMAS employs five specialized agents to manage the inference pipeline:

1. Solution Agent ($A_{sol}$): Generates $N$ candidate solution trajectories in parallel. It uses an $\epsilon$-greedy strategy: with probability $1-\epsilon$, it exploits accumulated experience and previous rollouts; with probability $\epsilon$, it explores guided by high-level guidelines to avoid redundancy.
2. Verification Agent ($A_{ver}$): Performs $M$ independent verification passes on each candidate, providing analytical feedback and scalar scores (correctness, partial correctness, or errors).
3. Summary Agent ($A_{sum}$): Aggregates verification results into concise rollout-level summaries, highlighting validated steps and logical flaws.
4. Experience Agent ($A_{exp}$): Extracts low-level reusable signals from the current rollouts to update the Experience Bank. This includes verified intermediate conclusions, concrete skills, and specific error-avoidance heuristics.
5. Guideline Agent ($A_{guide}$): Abstracts high-level strategic approaches from parallel rollouts to update the Guideline Bank. This records global solution directions and structural insights to steer subsequent agents away from previously attempted (and failed) patterns.

2.2. Hierarchical Memory Management

TMAS introduces two distinct memory banks to facilitate structured information flow:

• Experience Bank: Stores concrete, trajectory-specific reasoning signals (e.g., "verified anchor $T(2)=3$"). These are reused to refine local reasoning and avoid repeating specific mistakes.
• Guideline Bank: Stores high-level strategic logs (e.g., "attempted induction with splitting argument"). These are used to enforce diversity, ensuring new rollouts do not simply repeat previously explored strategic frameworks.

2.3. Hybrid Reward Reinforcement Learning

To align the model with the TMAS collaborative process, the authors design a Hybrid Reward System based on GRPO (Group Relative Policy Optimization). Unlike standard RL which optimizes only for final correctness, this system includes three complementary objectives:

1. Standard Correctness Reward: Preserves basic reasoning capability (reward +1 for correct answers, -1 otherwise).
2. Experience Utilization Reward: Encourages the model to effectively use the Experience Bank. If a problem is difficult to solve without the bank but solvable with it, the model receives a bonus proportional to the difficulty of the problem without the bank. This prevents the bank from being treated as passive context.
3. Novel Strategy Exploration Reward: Encourages the discovery of new high-level solution strategies. Rollouts that follow a guideline not covered by the existing bank receive a higher reward, while those repeating known strategies are penalized, even if correct.

3. Key Contributions

1. TMAS Framework: A novel multi-agent system that explicitly organizes information flow across agents, trajectories, and iterations. It transforms independent reasoning attempts into a coordinated iterative process via specialized Experience and Guideline agents.
2. Hierarchical Memory Mechanism: The introduction of separate banks for low-level experience (reusable local signals) and high-level guidelines (strategic avoidance) to balance exploitation of reliable findings with exploration of diverse paths.
3. Hybrid Reward RL Scheme: A training objective tailored to TMAS that jointly optimizes for basic reasoning competence, effective experience utilization, and novel strategy exploration, addressing the exploration-exploitation trade-off in iterative reasoning.

4. Experimental Results

The authors evaluated TMAS on challenging reasoning benchmarks: IMO-AnswerBench-50 and HLE-Math-100, using Qwen3-30B and Qwen3-4B as base models.

• Iterative Scaling: TMAS demonstrated stronger iterative scaling than existing baselines (Self-Refine, Verify-Refine, PaCoRe, RSE). While other methods plateaued or degraded in later iterations, TMAS continued to improve, achieving the best late-stage performance (e.g., 40.50% on IMO-AnswerBench-50 with Qwen3-30B at iteration 19).
• Impact of Hybrid RL: Training with the hybrid reward system significantly amplified scaling effectiveness. On the Qwen3-4B model, TMAS+Hybrid-RL not only achieved higher peak accuracy but also mitigated the performance degradation seen in Vanilla-RL (training with only correctness rewards) during later iterations.
• Model Efficiency: The hybrid RL training substantially narrowed the performance gap between the 4B and 30B models. On IMO-AnswerBench-50, the gap reduced from 23.44 to 9.62 points, suggesting that effective iterative test-time computation can allow smaller models to approach the performance of much larger ones.
• Ablation Studies:
  • Removing either the Experience or Guideline module degraded performance, with joint removal causing the most severe drop.
  • An intermediate exploration coefficient ($\epsilon=0.2$) yielded optimal results, confirming the need for a balance between discovery and exploitation.
  • Excessive verification counts (e.g., 16) or parallel solutions (e.g., 12) did not yield monotonic improvements, indicating diminishing returns and potential noise introduction.

5. Significance and Limitations

The paper claims that TMAS represents a significant step forward in organizing test-time compute by moving beyond simple trajectory aggregation or verification. By explicitly structuring the flow of information and separating memory into hierarchical levels, TMAS enables more effective collaboration among reasoning trajectories. The hybrid reward scheme is presented as a crucial enabler for training models to leverage this complex structure.

Limitations:

• Verification Bottleneck: The authors identify a "shared capability boundary" where, on the hardest problems, the verification agent (instantiated from the same model family) may struggle to distinguish between plausible but flawed solutions and correct ones. This can lead to unreliable feedback signals for the refinement loop.
• Computational Constraints: The evaluation was not conducted on frontier models (e.g., GPT-5.5) due to cost and API constraints.
• Data Construction: The current RL pipeline requires an external teacher model to pre-construct cold-start trajectories and memory-based training data. The authors suggest future work could dynamically incorporate trajectories from previous iterations to expand the data source.

In conclusion, TMAS offers a structured approach to scaling test-time compute that effectively balances exploration and exploitation through multi-agent synergy and hierarchical memory, demonstrating that additional computation can be translated into improved reasoning performance when organized correctly.