💬 NLP

Team of Thoughts: Efficient Test-time Scaling of Agentic Systems through Orchestrated Tool Calling

The paper introduces Team-of-Thoughts, a heterogeneous multi-agent framework that leverages orchestrator calibration and agent self-assessment to dynamically select specialized models as tools, significantly outperforming existing homogeneous systems in mathematical reasoning and code generation benchmarks.

Original authors: Jeffrey T. H. Wong, Zixi Zhang, Junyi Liu, Yiren Zhao

Published 2026-03-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jeffrey T. H. Wong, Zixi Zhang, Junyi Liu, Yiren Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a incredibly difficult puzzle, like a complex math problem or writing a piece of software that has never been built before.

In the past, you might have hired one genius to solve it. You'd give them the puzzle, and they would sit there, thinking hard, step-by-step, until they found the answer. This is how most AI systems worked: one big brain doing all the work.

But sometimes, that one genius gets stuck. Maybe they are great at math but terrible at coding. Or maybe they just take a wrong turn in their thinking and can't find their way back.

"Team-of-Thoughts" is a new way of working that changes the game. Instead of hiring one genius, it hires a specialized team of experts and puts them in a room with a Project Manager.

Here is how it works, broken down into simple parts:

1. The Problem with "One-Size-Fits-All"

Most current AI teams are like a group of clones. They all use the same brain (the same AI model) but pretend to be different people (e.g., "Act like a doctor," "Act like a lawyer").

The Flaw: If the "clone" brain is bad at math, every single member of the team is bad at math. They are all stuck in the same mental rut.

2. The Solution: A "Toolbox" of Different Brains

The Team-of-Thoughts framework realizes that different AI models have different superpowers.

Model A might be a math wizard but bad at writing code.
Model B might be a coding ninja but slow at logic puzzles.
Model C might be great at creative writing but terrible at numbers.

Instead of forcing them to pretend to be the same person, this system treats them like specialized tools in a toolbox. You don't use a hammer to screw in a bolt; you pick the right tool for the job.

3. The Two Secret Ingredients

To make this team work, the researchers added two smart features:

A. The "Manager" (The Orchestrator)

Imagine a project manager who doesn't do the work themselves but knows exactly who is good at what.

Before the team starts, this manager runs a quick test to see who is the best "boss" for the job.
Sometimes the biggest, most expensive AI isn't the best manager. A slightly smaller, faster AI might be better at coordinating the team.
Once the manager is picked, they act as the conductor of an orchestra. They don't play every instrument; they just tell the right musicians when to start playing.

B. The "Resume" (Self-Assessment)

Before the team gets to work, every expert agent writes a short resume about themselves.

Agent A says: "I'm great at solving geometry problems, but I often mess up algebra."
Agent B says: "I'm a coding expert, but I struggle with word problems."
The Manager reads these resumes. When a new question comes in, the Manager looks at the "resume" and instantly picks the best person for that specific task. They don't waste time asking the coding expert to do math.

4. How They Work Together (The Magic)

When a hard question arrives:

The Manager reads the question and checks the "resumes."
The Manager calls only the 2 or 3 experts who are perfect for that specific problem. (They don't call the whole team; that would be too noisy and slow).
The Experts solve their part of the problem quickly and send back just the answer (not the messy scratchpad notes).
The Manager takes those answers, combines them, and gives you the final, high-quality solution.

Why is this better?

Speed & Cost: It's cheaper because it doesn't waste money asking the wrong experts to work. It's faster because it skips the "thinking" steps that lead to dead ends.
Accuracy: It covers more ground. If the math expert gets stuck, the coding expert might see a different angle. The Manager combines these views to find the truth.
Real-World Results: In tests, this team beat the "single genius" approach by a huge margin. On tough math competitions, they got 96% right, compared to 80% for the single genius. On coding tests, they jumped from 66% to 78%.

The Catch (The Limitation)

The system works best when the team is small and focused. If you try to put 100 experts in the room and ask the Manager to pick the right ones, the Manager gets overwhelmed (like a manager with too many emails). They start making mistakes. So, the secret is quality over quantity: pick the right few, not the many.

In a Nutshell

Team-of-Thoughts is like moving from a "Lone Wolf" strategy to a "Special Forces" strategy. Instead of one soldier trying to do everything, you have a Commander who knows exactly which specialist (sniper, medic, engineer) to send for the mission, ensuring the job gets done faster, cheaper, and with much higher success.

1. Problem Statement

Current approaches to Test-time Scaling (TTS)—which aims to improve Large Language Model (LLM) performance by allocating more computational resources during inference—face two primary limitations:

Homogeneity: Most Multi-Agent Systems (MAS) rely on a single model family with varying "personas" (e.g., prompting one model to act as a critic and another as a solver). Because these agents share identical parameters ( $\theta_1 = \theta_2 = \dots = \theta_n$ ), they lack the distributional diversity required to explore complementary regions of the solution space. They are prone to the same inherent biases and local optima.
Static Workflows: Existing frameworks often use fixed role assignments or sequential reasoning chains (like Chain-of-Thought) that are computationally inefficient. They fail to dynamically leverage the unique, specialized expertise inherent in different, heterogeneous model architectures (e.g., a model specialized in math vs. one specialized in code).

The paper argues that to truly scale capabilities at test-time without retraining, systems must exploit heterogeneous model priors through a dynamic, orchestrated approach rather than static, homogeneous ensembles.

2. Methodology: Team-of-Thoughts

The authors propose Team-of-Thoughts, a heterogeneous MAS framework that treats diverse LLMs as specialized "tools" managed by a central Orchestrator. The system operates via a two-phase pipeline:

A. Pre-Inference Pipeline

Orchestrator Calibration:
- Not all models are equally good at coordination. The system evaluates candidate models to identify the best "manager" (orchestrator) for specific task domains (e.g., math vs. coding).
- The calibration score is based on the model's ability to aggregate tool-agent responses accurately within a fixed budget.
- Finding: The best orchestrator is not necessarily the largest model; it is the one with superior "integrative reasoning" capabilities for the specific domain.
Agent Self-Assessment (Profiling):
- Instead of the orchestrator guessing an agent's capability, each tool agent performs a self-audit on a validation set.
- Agents generate a natural language "capability profile" detailing their strengths, weaknesses, and reliability for specific task categories (e.g., "excels at number theory but struggles with combinatorics").
- This decouples profiling from the orchestrator's internal biases, providing an objective view of agent expertise.

B. Inference Phase

Dynamic Selection: Upon receiving a query, the Orchestrator consults the pre-computed capability profiles. It dynamically selects the subset of agents ( $k$ ) most compatible with the task.
Tool Calling: The orchestrator invokes the selected agents in parallel.
Response Aggregation: Agents return only their final solutions (not full reasoning traces). The orchestrator synthesizes these discrete outputs into a high-confidence final answer.
- Key Design Choice: The paper finds that including full reasoning traces from agents introduces "contextual noise" and degrades performance; filtering to final answers yields better aggregation.

C. Probabilistic Motivation

The framework is modeled as a Gaussian Mixture Model (GMM).

A single agent's reasoning is a heuristic "drift" of a single distribution, often getting stuck in local optima.
Team-of-Thoughts initializes the system with multiple "exploratory heads" (heterogeneous agents) covering different parts of the solution space.
The orchestrator effectively "jumps" to high-probability regions localized by specialized tools and synthesizes them, rather than relying on a linear, single-model reasoning chain.

3. Key Contributions

Heterogeneous MAS Architecture: A novel framework that treats diverse LLMs as dynamic tools rather than static personas, maximizing capability coverage.
Orchestration Calibration & Self-Assessment: Mechanisms to identify the optimal coordinator and enable agents to self-profile their domain expertise, allowing for precise, priority-based agent selection.
Efficient Scaling Strategy: Demonstrates that strategic parallel invocation of a small, calibrated subset of agents ( $k=2$ ) outperforms exhaustive reasoning or large, uncalibrated ensembles.

4. Experimental Results

The framework was evaluated across five benchmarks (AIME2024, AIME2025, Humaneval+, MBPP+, LiveCodeBench) using seven diverse model families (including Claude Sonnet 4.5, GPT-5 Mini, DeepSeek v3.2, etc.).

Performance Gains:
- AIME2024: Achieved 96.00% accuracy (vs. 80.00% for homogeneous baselines like AgentVerse).
- LiveCodeBench: Achieved 77.91% accuracy (vs. 65.93% for baselines).
- MBPP+: Achieved state-of-the-art results across all pass@k metrics.
Cost Efficiency: The framework achieves higher accuracy at lower costs compared to single-model baselines and homogeneous MAS, as shown in the Pareto frontier analysis (Figure 1).
Selection Strategy: The Agent Self-Assessment strategy outperformed both random invocation and orchestrator-led assessment, proving the value of objective, agent-generated profiles.
Scaling Dynamics:
- Optimal Activation: Performance peaks when activating 2 agents per call. Increasing activation to 8 or 16 agents causes performance degradation due to "selection logic overload" and aggregation noise.
- Ensemble Size: While the system can handle large ensembles (up to 100 tools), the orchestrator's ability to select the right tools diminishes as the pool grows beyond 50, highlighting a bottleneck in centralized selection logic.

5. Significance and Conclusion

Team-of-Thoughts represents a paradigm shift in test-time scaling. It moves away from the "bigger is better" or "more agents is better" mentality, focusing instead on strategic heterogeneity.

Efficiency: By replacing sequential, token-heavy reasoning with coordinated, parallel tool calls, it achieves superior token efficiency.
Robustness: It overcomes the limitations of single-model biases by synthesizing diverse expert perspectives.
Future Implications: The work suggests that the future of agentic systems lies in orchestrated collaboration between specialized, heterogeneous models. However, it also identifies a current limitation: the centralized orchestrator becomes a bottleneck as the tool pool scales, pointing to a need for better long-context reasoning and hierarchical aggregation in future models.

In summary, the paper demonstrates that efficient test-time scaling is best achieved not by brute-forcing computation with a single model, but by intelligently orchestrating a team of specialized, diverse models.