🤖 AI

Tandem: Riding Together with Large and Small Language Models for Efficient Reasoning

The article introduces Tandem, a collaborative framework in which a large language model acts as a strategic coordinator to generate critical insights that guide a smaller, more efficient model in executing full reasoning, thereby reducing computational costs by approximately 40% while maintaining high performance in tasks such as mathematical reasoning and code generation.

Original authors: Zichuan Fu, Xian Wu, Guojing Li, Yejing Wang, Yijun Chen, Zihao Zhao, Yixuan Luo, Hanyu Yan, Yefeng Zheng, Xiangyu Zhao

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zichuan Fu, Xian Wu, Guojing Li, Yejing Wang, Yijun Chen, Zihao Zhao, Yixuan Luo, Hanyu Yan, Yefeng Zheng, Xiangyu Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Overthinker"

Imagine a brilliant, world-class professor (the Large Language Model or LLM) who can solve incredibly difficult math problems. This professor, however, has a habit: before giving you the answer, he writes a 5,000-word essay explaining every single thought, every dead end he considered, and every tiny calculation.

Although the answer is usually correct, this process is slow and expensive. It is like hiring a master architect to build a simple birdhouse, but this architect spends three days drawing blueprints, calculating wood grain properties, and writing a history of carpentry before driving in a single nail.

On the other hand, you have a fast, energetic apprentice (the Small Language Model or SLM). He is fast and cheap, but if you simply ask him the difficult question, he often guesses incorrectly or gets stuck.

The Solution: The "Tandem" Team

The authors propose a new way of working called Tandem. Instead of letting the professor do all the work alone or letting the apprentice guess blindly, they work together in a Mentor-Intern relationship.

Here is how the "Tandem" system works:

1. The Mentor provides a "Cheat Sheet" (not the whole book)

The large model (Mentor) does not write the whole essay. Instead, it quickly generates a compact "cheat sheet" with four key insights:

Goal: What are we actually trying to solve?
Planning: What is the overarching strategy?
Retrieval: What specific facts or formulas do we need?
Action: What are the first few logical steps?

Imagine this as the professor giving the apprentice a detailed map and a compass, rather than driving the car for him.

2. The Intern drives the car

The small model (Intern) takes this "cheat sheet" and uses it to do the heavy lifting. Since it has the map, it does not get lost. It drives the car (generates the reasoning steps) much faster and cheaper to the destination than the professor could have done alone.

3. The "Stop Sign" Mechanism (cost-aware assessment)

This is the cleverest part of the system. The system does not blindly follow a rule like "Let the professor always speak for 5 minutes."

Instead, the small model has an integrated confidence meter. While reading the Mentor's hints, it checks its own internal feelings:

"Do I understand this well enough to finish the work?"
"Am I confused or do I feel secure?"

If the small model feels secure (low "entropy" or uncertainty), it raises a stop sign. The Mentor stops speaking immediately, and the small model finishes the answer. If the small model is still confused, the Mentor adds just a little more guidance.

The Results: Faster, Cheaper, and Smarter

The paper tested this on difficult math problems and code generation tasks. Here is what they found:

The Sweet Spot: The team consisting of a "big brain" (32B model) and a "small brain" (7B model) working together solved problems better than the big brain alone.
The Savings: They achieved this higher accuracy with 40% less computing power (and money) at the same time.
The Magical Transfer: The "confidence meter" (the classifier that decides when to stop) was trained on math problems. Surprisingly, it worked just as well on coding problems without needing to be retrained. It is like a driver who learned to stop at red lights in New York and can immediately stop at red lights in Tokyo without needing a new lesson.

Why This Matters

The paper argues that we do not need to force our largest and most expensive AI models to perform every single reasoning step. By letting them act as strategic leaders while smaller, cheaper models handle the execution, we can get the best of both worlds: the deep thinking of a huge model with the speed and efficiency of a small one.

In short: Do not ask the CEO if he should file the papers. Ask the CEO to write the memo, and let the efficient assistant file the papers. The "Tandem" system automates this perfect division of labor.

1. Problem Statement

Recent advances in Large Language Models (LLMs) have introduced "thinking" paradigms (e.g., Chain-of-Thought, DeepSeek-R1), where models perform explicit, step-by-step reasoning before generating an answer. While this significantly improves accuracy and interpretability in complex tasks such as mathematical reasoning and code generation, it incurs massive computational overhead.

The Bottleneck: Reasoning chains in modern LLMs are often 5–10 times longer than standard outputs, leading to high inference latencies and operational costs (TFLOPs).
Limitations of Existing Solutions:
- Reinforcement Fine-tuning (RFT): Attempts to shorten reasoning traces require continuous training, which can degrade general capabilities and is impossible for closed-source models (accessible only via API).
- Standalone Small Language Models (SLMs): While efficient, they often lack the depth of reasoning required to solve complex problems correctly.
- Naive Collaboration: Simply passing full reasoning chains from an LLM to an SLM is inefficient and exceeds the SLM's comprehension capacity.

Core Question: How can we preserve the benefits of high-quality reasoning from LLMs while making them computationally efficient enough for practical deployment?

2. Methodology: The Tandem Framework

The authors propose Tandem, a collaborative framework that synergistically combines a Large Language Model (LLM) as a "Mentor" and a Small Language Model (SLM) as an "Intern." The system is designed to mimic human cognitive architecture (ACT-R) by decomposing reasoning into functional modules.

A. Mentor-Intern Architecture

The Mentor (LLM): Does not generate the full solution. Instead, it produces a compact set of structured reasoning insights to guide the SLM.
The Intern (SLM): Receives the insights and executes the detailed reasoning steps to generate the final answer.

B. Structured Reasoning Insights

Instead of raw reasoning chains, the LLM is prompted to output four specific components:

Goal: Defines the objective and constraints.
Planning: Outlines the overarching strategy and decomposition into sub-problems.
Retrieval: Retrieves relevant facts, formulas, or knowledge.
Action: Executes concrete logical steps or calculations.

C. Cost-Aware Continuous Evaluation (The Core Innovation)

To balance efficiency and reliability, Tandem does not use a fixed amount of guidance. Instead, it employs an adaptive mechanism:

Stepwise Generation: The LLM generates insights at increasing effort levels (Low, Medium, High).
Sufficiency Classification: After each step, the SLM processes the accumulated insights ( $Q \oplus I_t$ $Q \oplus I_{t}$ ). A dedicated Sufficiency Classifier ( $C$ $C$ ) analyzes the internal state of the SLM to determine whether the current guidance is sufficient to solve the problem.
- Features: The classifier uses statistical features derived from the SLM's token-level probability distribution, specifically Perplexity (PPL) and Entropy.
- Logic: Lower entropy and perplexity indicate that the SLM is confident and the guidance is sufficient.
Adaptive Termination:
- If the classifier predicts Sufficiency ( $y=1$ ): The LLM stops generating. The SLM proceeds to immediately generate the final answer.
- If Insufficient ( $y=0$ ): The LLM continues to the next, more detailed level.
- Fallback: If all levels are exhausted without a "sufficient" signal, the system selects the level with the highest sufficiency score to generate the answer.

3. Key Contributions

New Collaborative Paradigm: Introduction of a "Mentor-Intern" dynamic where the LLM provides high-level strategic guidance (Goal, Planning, Retrieval, Action) instead of full reasoning chains, allowing the SLM to take over execution.
Cost-Aware Evaluation Mechanism: Proposal of a dynamic stopping criterion based on a classifier trained on SLM uncertainty metrics (PPL/Entropy). This enables the system to adaptively allocate computational resources, stopping early for simple problems and deepening guidance for complex ones.
Cross-Domain Transferability: It is demonstrated that the sufficiency classifier trained on mathematical reasoning (MATH) can be effectively transferred to code generation (HumanEval) without retraining, proving that the features capture domain-independent confidence signals.

4. Experimental Results

Experiments were conducted on MATH (mathematical reasoning) and GSM8K (grade school math) as well as HumanEval (code generation).

Performance vs. Cost:
- Accuracy: The Tandem framework (32B LLM + 7B SLM) achieved an accuracy of 83.46% on MATH, outperforming the standalone 32B LLM (80.90%) and the standalone 7B SLM (77.14%).
- Efficiency: It reduced computational costs by approximately 40% compared to the standalone 32B LLM (using only 59% of the cost).
- Comparison: Outperformed baselines such as "Budget Forcing" (truncating reasoning) and "LLM Cascade" (static routing) in both accuracy and efficiency.
Generalization:
- Cross-Family: Effective collaboration was observed between different model families (e.g., DeepSeek LLM + Qwen SLM).
- API Compatibility: The framework works with closed-source API models (e.g., GPT-4o-mini) as mentors, as it requires only text output and no model weights.
- Scaling: Optimal performance occurs with a moderate capability gap between the LLM and SLM. If the gap is too large (e.g., 32B guiding 1.5B), the SLM cannot parse the guidance; if too small, gains are marginal.
Latency: In real-time tests, Tandem achieved a 1.8x speedup over the Mentor-only baseline (32B) while maintaining higher accuracy.

5. Significance and Impact

Democratization of Reasoning: Tandem makes high-level reasoning accessible and affordable for applications with strict budget constraints or real-time latency requirements.
Efficiency Without Retraining: Unlike methods requiring Reinforcement Fine-Tuning (RFT), Tandem works with pre-trained models and closed-source APIs, enabling immediate deployment.
Cognitive Alignment: By structuring reasoning into Goal, Planning, Retrieval, and Action, the framework aligns with human cognitive processes, pointing toward a promising direction for future agentic systems.
Practical Deployment: The ability to transfer the sufficiency classifier across domains without retraining (from math to code) significantly lowers the barrier to adopting this framework in diverse real-world applications.

In summary, Tandem demonstrates that lightweight, high-quality reasoning guidance from a large model can effectively replace full reasoning chains, enabling small models to achieve state-of-the-art performance at a fraction of the computational cost.