🤖 machine learning

Supplement Generation Training for Enhancing Agentic Task Performance

This paper proposes Supplement Generation Training (SGT), a cost-effective strategy that trains a smaller language model to generate dynamic supplemental text for large foundation models, thereby enhancing agentic task performance without the need for expensive post-training or model modification.

Original authors: Young Min Cho, Daniele Bonadiman, Divya Bhargavi, Tamer Alkhouli, Salvatore Romeo, Dongwei Jiang, Khushbu Pahwa, Yubin Ge, Etsuko Ishii, Monica Sunkara, Yi Zhang

Published 2026-04-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Young Min Cho, Daniele Bonadiman, Divya Bhargavi, Tamer Alkhouli, Salvatore Romeo, Dongwei Jiang, Khushbu Pahwa, Yubin Ge, Etsuko Ishii, Monica Sunkara, Yi Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Overworked Genius"

Imagine you have a brilliant, world-class expert (let's call them The Executive). This person knows everything, can solve complex math problems, write code, and answer deep philosophical questions. However, there are two big catches:

They are expensive: You can't hire them to work on every single tiny problem.
They are locked up: You can't teach them new tricks or retrain them for every new job because they are a "closed-source" system (like a black box).

Currently, if you want to use this Executive for a new task, you have to write them a perfect set of instructions (a prompt). But writing the perfect instruction is hard, and if the task changes, you have to start over.

The Solution: The "Super Assistant"

The authors of this paper propose a clever workaround. Instead of trying to retrain the expensive Executive, they train a small, cheap, and fast Assistant (a smaller AI model).

The Assistant's job isn't to solve the problem itself. Its job is to prepare the Executive for the job.

Think of it like this:

The Old Way: You hand a complex legal case directly to the Lawyer (The Executive) and say, "Solve this." The Lawyer might miss a detail because they didn't have the right context.
The New Way (SGT): You hand the case to your Legal Assistant first. The Assistant reads the file, realizes the Lawyer needs a specific summary of the evidence, a warning about common mistakes, or a step-by-step plan. The Assistant writes these notes down, attaches them to the file, and then hands the whole package to the Lawyer.

The Lawyer solves the case much better because they were given the perfect "cheat sheet" tailored specifically for that one case.

How Does the Assistant Learn? (The Training)

The paper introduces a method called Supplement Generation Training (SGT). Here is how they teach the Assistant:

The "Warm-Up" (SFT): First, they show the Assistant examples of different types of notes it could write (e.g., "Here is a summary," "Here is a list of mistakes to avoid," "Here is a step-by-step plan"). This teaches the Assistant the format of being helpful.
The "Trial and Error" (DPO): This is the magic part. The Assistant tries to write notes for a problem. The Executive tries to solve it.
- If the Executive solves it successfully, the Assistant gets a "Gold Star" (a reward).
- If the Executive fails, the Assistant gets a "Red Flag."
- Over time, the Assistant learns: "Oh, when the problem is about coding, writing a 'list of mistakes' works best. When it's about math, a 'step-by-step plan' works best."

The Assistant learns to dynamically choose the right type of note for the specific problem, rather than using the same note for everything.

The Results: Why It Matters

The researchers tested this on five different difficult tasks (like writing database queries, coding, and answering complex trivia).

The Result: By using this "Super Assistant" to prep the "Executive," the system got 21% better at solving problems compared to just asking the Executive directly.
The Efficiency: They trained a tiny model (1.7 billion parameters) to do this prep work. This is much cheaper and faster than trying to retrain the giant model (which might have hundreds of billions of parameters).

The "Search and Focus" Strategy

One of the coolest findings was how the Assistant learned.

At first: The Assistant tried everything. It wrote summaries, lists, plans, and rephrased questions randomly. It was like a chef tasting every spice.
Later: The Assistant "focused." It realized that for coding tasks, "Mistake Warnings" were the best spice. For logic puzzles, "Comparing Right vs. Wrong answers" was the best spice.

It stopped guessing and started being a specialized expert at preparing the Executive.

Summary

Supplement Generation Training (SGT) is a way to make big, expensive AI models smarter without changing them. You train a small, cheap AI to act as a "context coach." This coach reads the user's question, figures out exactly what extra help the big AI needs (like a summary, a warning, or a plan), writes that down, and passes it along. The big AI then uses that extra help to give a much better answer.

It's the difference between sending a general into battle with a vague map versus sending them with a detailed, up-to-date tactical briefing written by a specialized intelligence officer.

1. Problem Statement

The deployment of large foundation models (LFMs) for agentic tasks faces significant hurdles:

Computational Cost & Obsolescence: Fine-tuning massive models for every new task or domain is computationally expensive and time-consuming. Furthermore, as new foundation models are released rapidly, task-specific fine-tunes often become obsolete before their costs can be amortized.
Closed-Source Constraints: Many state-of-the-art models are available only via API, preventing gradient access and traditional fine-tuning.
Limitations of Current Prompt Optimization: Existing Automatic Prompt Optimization (APO) methods (e.g., DSPy, TextGrad, Local Prompt Optimization) primarily select, rearrange, or edit existing templates. They generally fail to synthesize new, instance-specific reasoning structures or content tailored to the specific input query.

The authors argue that instead of modifying the "Executive" (the large, frozen actor model), optimization should focus on the "Assistant" (a smaller model) that prepares the context. The goal is to generate dynamic, input-specific supplements that guide the frozen actor to perform better without weight updates.

2. Methodology: Supplement Generation Training (SGT)

The proposed framework, Supplement Generation Training (SGT), trains a small, open-source Language Model (the Supplement Generator, $\pi_S$ ) to produce textual supplements ( $s$ ) that are concatenated with the original input query ( $q$ ) before being passed to a frozen, large Actor Model ( $\pi_A$ ).

Core Components

Supplement Definition: A supplement is auxiliary text (e.g., reasoning steps, background info, error warnings) designed to enhance the actor's performance. The authors define eight initial types: Answer, Background, Chain-of-Thought (CoT), Rephrase, Summary, Mistakes, One-shot, and Pairs.
Proxy Reward Signal: Since the supplement generator does not solve the task directly, its quality is evaluated indirectly. The Actor Model ( $\pi_A$ ) generates an output $y$ based on $(q, s)$ . A reward model $R$ compares $y$ against the gold standard $y^*$ . If the task is solved successfully, the supplement $s$ receives a positive reward; otherwise, it is negative. This acts as a proxy reward for training $\pi_S$ .

Training Pipeline

The training process consists of two distinct phases:

Warm-Start Supervised Fine-Tuning (SFT):
- Goal: Teach the untrained model to generate supplements in the correct format and select from the predefined types.
- Process: The model is prompted with all 8 supplement types (plus a "Free Style" option) for each query. The resulting supplements are tested on the Actor. Only supplements that lead to successful Actor outcomes are paired with their queries to create a dataset for SFT.
- Outcome: A model ( $\pi_1$ ) capable of generating valid supplements.
Iterative Direct Preference Optimization (DPO):
- Goal: Refine the model to select the best supplement types (including discovering new ones) and improve the quality of generation.
- Process: An iterative loop where the current model generates a diverse set of supplements (Pre-defined, Out-of-Distribution, and Concatenated types). These are evaluated by the Actor to form Preference Pairs:
  - Cross-Type Pairs: Comparing different supplement types to teach type selection.
  - Within-Type Pairs: Comparing different generations of the same type to teach quality refinement.
- Strategy: The process employs a "Search-and-Focus" strategy. Early iterations explore diverse types (expanding the search space), while later iterations concentrate on the most effective types (refining the space).
- Loss Function: Combines DPO loss with Negative Log-Likelihood (NLL) loss to ensure stability.

3. Key Contributions

SGT Framework: A novel approach that decouples task-specific optimization from the foundation model by training a lightweight generator to create dynamic, instance-specific context.
Efficient Training Pipeline: A scalable procedure combining Warm-Start SFT and Iterative DPO, leveraging outcome-based rewards from a black-box actor without requiring gradient access to the actor.
Comprehensive Empirical Validation: Extensive evaluation across 5 diverse benchmarks (Spider, DS-1000, HotpotQA, HLE, superGPQA) and multiple actor families (Sonnet, GPT-OSS), demonstrating robust generalization.

4. Experimental Results

The authors evaluated SGT using a Qwen3-1.7B model as the supplement generator and v3.5-sonnet-v2 and GPT-OSS-120B as the frozen actors.

Performance Gains: SGT achieved an average improvement of 21% over the baseline actor (without supplements) across all benchmarks.
- Iteration 1 of DPO already outperformed all baselines (+10%).
- By Iteration 5, the gain reached +21%.
Comparison with Baselines:
- vs. Inference-Time Scaling (CoT): SGT (+21%) significantly outperformed standard Chain-of-Thought prompting (+5%).
- vs. APO Methods: SGT outperformed TextGrad and DSPy, which showed only marginal gains (1-4%).
- vs. SFT-Only: SFT-only training yielded a modest +5% gain, highlighting the critical role of the iterative DPO preference optimization.
Task-Specific Insights:
- Structured Tasks (Spider, DS-1000): Showed the largest gains. Supplements helped externalize intermediate reasoning and clarify constraints.
- Open-Ended Tasks (HotpotQA): Showed modest gains, as these rely more on factual recall than structural reasoning.
- Hard Tasks (HLE, superGPQA): Showed meaningful relative improvements, suggesting supplements help mitigate model uncertainty in ambiguous scenarios.

5. Qualitative Analysis & Significance

Distribution Shift: Qualitative analysis revealed a "Search-and-Focus" strategy. Initially, the model explored various supplement types. As training progressed, the distribution shifted heavily toward "Pairs" (contrasting correct/incorrect examples) for structured tasks, while adapting to task-specific needs (e.g., "Summary" for code tasks).
Significance:
- Cost-Effectiveness: SGT allows the use of small, fast-to-train models (1.7B) to enhance massive, expensive, closed-source models without retraining them.
- Sustainability: It offers a solution to the "obsolescence" problem of fine-tuning; the supplement generator can be quickly adapted to new tasks or new actor models.
- Interpretability: Unlike black-box fine-tuning, the supplements generated are human-readable, providing insight into why a model succeeded or failed (e.g., identifying specific mistakes or reasoning gaps).

In conclusion, the paper demonstrates that optimizing the input context via a learned, small-scale generator is a superior and more sustainable strategy for enhancing agentic AI performance compared to traditional fine-tuning or static prompt engineering.