💻 computer science

Teacher-Free Self-Training Amplifies but Does Not Compound: A Pass@ $K$ Crossover on a Free-Verifier Domain

This paper demonstrates that teacher-free self-training on a verified DSL domain amplifies a model's ability to express existing capabilities by concentrating probability mass, but does not compound its underlying reach, as evidenced by the base model eventually outperforming the trained model at higher sampling budgets.

Original authors: Igor Lima Strozzi

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Igor Lima Strozzi

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Does Practice Make Perfect, or Just Better at Showing Off?

Imagine you have a student (an AI model) who is trying to learn a new skill, like solving logic puzzles. The student is allowed to practice by solving puzzles, checking their own answers with a perfect answer key, and then studying the ones they got right to get better.

The big question this paper asks is: When the student practices this way, are they actually learning new things they couldn't do before (compounding), or are they just getting better at finding the solutions they were already capable of but rarely found (amplification)?

Many people hope for the first option (learning new superpowers). This paper argues that, in this specific setup, the student is only doing the second option (getting better at showing off existing skills).

The Setup: The "Trapdoor" Game

To test this fairly, the researchers built a special game called a "Trapdoor Domain." Think of it like a magic box with three rules:

The Generator (The Student): A small AI that tries to write a program to solve a puzzle.
The Verifier (The Perfect Answer Key): A simple computer program that checks if the answer is 100% correct. It's not an AI; it's just a strict rule-checker. It costs nothing to run and never makes mistakes.
The Critic (The Coach): A small AI that guesses which of the student's answers is most likely to work on new puzzles, not just the ones it just saw.

Why is this special? Usually, when AI teaches itself, it relies on a "teacher" (a bigger, smarter AI) to grade its work. Here, there is no teacher. The "Answer Key" is just a math rule. This means if the student learns something new, it must be because the student improved, not because the teacher taught it.

The Experiment: Three Rounds of Practice

The researchers let the student practice in rounds:

Round 1: The student tries to solve puzzles. The Answer Key checks them. The student studies the correct ones.
Round 2: The student tries again, using what it learned.
Round 3: One more time.

They measured two things:

How often the student got it right on the first try (Pass@1).
How often the student got it right if they were allowed to try 64 times (Pass@64).

The Findings: Amplification, Not Compounding

Here is what they discovered, using a few metaphors:

1. The "Coach" Helps More Than the "Answer Key"

When the student generated 8 possible answers, the "Answer Key" could only say "Yes, this works for the examples I see" or "No." It couldn't tell which answer would work on new puzzles.
The "Coach" (the Critic) was much better at picking the right answer. It improved the student's success rate by about 9% compared to just picking randomly.

The Catch: This improvement only happened on puzzles where the student was confused (where different answers looked okay). The Coach didn't create new skills; it just helped the student pick the best existing option.

2. The Ceiling Gets Higher, But Slower

As the student practiced, their performance got better.

Round 1 to 2: Big jump in performance.
Round 2 to 3: Smaller jump.
The Pattern: The gains slowed down every time. This is called Amplification. The student was digging deeper into a mine they already knew existed, finding the gold that was already there but hard to reach. They weren't discovering a new mine.

3. The "Overtaking" Test (The Smoking Gun)

This is the most important part. The researchers compared the "Trained Student" against the "Original Student" (before any practice).

At low effort (trying 8 times): The Trained Student wins. They are sharper and more consistent.
At high effort (trying 64 times): The Original Student wins.

The Metaphor: Imagine the Original Student has a wide, shallow net. They might miss a fish on the first try, but if they cast the net 64 times, they catch almost everything because their net is wide.
The Trained Student has a narrow, deep net. They are very good at catching the fish that are easy to find, so they win when they only get to cast once or twice. But because they focused so much on those easy fish, they forgot how to cast widely. When they try 64 times, they actually catch fewer total fish than the Original Student because their "net" has shrunk.

The Conclusion: Sharpening, Not Growing

The paper concludes that this self-training method amplifies capability but does not compound it.

Amplification: The model gets better at finding the solutions it was already capable of finding if it tried hard enough. It concentrates its energy on the "easy" wins.
No Compounding: The model did not break through a barrier to solve problems it was fundamentally incapable of solving before. It didn't expand its reach; it just tightened its focus.

A Warning About "Zero" Scores

The paper also points out a common mistake in AI research. Sometimes, a model gets 0% on a hard test, and after training, it gets 10%. People say, "Look! It learned something new!"
The researchers say: Wait. In this specific type of puzzle, getting 0% often just means you didn't try enough times (undersampling). If you tried 64 times, the "untrained" model could actually solve those "impossible" puzzles too. So, climbing off zero isn't always a sign of a new superpower; sometimes it's just a sign of better luck or more attempts.

Summary

The AI didn't learn to fly; it just learned to jump higher than before. It became a specialist at finding the solutions that were already within its reach, but it didn't gain the ability to reach places it couldn't reach before.

Technical Summary: Teacher-Free Self-Training Amplifies but Does Not Compound

1. Problem Statement

The central question addressed is whether self-improvement loops for language models (LLMs) that train on their own verified outputs compound capability (acquiring genuinely new reachability beyond the base model's distribution) or merely amplify capability (surfacing rare, reachable solutions the base model could already generate given sufficient sampling).

Existing literature is divided:

Elicitation view: Training on verified outputs merely sharpens the model's ability to access latent capabilities, often resulting in the base model outperforming the trained model at large sampling budgets (pass@K) [11].
Compounding view: Prolonged reinforcement learning (RL) can lift models past their base distribution, solving instances the base cannot reach at any budget [8].

A critical methodological gap exists: most studies rely on "teachers" (larger models, reward models, or humans) that cap the ceiling of novelty by construction, or they declare "emergence" based on metrics moving off apparent zeros, which may be sampling artifacts rather than true capability frontiers.

2. Methodology: The "Trapdoor" Constellation

To make the distinction between amplification and compounding decidable, the authors construct a teacher-free "constellation" on a specific domain:

Domain: A FlashFill-style "trapdoor" Domain Specific Language (DSL) for string transformations.
- Forward generation: Cheap to synthesize verified (problem, solution) pairs.
- Backward solving: Hard for the model to invert.
- Verification: A free, total, and exact interpreter (no learned parameters) that checks correctness exactly.
System Roles:
1. Verifier: The exact interpreter. It filters candidates consistent with visible examples but cannot adjudicate generalization on held-out inputs.
2. Generator: A 4-bit Qwen3-4B model with a LoRA adapter, mapping visible examples to candidate programs.
3. Critic: A separate 4-bit LoRA adapter trained as a generative yes/no classifier to rank candidates based on their likelihood of generalizing to held-out inputs.
4. Conductor: Orchestrates the loop without a model.
Training Loop (STaR):
- The generator proposes $K$ candidates.
- The verifier filters out those failing visible examples.
- The critic ranks survivors; the top candidate is selected.
- The verifier adjudicates the selected candidate on held-out examples.
- Only model-generated, verifier-confirmed generalizing programs are used as supervised fine-tuning (SFT) targets for the next round.
- Crucial Constraint: No larger model or external teacher is involved; the ceiling is not capped by a teacher's competence.

Experimental Setup:

Hardware: Single 24 GB GPU (RTX 3090).
Model: Qwen3-4B-Instruct (4-bit quantization).
Banks: Two frozen evaluation banks (Easy: max 3 atoms; Hard: max 4 atoms, out-of-training atom count).
Trajectory: Three rounds (v0 $\to$ v1 $\to$ v2) of self-training.

3. Key Contributions

A Fully Teacher-Free Constellation: A generator/critic/verifier system runnable on a single GPU, where the verifier is an exact interpreter, ensuring no "distillation escape hatch" limits novelty.
Critic-Guided Selection Superiority: Demonstrated that a learned critic beats a verifier-filtered "best-of-k" (naive) baseline by +9.1 percentage points (across 6/6 seeds). The gain is localized entirely to tasks where candidates disagree on held-out inputs (the "divergent subset"), confirming the critic recovers existing capability rather than creating new ones.
Headroom-Paced Amplification: Self-training raises the performance ceiling but never accelerates. The per-round gain decelerates as the model approaches the reachable frontier of the base distribution.
The Pass@K Crossover: A measured crossover where the trained model wins at the operating budget (pass@8) but the base model overtakes the trained model at large budgets (pass@64) on every trajectory. This confirms the model concentrates probability mass on fewer solutions rather than expanding its reach.
Methodological Correction on "Emergence": In this compositional DSL, apparent "zero-capability" cells at small sample counts are undersampling artifacts, not genuine frontiers. Declaring emergence based on climbing off zero is invalid without population-scale verification.

4. Results

Selection Performance

Critic vs. Naive: The critic improves selection accuracy significantly, particularly on the divergent subset (where candidates disagree on held-out data).
- Seed 1234: Naive 53.3% $\to$ Critic 93.3%.
- Seed 7: Naive 61.9% $\to$ Critic 92.9%.
Consistency: The total improvement equals the improvement on the divergent subset exactly, proving the critic only acts where structural ambiguity exists.

Self-Training Trajectory (STaR)

Ceiling Rise: The verifier ceiling (pass@8) increases from round 0 to round 2 on both banks.
- Easy Bank: 46.7% $\to$ 53.1% $\to$ 55.4%.
- Hard Bank: 37.9% $\to$ 41.1% $\to$ 45.0%.
Deceleration: The gain per round is non-accelerating.
- Easy: +6.4% (v0 $\to$ v1) $\to$ +2.3% (v1 $\to$ v2).
- Hard: +3.2% (v0 $\to$ v1) $\to$ +3.9% (v1 $\to$ v2) in a single seed, but across 4 independent training trajectories, the trend is consistently decelerating (+5.1% $\to$ +2.8%).
No Collapse: Diversity (cardinality of generalizing programs per task) increased (2.79 $\to$ 4.05), and no class regressed to zero capability.

The Crossover Diagnosis (Amplification vs. Compounding)

Operating Budget (pass@8): The trained model (v2) outperforms the base (v0).
- Easy: v2 (53.0%) > Base (46.0%).
- Hard: v2 (46.0%) > Base (38.0%).
Large Budget (pass@64): The base model overtakes the trained model.
- Easy: Base (68.0%) > v2 (61.5%).
- Hard: Base (57.5%) > v2 (52.5%).
Interpretation: The trained model concentrates probability mass on "rare-but-reachable" solutions, making them common (winning at low $K$ ), but reduces the model's ability to explore the full tail of the distribution (losing at high $K$ ). The trained model's ceiling remains strictly under the base model's large- $K$ envelope.

Zero-Capability Frontiers

Classes that appeared to have 0% success at low sample counts (e.g., specific 4-atom structures) were found to have non-zero success rates (25–50%) when measured at population scale.
The "real wall" is not at zero, but at the gap between the base model's ability to find a fitting-but-wrong program (high pass@64) and a generalizing program. This gap (~11 pp) is the true capacity bound.

5. Significance and Claims

The paper claims to provide a negative result regarding compounding in a specific, rigorous regime:

Amplification, Not Compounding: In a teacher-free setting with a free, exact verifier, self-training (STaR) amplifies the base model's existing reachability but does not compound to create new reachability. The "ceiling" is paced by the remaining headroom of the base model.
Methodological Validity: The study challenges the validity of "0% $\to$ climb = emergence" tests in compositional domains, arguing that such zeros are often undersampling artifacts.
Dissociation of Diversity and Coverage: The study observes that while the harvested diversity of solutions increases during data generation, the eval-time pass@K coverage of the trained model narrows. This aligns with theories of diversity collapse in SFT, where accuracy optimization concentrates probability mass on modal solutions.

Limitations and Scope:

The result is domain-internal to the specific FlashFill-style DSL and the teacher-free, exact-verifier regime.
The "no compounding" finding may be specific to the lack of an exploration bonus or a noisy verifier; prolonged RL with exploration or learned reward models might behave differently.
Statistical power is modest (K=4 training trajectories), though sufficient to reject acceleration.
The study does not claim to resolve the debate for all RL/SFT regimes, but specifically isolates the "teacher-free, exact-verifier" case to show that without external guidance or exploration, the loop cannot break the base model's distributional envelope.

In summary, the paper argues that without a teacher to import new reasoning patterns or an exploration mechanism to push beyond the reachable frontier, self-training is a process of sharpening and surfacing existing capabilities, not compounding them into new ones.