🤖 machine learning

Primal Generation, Dual Judgment: Self-Training from Test-Time Scaling

This paper introduces DuST, a self-training framework that leverages test-time sampling to create a "dual judgment space" where models learn to rank candidate programs via on-policy reinforcement learning, thereby improving both their ability to judge code correctness and generate high-quality solutions without direct rewards for correct generation.

Original authors: Yizhu Jiao, Ruixiang Zhang, Richard Bai, Jiawei Han, Ronan Collobert, Yizhe Zhang

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yizhu Jiao, Ruixiang Zhang, Richard Bai, Jiawei Han, Ronan Collobert, Yizhe Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to write computer code. Traditionally, you teach it by saying, "Here is a problem. Write a solution. Run it. If it works, great! If it crashes, try again." This is like giving the robot a single "Pass" or "Fail" grade for each attempt. The robot learns, but it only knows that it failed, not why it failed compared to other things it could have written.

This paper introduces a new teaching method called DuST (Dual Self-Training). It changes the game by using a concept called "Primal Generation" vs. "Dual Judgment."

Here is the breakdown using simple analogies:

1. The Old Way: The "Pass/Fail" Quiz

Primal Generation:
Think of the robot as a student taking a test.

The Process: The student writes one answer. The teacher checks it.
The Feedback: The teacher gives a single red "X" or a green checkmark.
The Problem: If the student gets an "X," they know they are wrong, but they don't know how close they were to being right, or why their specific answer was worse than another possible answer they might have thought of. They just know "Don't do this."

2. The New Idea: The "Taste Test" Panel

Test-Time Scaling (The Setup):
Before the paper's method, researchers tried to help by having the robot write many answers (say, 4 or 5) at once and picking the best one. This is like a "Taste Test."

The Flaw: In the old way, once the robot picks the best answer, the other 4 answers are thrown in the trash. The robot learns nothing from the fact that Answer #2 was "almost right" while Answer #4 was "completely wrong." That valuable comparison data is wasted.

3. The DuST Solution: Learning from the "Also-Rans"

Dual Judgment Space:
The authors argue that the robot should learn from the comparison between its own answers, not just the final winner.

The Analogy: Imagine a cooking competition.
- Old Way: The judge tastes one dish and says, "Burnt. Fail." The chef learns nothing about what makes a dish good.
- DuST Way: The chef makes 4 different versions of the same dish. The judge tastes all 4.
  - Dish A: Perfect.
  - Dish B: A little too salty.
  - Dish C: Undercooked.
  - Dish D: Burnt.
- The Lesson: Instead of just telling the chef "Dish A is good," the teacher says, "Dish A is the winner because it's balanced. Dish B failed because of salt. Dish C failed because of heat." The chef learns the differences between success and failure.

4. How DuST Works (The Recipe)

The paper describes a specific loop the robot goes through:

Generate: The robot writes a batch of code solutions (the "Taste Test" batch).
Judge: The robot runs them all in a safe sandbox to see which ones actually work (Pass/Fail).
Group: The robot looks at the batch. If all failed or all passed, it throws that batch away. It only keeps batches where some worked and some failed. This is the "Mixed Group."
Rank (The Training): The robot is asked to rank these mixed groups from "Best" to "Worst."
- Crucial Point: The robot is never rewarded for writing the code itself. It is only rewarded for correctly identifying which code is better.
- It gets points for saying, "This working code is better than this broken code."
The Magic Transfer: Even though the robot was only trained to be a judge, it surprisingly gets better at being a writer. By learning to spot the subtle differences between a working solution and a broken one, the robot starts to "understand" what makes code work. It internalizes these rules and starts writing better code on its own.

5. The Results: Why It Matters

The paper tested this on several different AI models (ranging from small to very large) using a standard coding test called LiveCodeBench.

Better Judges: The models became much better at spotting correct code (improving their "ranking" score).
Better Writers: Surprisingly, the models also got better at writing code from scratch, even though they were never directly told "Write a correct program."
The "One-Shot" Miracle: Before this training, a model might need to write 4 answers and pick the best one to get a high score. After DuST training, the model could write just one answer and get the same high score. It learned to generate the "best" answer immediately, rather than needing to guess and check.

6. The Secret Sauce: RL vs. Just Copying

The authors did a final experiment to see why this worked.

SFT (Supervised Fine-Tuning): They tried just teaching the robot to copy the correct rankings (like a student memorizing an answer key). This made the robot a better judge, but did not make it a better writer.
RL (Reinforcement Learning): They used a method called GRPO, where the robot learns by trying, failing, and adjusting its own behavior based on the rewards. This was the key. The "active learning" of ranking allowed the robot to change how it thinks, which improved its writing.

Summary

DuST is a method where an AI learns to code by becoming a master critic of its own work. By training the AI to distinguish between its own "good" and "bad" attempts, it learns the hidden rules of correctness. This knowledge then leaks back into its writing ability, making it a better coder without ever being directly told how to write a perfect program. It turns the "wasted" data of failed attempts into a powerful teacher.

Technical Summary: Primal Generation, Dual Judgment: Self-Training from Test-Time Scaling

Problem Statement

Code generation models are typically trained in the primal space, where a model produces a candidate solution and receives sparse, binary execution feedback (pass/fail). While test-time scaling (sampling multiple candidates and selecting the best via a judge) improves inference accuracy, the comparative information revealed during this process—specifically the structural differences between successful and failed attempts—is discarded after inference. Existing "generate-then-judge" systems use this comparative structure only for inference-time ranking, failing to update the generator itself. Consequently, the model may continue to assign high probability to plausible but incorrect code in future rollouts.

The paper argues that the comparative structure of test-time scaling defines a dual judgment space. This space offers a denser training signal than the primal space: instead of isolated pass/fail bits, it provides relative correctness structures across plausible attempts, revealing what distinguishes success from failure.

Methodology: DuST (Dual Self-Training)

The authors introduce DuST, a framework for self-training from this dual judgment space. The core hypothesis is that a model trained to discriminate correct from incorrect code in the dual space acquires correctness-sensitive features that benefit primal generation, even without direct rewards for generating correct code.

1. Data Construction

Sampling: For a coding problem $x$ , the base model samples a large pool of candidate codes ( $M$ candidates).
Execution & Labeling: Candidates are executed in a sandbox to obtain binary correctness labels ($1$ for pass, $0$ for fail).
Grouping: Candidates are partitioned into groups of size $K$ . Only mixed-quality groups (containing at least one correct and one incorrect candidate) are retained. Groups with all correct or all incorrect candidates are discarded as they lack ranking contrast.
Dataset: The resulting dataset consists of problems paired with self-generated candidate sets and execution-grounded labels.

2. Training Objective

The model is trained to rank candidates within a group based on execution correctness.

Reward Function: The reward is purely discriminative. It measures how often execution-correct candidates are ranked above execution-incorrect candidates.
- $R_{exec}$ calculates the fraction of correct-over-incorrect pairs $(i, j)$ where the predicted rank of $i$ is better than $j$ .
- Malformed responses receive a negative format penalty.
Optimization: The framework uses Group Relative Policy Optimization (GRPO).
- The candidate code samples are held fixed (off-policy regarding generation).
- The model samples judgment responses (reasoning traces and rankings) on-policy.
- The policy is updated to maximize the ranking reward, regularized against a frozen reference policy.

3. Key Distinction

Unlike standard Reinforcement Learning (RL) for code generation, DuST never directly rewards the generation of correct programs. The supervision comes entirely from judging the correctness contrasts among the model's own attempts. The updated parameters are the same ones used for subsequent code generation.

Key Contributions

Dual Judgment Space Framework: The paper proposes viewing test-time scaling not just as an inference strategy but as a source of dense, contrastive training signals in a dual space.
Discriminative Self-Training: DuST demonstrates that training a model to rank candidates by execution correctness (a discriminative task) improves its generative capabilities (a generative task) without direct generation rewards.
Mechanism Isolation: Through ablation studies comparing Supervised Fine-Tuning (SFT) and GRPO, the paper identifies on-policy RL as the critical mechanism for transferring dual-space learning back to primal generation. SFT improves judgment but fails to improve generation, whereas GRPO improves both.
Reward Density: The paper highlights that the dense contrastive signal of ranking (exposing $O(N^2)$ pairwise comparisons) is more effective for transfer than sparse primal rewards or selection-based rewards.

Experimental Results

The method was evaluated on LiveCodeBench (v5 and v6) across five models spanning two families (Qwen, GPT-oss) and three scales (4B to 30B).

Test-Time Scaling Accuracy: DuST consistently improves Best-of-4 accuracy. For Qwen3-30B-Thinking on LiveCodeBench v6, accuracy improved by +4.1% (from 68.7% to 72.6%).
Judgment Quality: The trained model's ability to rank candidates improved significantly, with NDCG increasing by +6.2 for Qwen3-30B-Thinking.
Primal Generation (Pass@1): Remarkably, single-sample generation quality improved without direct rewards. For Qwen3-30B-Thinking, pass@1 increased by +3.1% (from 65.4% to 68.5%).
Generalization: Improvements were observed across all model sizes (4B–30B) and families, suggesting the approach is architecture-agnostic.
Comparison to SFT: When trained on the same ranking data, SFT improved judgment (NDCG) but left generation (pass@1) unchanged. Only GRPO improved both, confirming that the on-policy feedback loop is essential for the transfer.
Reward Sensitivity: Ranking rewards (Pairwise Accuracy/NDCG) outperformed selection rewards (top-1 only) in transferring to generation, likely because ranking exposes the full contrastive structure of correct vs. incorrect code.

Significance and Claims

The paper claims that the primal (generation) and dual (judgment) spaces of code generation are more tightly coupled than standard training assumes.

Transfer of Understanding: The authors argue that by learning to distinguish why a solution fails or succeeds relative to other plausible attempts, the model internalizes features of executable correctness (e.g., algorithmic invariants, edge cases). These features subsequently shift the probability mass of the generator toward correct trajectories.
Efficiency: The trained model's single rollout matches the base model's Best-of-4 performance, effectively "baking in" the benefits of test-time scaling into the model weights.
Limitations of Current Paradigms: The work suggests that discarding comparative information after inference is a missed opportunity. The contrastive structure revealed at test time is a rich, largely untapped training signal.

The paper concludes that on-policy reinforcement learning in the dual judgment space is a viable and effective mechanism for improving code generation, offering a new pathway for self-improvement that does not rely on external human preferences or direct generation rewards.