🤖 AI

Self-Improving Code Generation via Semantic Entropy and Behavioral Consensus

The paper introduces ConSelf, a self-improving framework for code language models that enhances generation capabilities without external teachers or test oracles by utilizing code semantic entropy for curriculum construction and consensus-driven direct preference optimization to mitigate noisy self-supervision.

Original authors: Huan Zhang, Wei Cheng, Wei Hu

Published 2026-04-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Huan Zhang, Wei Cheng, Wei Hu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to write computer code. Usually, to teach a robot, you need a super-smart teacher (a human expert or a more advanced AI) to show it the right answers, and you need a strict referee (a test system) to tell you immediately if the robot's code works or fails.

But what if you don't have a teacher? What if you don't even have a referee to check the answers? You only have a list of problems and some test inputs (like "try this number"), but no way to know if the robot's solution is actually correct.

This is the challenge the paper ConSelf tackles. It asks: Can a robot learn to write better code just by practicing on its own, without a teacher or a referee?

Here is how they solved it, using two simple but powerful ideas:

1. The "Confusion Meter" (Code Semantic Entropy)

The Problem:
If you ask a robot to solve a math problem it doesn't understand, it might guess wildly. It might try 10 different wrong answers, all looking very different from each other. If you try to teach the robot using these 10 wrong guesses, you're just teaching it to be confused. It's like trying to learn to swim by watching someone flail around in the deep end; you'll just learn to panic.

The Solution:
The researchers invented a "Confusion Meter" called Code Semantic Entropy.

How it works: Instead of asking the robot, "Are you sure?" (which it might lie about), they ask, "If you try this problem 10 times, do you get the same result?"
The Analogy: Imagine asking a group of people to solve a riddle.
- If everyone gives the same answer (even if it's wrong), the group is confident but stuck.
- If everyone gives a totally different answer, the group is confused.
- The "Confusion Meter" measures how different the answers are.
The Filter: If the answers are all over the place (high confusion), the robot knows, "I have no idea how to solve this. Let's skip it." If the answers are consistent, it knows, "I have a strategy for this."

This allows the robot to build a Curriculum (a study plan). It ignores the problems it's too confused to solve and the problems it's already mastered, focusing only on the "Goldilocks" problems that are just hard enough to learn from.

2. The "Crowd Wisdom" Filter (Behavioral Consensus)

The Problem:
Even after filtering out the impossible problems, the robot still has to learn from its own guesses. Sometimes, the robot might generate a "winner" (a solution it thinks is good) and a "loser" (a bad one). But what if the "winner" is actually just a lucky guess that happens to look good? If the robot blindly trusts this, it might learn the wrong thing.

The Solution:
They use a method called Consensus-Driven Optimization.

The Analogy: Imagine a game show where the robot has to pick the right answer. Instead of just picking one "winner," it looks at the whole crowd of guesses.
The Logic: If 8 out of 10 guesses do the same thing, that behavior is likely correct (or at least stable). If only 1 guess does something unique, it's probably a weird mistake.
The Weighting: The robot gives more "credit" to the solutions that the crowd agrees on. If the robot picks a "winner" that no one else agrees with, the robot says, "Hmm, this might be a fluke," and learns from it very slowly. If the "winner" is what everyone else is doing, it learns from it quickly.

The Result: A Self-Improving Loop

By combining these two steps, the robot creates a feedback loop:

Pick the right problems: Use the Confusion Meter to skip the impossible and the boring.
Learn safely: Use the Crowd Wisdom to make sure it's not learning from its own hallucinations.

Why is this a big deal?
Usually, to make AI better at coding, you need expensive super-computers to generate "teacher" answers or human experts to write test cases. This paper shows that an AI can actually teach itself using only the problems and the ability to run the code. It's like a student who, instead of waiting for a teacher, looks at their own practice tests, figures out which ones they are confused about, and studies those specifically, while ignoring the ones they already know or the ones that are too hard.

In short: ConSelf is a robot that learns to code by knowing what it doesn't know (and skipping those problems) and by trusting what the majority of its own guesses agree on.

1. Problem Statement

The paper addresses a critical bottleneck in improving Large Language Models (LLMs) for code generation: the reliance on costly external supervision. Current state-of-the-art methods typically require:

Superior Teacher Models: To generate reference solutions (often proprietary and expensive via APIs).
Reliable Test Oracles: To verify the correctness of generated code (the "test oracle problem," where generating correct expected outputs is as hard as solving the problem itself).

In real-world scenarios, obtaining problem descriptions and test inputs is easy, but reference solutions and oracles are scarce. The authors pose the question: Can a code language model improve itself using only problem descriptions and test inputs, without access to a superior teacher or ground-truth oracles?

The core challenge in this unsupervised setting is noise. Without external validation, self-generated solutions for difficult problems are often flawed. Training on these "noisy" data points can degrade model performance rather than improve it. The paper identifies two sub-problems:

What to learn? How to identify which problems are learnable (i.e., where the model has a chance of improvement) and filter out intractable ones.
How to learn? How to train robustly when the preference labels (which solution is better) are inherently noisy and self-generated.

2. Methodology: ConSelf

The authors propose ConSelf, a self-improving framework consisting of three main stages: Observation-Guided Sampling, Curriculum Curation, and Consensus-Driven Optimization.

A. Observation-Guided Sampling

To ensure a diverse candidate pool (a prerequisite for uncertainty estimation), the model does not generate code directly. Instead:

Observation Generation: The model first generates $N_{obs}$ distinct textual "observations" (hints, constraints, strategies, or definitions) for a given problem.
Code Generation: For each observation, the model generates $N_{code}$ candidate programs.
This two-step process forces the model to explore a wider solution space, increasing the behavioral diversity of the candidate set $C_p$ .

B. Curriculum Curation via Code Semantic Entropy

To solve "What to learn?", the authors introduce Code Semantic Entropy, a novel metric to measure a model's epistemic uncertainty for a specific problem.

Execution-Based Clustering: Instead of relying on internal model states (like token entropy or log-likelihood), the method executes all non-crashed candidate programs against the available test inputs.
Behavioral Clustering: Candidates are grouped into clusters based on identical execution traces (output vectors).
Entropy Calculation: The Shannon entropy is calculated over the distribution of these behavioral clusters.
- Low Entropy: The model converges on a single behavior (either all correct or all the same error).
- High Entropy: The model produces chaotic, diverse, and conflicting behaviors (indicating the problem is too hard or the model is confused).
Filtering: Problems with extremely high entropy (noise) or zero entropy (trivial or stuck) are filtered out. Only problems with moderate entropy (indicating learnable uncertainty) are retained to form the curriculum.

C. Consensus-Driven Direct Preference Optimization (Con-DPO)

To solve "How to learn?", the authors propose Con-DPO, an adaptation of Direct Preference Optimization (DPO).

Consensus Scoring: Since there are no ground-truth labels, the "winner" (preferred solution) is selected based on behavioral consensus. A candidate's score is the frequency with which its execution output matches the outputs of other candidates. High consensus implies higher reliability.
Weighted Loss: Standard DPO treats all preference pairs equally. Con-DPO introduces a consensus weight ( $R(y_w)$ ) to the loss function.
$\mathcal{L}_{Con-DPO} = -\mathbb{E} [ R(y_w) \cdot \log \sigma(\hat{r}(y_w, y_l)) ]$
This adaptively down-weights the learning signal for preference pairs where the "winner" has low consensus (likely noisy), preventing the model from overfitting to spurious patterns.

3. Key Contributions

Code Semantic Entropy: A new uncertainty metric grounded in observable functional behavior (execution traces) rather than internal model confidence or linguistic similarity. It is shown to be significantly more robust than token-level entropy or negative log-likelihood.
ConSelf Framework: A dual-strategy approach combining entropy-based curriculum filtering (to select learnable problems) and consensus-driven DPO (to handle noisy self-supervision).
Unsupervised Self-Improvement: Demonstrates that code LLMs can improve significantly without reference solutions or test oracles, relying solely on problem descriptions and test inputs.

4. Experimental Results

The authors evaluated ConSelf on three backbone models (CodeLlama-7B, DeepSeek-Coder-6.7B, Qwen2.5-Coder-7B) across four benchmarks (HumanEval, MBPP, EvalPlus, LiveCodeBench).

Performance Gains: ConSelf achieved relative improvements of 2.73% to 3.95% over base models.
- Example: Qwen2.5-Coder-7B improved from a base average of 71.56% to 74.39%.
Comparison with Baselines:
- Self-SFT (fine-tuning on self-generated "winners") and Self-DPO (standard DPO on unfiltered data) often degraded performance, confirming that training on noisy data is harmful.
- PFPO (Pseudo Feedback) showed mixed results, sometimes improving and sometimes degrading, highlighting the brittleness of pseudo-oracles.
Data Efficiency: ConSelf achieved these gains using significantly fewer training examples (e.g., ~~6,372 examples for DeepSeek-Coder) compared to baselines that used the full unfiltered dataset (~~46,000+), proving the effectiveness of the curriculum curation.
Metric Validation: Experiments showed that Code Semantic Entropy had a much stronger negative correlation with problem difficulty (pass rate) and higher AUC scores in distinguishing solvable vs. unsolvable problems compared to internal confidence metrics.

5. Significance

Solving the Oracle Problem: The work provides a practical pathway for improving code models in scenarios where test oracles are unavailable, a common constraint in real-world software engineering.
Shift from Internal to External Signals: It challenges the reliance on internal model states (logits/entropy) for uncertainty estimation, arguing that execution-based behavioral analysis is the only reliable signal for code generation tasks.
Robust Self-Training: It establishes that self-improvement is possible but requires a "dual-pronged" strategy: rigorous filtering of the curriculum to ensure learnability, and robust optimization algorithms that tolerate residual noise.
Reproducibility: By removing the dependency on proprietary teacher models and expensive oracle generation, the approach lowers the barrier for research and development in code generation.