💬 NLP

CodeEvo: Interaction-Driven Synthesis of Code-centric Data through Hybrid and Iterative Feedback

This paper introduces CodeEvo, a dual-agent framework that synthesizes high-quality, logically complex instruction-code pairs through iterative hybrid feedback, resulting in the CodeEvo-100K dataset that significantly enhances the performance of code generation models.

Original authors: Qiushi Sun, Jinyang Gong, Lei Li, Qipeng Guo, Fei Yuan

Published 2026-07-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qiushi Sun, Jinyang Gong, Lei Li, Qipeng Guo, Fei Yuan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to be a brilliant programmer. You can't just sit down and write a million coding problems by hand; that would take forever, and you'd run out of ideas. So, scientists have started using other, smarter robots (called Large Language Models) to write these practice problems for the student robot. But here's the catch: when these AI teachers try to make up new problems, they often get it wrong. They might write a math question that has no answer, or a coding task that crashes the computer the moment you try to run it. It's like a teacher handing out a test where half the questions are nonsense. This paper, titled CodeEvo, tackles this messy problem by inventing a new way for AI to teach itself how to write perfect, solvable coding challenges without needing a human to check every single one.

The core idea relies on two main concepts. First, there's the instruction, which is the problem statement (like "Write a function that sorts a list"). Second, there's the code, which is the solution. The goal is to generate pairs of these that are not only correct but also get progressively harder and more interesting. The authors suggest that the old way of doing this—just asking an AI to "make it harder"—is too vague and leads to broken results. Instead, they propose a system where the AI acts like a team of two: a Coder who tries to solve the problem, and a Reviewer who grades the work and designs the next challenge.

The Two-Agent Dance: Coder and Reviewer

The authors built a framework called CodeEvo, which is essentially a high-tech, automated workshop where two AI agents work together in a loop. Think of it like a video game where one player (the Coder) tries to beat a level, and a second player (the Reviewer) acts as both the level designer and the referee.

In the old methods, the "level designer" would just shout, "Make this level harder!" and hope for the best. Often, the result was a level that was impossible to beat or didn't make sense. CodeEvo changes the game. Before the Reviewer even writes a new problem, it creates a Schema. Imagine this as a blueprint or a recipe card. The Reviewer looks at the current problem and picks specific "ingredients" (like keywords: Matrix, Recursion, Big Integer) to mix in. The Schema is a plan that says, "Okay, we are going to take the current logic and add a matrix multiplication step to make it tricky, but we must keep it solvable." This ensures the new problem is grounded in real logic, not just random word salad.

Once the Reviewer designs the blueprint, the Coder tries to write the code to solve it. But here's where the magic happens: the system doesn't just accept the code. It puts the code through a Hybrid Feedback loop.

The Double-Check System

Usually, when an AI writes code, it might look correct but fail when you actually run it. To fix this, CodeEvo uses a two-step verification process. First, it runs the code through a compiler (a tool that checks if the code follows the strict rules of the programming language). If the code crashes or has a syntax error, the compiler says "No." But sometimes, code runs without crashing but still gives the wrong answer.

So, the Reviewer steps in again. This time, it acts like a human judge, reading the code and the problem description to see if the logic actually matches. It asks, "Did you actually solve the problem, or did you just get lucky?" By combining the strict, unfeeling check of the compiler with the smart, contextual check of the AI Reviewer, the system filters out bad data. If the code fails, the Coder gets a detailed report on what went wrong and tries again. This happens over and over until the code is perfect.

Building a Mountain of Data

Using this method, the authors created a massive dataset called CodeEvo-100K. This isn't just a random pile of 100,000 problems; it's a carefully curated collection with "stepped difficulty." They started with simple seed problems and let the agents evolve them. Some problems stayed easy, some became medium, and a special "Hard" set was created by pushing the agents through three or more rounds of refinement.

The results are impressive. When they took other AI models and trained them on this new dataset, those models got significantly better at solving coding problems than models trained on other synthetic data. In fact, a smaller amount of high-quality CodeEvo data (about 17,000 hard problems) actually worked better than a much larger dataset (75,000 problems) made by older methods. This suggests that the quality of the training data matters more than just having a huge volume of it.

Why This Matters

The paper explicitly argues against the idea that you can just use simple, rigid rules (heuristics) to generate good coding data. They show that without a structured plan (the Schema) and a rigorous checking system (the Hybrid Feedback), the generated data is often full of errors or impossible tasks. They also found that while the process is slower than just asking an AI to spit out code once, the resulting data is far superior.

In short, CodeEvo suggests that if you want to build a smart coding AI, you shouldn't just feed it a million random problems. Instead, you should build a system where AI agents teach each other, plan their lessons carefully, and grade each other's homework with a fine-tooth comb. The result is a dataset that is not only huge but also logically sound, executable, and ready to train the next generation of code-generating robots.

Technical Summary: CodeEvo

Problem Statement

The training of Large Language Models (LLMs) for code generation relies heavily on high-quality instruction-code pairs. While manual curation provides ideal data, it is labor-intensive, difficult to scale, and increasingly exhausted. Automated synthesis methods, such as Self-Instruct and Evol-Instruct, have emerged as alternatives but often suffer from significant limitations. These approaches typically rely on rigid, predefined heuristics (e.g., "make it harder") that result in instructions that are poorly grounded, semantically vague, or logically inconsistent. Furthermore, the generated code frequently lacks functional correctness, failing to execute or pass unit tests due to the absence of robust verification mechanisms during the synthesis process. The core challenge addressed by this work is designing a fully automated, reference-free synthesis pipeline that produces instruction-code pairs which are both well-grounded and executable.

Methodology

The authors propose CodeEvo, an interaction-driven synthesis framework that orchestrates a dual-agent architecture to generate high-quality code-centric data. The framework operates without human annotation or gold references, relying instead on a small set of seed instructions.

Core Components

Dual-Agent Architecture:
- Coder Agent: Responsible for generating candidate solutions and test cases based on given instructions. It iteratively refines its code based on feedback.
- Reviewer Agent: Acts as an orchestrator and judge. It evaluates candidate solutions, provides tailored feedback, and dynamically constructs new instructions.
Schema-Guided Interleaved Synthesis:
To overcome the vagueness of heuristic-based instruction evolution, CodeEvo introduces a schema-driven process. Before generating a new instruction, the Reviewer formulates a Schema—a structured blueprint outlining the logic, desired complexity, and combination of concepts. This process is guided by task-specific keywords. The Reviewer then generates the new instruction based on this Schema. This bidirectional mechanism allows for the strategic integration of keywords to increase complexity or the selective omission of keywords to create simplified variants if a task proves intractable, ensuring logical coherence and groundedness.
Hybrid Feedback Loop:
To ensure functional correctness, CodeEvo employs a hybrid verification protocol that synergizes two distinct signals:
- Deterministic Compiler Feedback: The Coder executes the generated code against test cases, providing a pass/fail signal.
- Semantic Evaluation: The Reviewer performs a deep semantic audit, scrutinizing the solution's alignment with the instruction, the handling of complex constraints, and potential edge cases missed by the test suite.
  These signals are fused into a comprehensive hybrid feedback ( $f_{hybrid}$ ) that guides the iterative refinement of the code. If a solution fails, the Coder refines it; if the instruction itself is too difficult, the Reviewer can generate a simpler variant.

Workflow

The synthesis process begins with seed instructions (e.g., from LeetCode or math problems) paired with keyword sets. The agents engage in a collaborative refinement loop:

The Reviewer plans a Schema and writes a new instruction.
The Coder generates code and test cases.
The system validates the solution using the hybrid feedback mechanism.
If valid, the pair is added to the dataset; if invalid, the Coder refines the code or the Reviewer adjusts the instruction difficulty.
This process captures the complete evolutionary trajectory, including intermediate reasoning and refinement steps.

Key Contributions

CodeEvo Framework: A novel interaction-driven synthesis framework that systematically improves the quality of code-centric synthetic data from both instruction and code perspectives.
Hybrid Feedback Mechanism: A design that integrates the rigorous determinism of compiler verification with the adaptive generative flexibility of LLM agents to ensure functional correctness and logical integrity.
CodeEvo-100K Dataset: A large-scale dataset of instruction-code pairs featuring stepped difficulty levels (Easy, Medium, Hard) and capturing the complete evolutionary trajectory of the synthesis process.
Insights into Data Synthesis: Extensive analysis providing insights into the quality, diversity, scalability, and difficulty attributes of synthetic code data.

Experimental Results

The authors evaluated models fine-tuned on CodeEvo data across multiple backbones (including Qwen3-8B, Qwen2.5-Coder-7B, InternLM3-8B, StarCoder2-7B, and DeepSeek-Coder-6.7B) and benchmarks (HumanEval, MBPP, BigCodeBench, and LiveCodeBench).

Performance Superiority: Models fine-tuned on CodeEvo data consistently outperformed established baselines (Zero-Shot, Evol-Instruct, and OSS-Instruct) across the majority of benchmarks.
Data Efficiency: Notably, a subset of merely 17K "hard" data generated by medium-scale (32B) agents elicited a larger performance boost than the 75K OSS-Instruct dataset. This suggests that the targeted, feedback-driven synthesis algorithm is more critical than sheer data volume.
Generalization: The framework demonstrated robustness across different agent scales (medium-scale 32B vs. large-scale 120B) and different backbone models.
Quality and Difficulty: Human evaluations indicated that CodeEvo-generated instructions were rated as significantly more difficult and consistent in difficulty compared to those from Evol-Instruct. Diversity analysis showed that CodeEvo achieved lower average cosine similarity in instruction embeddings, indicating higher semantic diversity.

Significance and Claims

The paper claims that CodeEvo offers a scalable and verifiable path toward advancing code intelligence with minimal human supervision. The authors emphasize that the design of the synthesis framework is more decisive for performance than the volume of data. By moving beyond rigid heuristics to an interaction-driven, schema-guided, and hybrid-verified process, CodeEvo addresses the critical gaps of ungrounded instructions and non-executable code found in previous automated synthesis methods. The work demonstrates that high-quality, grounded instruction-code pairs can be synthesized autonomously, potentially reducing the reliance on expensive manual curation.

The authors acknowledge limitations, including dependence on the capabilities of the backbone LLMs, computational throughput constraints due to multi-turn interactions, and the inherent risk of imperfect test case coverage generated by agents. However, they posit that the hybrid feedback mechanism effectively mitigates these issues to a significant degree.