💬 NLP

Evaluating and Achieving Controllable Code Completion in Code LLM

This paper introduces the Controllable Code Completion Benchmark (C3-Bench) to address the lack of instruction-following evaluation in code LLMs, reveals significant performance gaps between open-source and proprietary models, and proposes a data synthesis pipeline that enables a fine-tuned model to achieve state-of-the-art results on the new benchmark.

Original authors: Jiajun Zhang, Zeyu Cui, Lei Zhang, Jian Yang, Jiaxi Yang, Qiang Liu, Zilei Wang, Binyuan Hui, Liang Wang, Junyang Lin

Published 2026-01-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiajun Zhang, Zeyu Cui, Lei Zhang, Jian Yang, Jiaxi Yang, Qiang Liu, Zilei Wang, Binyuan Hui, Liang Wang, Junyang Lin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master chef (the AI) working in a busy kitchen. For a long time, the way we tested these chefs was simple: we gave them a half-cooked dish (the code before the missing part) and a half-cooked dish after it (the code after the missing part), and asked them to fill in the middle. If the final dish tasted good (passed the unit tests), the chef got a gold star.

The Problem: The "Recipe" Was Missing
The authors of this paper argue that this old way of testing is flawed. In the real world, a head chef doesn't just say, "Fill in the middle." They say, "Fill in the middle, but use only vegetarian ingredients," or "Fill in the middle, but make it exactly three lines long," or "Fill in the middle, but use a specific type of knife."

Current AI models are great at making the dish taste good, but they often ignore the specific instructions on how to make it. They might use meat when you asked for veggies, or write a paragraph when you asked for a single line. The old tests didn't catch this because they only checked if the food was edible, not if the chef listened to the order.

The Solution: C3-Bench (The "Instruction-Following" Test)
To fix this, the team created a new test called C3-Bench (Controllable Code Completion Benchmark). Think of this as a new, much stricter cooking competition.

Instead of just asking for a missing piece of code, every test in C3-Bench comes with a specific, fine-grained instruction. They split these instructions into two main categories:

The "How-To" Instructions (Implementation-Control):
- Analogy: "Build a bridge, but you must use a suspension design, not a beam design."
- The AI has to write code that works and follows a specific style, algorithm, or structure. For example, "Sort this list using a specific sorting method" or "Handle errors in this specific way."
The "Size" Instructions (Scale-Control):
- Analogy: "Write a sentence that is exactly 10 words long," or "Write a paragraph that is exactly 3 sentences."
- The AI has to generate code that fits a strict size limit, like exactly 5 lines of code or a specific block of logic, no more and no less.

What They Found
The team tested over 40 different AI models (both free, open-source ones and expensive, closed-source ones) on this new test. Here is what they discovered:

The "Overconfident" Open-Source Models: On the old tests, many free, open-source AI models performed just as well as the expensive, top-tier ones. However, on this new C3-Bench, they struggled significantly. It turns out they had "memorized" the old tests but didn't actually know how to follow complex instructions. They were like students who memorized the answer key but couldn't solve a new problem with a twist.
The Gap: There is a huge gap between models that can just "make code work" and models that can "make code work exactly how you asked." Even some of the smartest, most expensive models had trouble with the "Size" instructions (like writing exactly the right number of lines).
The Fix: The authors didn't just point out the problem; they built a solution. They created a pipeline to automatically generate thousands of new training examples where an AI writes code and follows a specific instruction. They used this data to train a new version of their model, called Qwen2.5-Coder-C3.
The Result: This new model became the best at following instructions in code completion, beating almost everyone else on the new test while still being good at the old tests.

The Takeaway
This paper introduces a new way to measure AI coding skills that actually matters for real-world use: Can the AI listen?

They found that many current AI models are like talented musicians who play the right notes but ignore the conductor's tempo changes. By creating this new benchmark and a method to train models to listen better, the authors are helping developers build AI tools that don't just write code, but write the right code according to the user's specific, detailed needs. They have made all their data and models available for others to use and improve upon.

Technical Summary: C3-Bench: Evaluating and Achieving Controllable Code Completion in Code LLM

Problem Statement

While Large Language Models (LLMs) have significantly advanced code completion capabilities, current evaluation methodologies remain insufficient for practical software engineering scenarios. Traditional benchmarks (e.g., HumanEval, CrossCodeEval, SAFIM) primarily assess the functional correctness of generated code based on context, often relying on unit tests or similarity metrics. However, they largely overlook a critical aspect of LLM-assisted programming: the model's ability to follow fine-grained user instructions during the completion process. In real-world development, developers frequently require models to generate code that not only functions correctly but also adheres to specific implementation constraints, structural requirements, or scope limitations. The absence of benchmarks that rigorously evaluate this "instruction-following" capability in code completion creates a gap between benchmark performance and practical utility.

Methodology

1. Controllable Code Completion (C3) Definition

The authors introduce the concept of Controllable Code Completion (CCC), extending the traditional tuple $(P, S, G, T)$ (Prefix, Suffix, Ground Truth, Unit Test) to include $I$ (Fine-Grained Instruction). The task requires the model to generate middle code $G$ such that $M(P, S, I) \rightarrow G$ , satisfying both functional correctness (verified by $T$ ) and instruction adherence (alignment with $I$ ).

2. C3-Bench Construction

The Controllable Code Completion Benchmark (C3-Bench) comprises 2,195 high-quality Python tasks derived from HumanEval and SAFIM, enhanced through a four-step synthesis pipeline:

Middle Code Extraction: Utilizing Abstract Syntax Trees (AST) to extract logically complete code blocks and mask nodes to create diverse middle segments.
Equivalent Implementation Generation: Generating multiple functionally equivalent implementations for the same context to test instruction adherence.
Data Filtering: Ensuring code readability (PEP8), appropriate length, and significant implementation diversity.
Instruction Generation:
- Implementation-Control Completion (ICC): Tasks requiring specific implementation strategies (e.g., algorithmic approaches, control flow, data structures). Includes 1,286 instances with unit tests.
- Scale-Control Completion (SCC): Tasks requiring code generation within precise scope limits (e.g., specific line counts, statement blocks). Includes 909 instances without unit tests, focusing on structural conformity.

3. Evaluation Metrics

Pass@1: Measures functional correctness via unit testing (ICC only).
Instruction-Following Rate (IF): Measures adherence to specific requirements, assessed only on functionally correct cases.
Edit Similarity (ES): A static analysis metric for preliminary quality validation.
Automated Judging: An LLM-based system (using Claude3.5-Sonnet) validates instruction adherence with 98% agreement against human experts.

4. Model Training (Qwen2.5-Coder-C3)

To address the scarcity of instruction-completion training data, the authors developed an automated synthesis pipeline:

Seed Generation: Used Claude3.5-Sonnet to create 1,000 high-quality seed pairs.
Large-Scale Synthesis: Leveraged Qwen2.5-Coder-32B-Instruct to generate 200,000 instruction-completion pairs from GitHub repositories.
Fine-Tuning: Applied Supervised Fine-Tuning (SFT) on Qwen2.5-Coder-1.5B and 32B variants using the synthesized data, resulting in Qwen2.5-Coder-C3.

Key Results

1. Performance Gaps in Instruction Following

Comprehensive evaluations of over 40 mainstream LLMs (both open-source and proprietary) reveal significant limitations:

Open-Source vs. Proprietary: While open-source models often match or exceed proprietary models on traditional functional correctness benchmarks (e.g., CrossCodeEval, RepoEval), they exhibit a substantial performance gap in instruction-following capabilities on C3-Bench.
Instruction Sensitivity: Removing instructions from prompts leads to a significant drop in IF rates while Pass@1 remains largely unchanged, confirming that current models do not inherently prioritize instruction adherence over functional generation.
Scale-Control Challenges: Models struggle particularly with Scale-Control Completion (SCC) tasks, indicating limitations in training objectives regarding code scope constraints.

2. Effectiveness of C3-Tuning

The Qwen2.5-Coder-C3 model achieves State-of-the-Art (SOTA) performance on C3-Bench:

It demonstrates substantial improvements in IF rates compared to its base instruction-tuned counterpart (e.g., +23.7% IF on ICC for the 32B model).
It surpasses leading proprietary models (e.g., GPT-4o, o1-series) in Scale-Control tasks while maintaining competitive performance on conventional benchmarks.
The results suggest that open-source models may overfit to existing functional benchmarks and lack generalization in controllable scenarios, but this can be remedied through targeted instruction synthesis.

3. Correlation with Advanced Capabilities

Model rankings on C3-Bench show strong correlations with performance on tasks requiring extensive context understanding (ExecRepoBench) and human preference evaluation (Chatbot Arena-Coding), suggesting that instruction-following ability is a proxy for broader code comprehension.

Significance and Claims

The paper claims that C3-Bench provides the first instruction-guided benchmark for code completion, addressing a critical blind spot in current evaluation landscapes. Its significance lies in:

Revealing Systematic Overestimation: Demonstrating that existing benchmarks systematically overestimate models' capabilities in practical, instruction-heavy development scenarios.
Bridging the Gap: Identifying that open-source code LLMs, despite strong functional performance, lag behind proprietary models in instruction adherence, a gap that can be closed via targeted data synthesis.
Advancing Open-Source Development: Providing a straightforward, reproducible pipeline for synthesizing instruction-completion data, enabling the community to enhance open-source models without relying on proprietary data.

The authors conclude that future research must prioritize instruction-following capabilities in code LLMs to ensure their reliability and controllability in real-world software engineering applications. All code, datasets, and models are open-sourced to facilitate further research.