💻 computer science

Balancing Usefulness and Naturalness: An LLM-based Curation Pipeline for Code Review Comments

This paper proposes two LLM-based curation pipelines, including the CuREV dataset and an exemplar-guided approach, to systematically improve the quality, clarity, and realism of code review comments, thereby enhancing the performance of downstream automation tasks like comment generation and code refinement.

Original authors: Oussama Ben Sghaier, Martin Weyssow, Houari Sahraoui

Published 2026-07-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Oussama Ben Sghaier, Martin Weyssow, Houari Sahraoui

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine code review as a giant, chaotic group chat where developers try to fix a massive, shared Lego castle. Sometimes, the feedback is brilliant: "Hey, that blue brick is in the wrong spot; let's swap it for a red one to make the tower stable." But often, the chat is full of noise: "Ugh, gross," "Same here lol," or mean-spirited rants that don't actually help build anything.

For a long time, scientists tried to teach robots (specifically, Large Language Models or LLMs) to act like helpful reviewers by feeding them this entire, messy chat history. The problem? The robots learned the bad habits too. They started spitting out vague, rude, or confusing comments because that's what they saw in their training data. It's like trying to teach a chef to cook by feeding them a mix of gourmet meals and burnt toast; eventually, the chef just starts serving burnt toast.

The Main Discovery: Cleaning the Kitchen
The authors of this paper decided to clean up the "kitchen" before teaching the robots. They didn't just throw away the bad data; they created two different "curation pipelines" (think of them as two different ways to organize a messy library) to turn the noisy chat logs into a high-quality textbook for robots.

Pipeline 1: The "Strict Editor" (CuREV)
The first method, called CuREV, was like hiring a strict editor who rewrites every single comment in the chat.

What they did: They took every comment, even the good ones, and forced them to be clear, short, polite, and to the point.
The Result: The robots learned super fast because the instructions were so uniform. When asked to write a review, the robot would almost always start with the same phrase: "Consider..."
The Catch: While the robots got very good at following the "Consider..." rule, they sounded a bit robotic and repetitive. It was like a choir where everyone sings the exact same note perfectly, but there's no variety. The paper shows this method improved the robot's ability to generate comments (scoring 11.26 on a test called BLEU, up from 7.71 with the messy original data), but it made the robot's voice sound too much like a single template.

Pipeline 2: The "Smart Mentor" (CuREV+)
The second method, CuREV+, was smarter. Instead of rewriting everything, the authors acted like a wise teacher.

What they did: They first sorted the comments into "Good" and "Bad."
- The Good comments (clear, polite, helpful) were left exactly as they were. They were kept as "exemplars" (perfect examples).
- The Bad comments (rude, vague, or messy) were rewritten by the robot, but this time, the robot was shown the "Good" examples first to inspire it.
The Result: The final dataset was a mix of real human voices and polished, helpful suggestions. The robots learned to be clear and polite without losing the natural variety of human conversation.
The Proof: When tested on writing code reviews, this method scored 11.05 on the BLEU test—almost as good as the strict editor, but with a huge bonus: the comments sounded much more human.
- Diversity Check: The strict editor (CuREV) used the word "consider" over 142,000 times. The smart mentor (CuREV+) used it only 10,000 times, mixing in phrases like "Can we," "Should we," and "I think."
- Real-World Test: When the robots had to actually fix code based on the comments, the CuREV+ method was the winner. It helped the robot fix code with a score of 0.49 (CodeBLEU), beating the strict editor's 0.44 and the messy original data's 0.36.

What the Paper Rules Out
The paper explicitly argues against the idea that we should just feed raw, messy data to robots. They show that training on the original, uncurated data leads to robots that generate irrelevant, unclear, or even uncivil comments. They also suggest that simply rewriting everything (like the first pipeline) might be too extreme, as it strips away the natural diversity of how humans actually talk. The paper argues that you need a balance: keep the good human voices, and only fix the bad ones.

How Sure Are They?
The authors are quite confident in their numbers, but they are careful about how they phrase it.

They measured the quality of the comments using a sophisticated system where a powerful AI (Llama-3.1-70B) acted as a judge, checking for clarity, civility, and relevance. They double-checked this with human reviewers and found the AI agreed with humans almost perfectly (a score of 0.88 for agreement).
They simulated the robots learning by training them on these new datasets and testing them on specific tasks (writing comments and fixing code). The improvements were real and measurable in these tests.
They suggest that this approach creates a better balance between "usefulness" (getting the job done) and "naturalness" (sounding like a human). They don't claim this is the final, perfect solution for all time, but the data strongly supports that CuREV+ is a significant step forward compared to the old, messy way of doing things.

In short, the paper suggests that if you want a robot to be a great code reviewer, don't just feed it a dictionary of perfect sentences. Instead, show it the best human examples, fix the bad ones, and let it learn from the mix. That way, it learns to be helpful without sounding like a broken record.

Technical Summary: Balancing Usefulness and Naturalness in Code Review Curation

Problem Statement

Code review is a fundamental practice in software development, relying on written comments to ensure code quality, maintainability, and correctness. However, the automation of this process using Large Language Models (LLMs) is hindered by the poor quality of existing training datasets. Publicly available code review datasets, often mined directly from repositories, suffer from significant noise, including irrelevant, uncivil, vague, and grammatically incorrect comments. Training LLMs on such data reduces learning efficiency and risks amplifying undesirable behaviors, such as generating unhelpful or inappropriate feedback.

Furthermore, previous attempts to curate these datasets, such as the authors' prior work introducing CuREV, involved systematically reformulating all comments to improve clarity, conciseness, and civility. While this improved quality, it resulted in a homogenized dataset that lost the natural stylistic variability and diversity of human-written reviews, making the data less representative of real-world developer interactions.

Methodology

The authors propose a comprehensive methodology to assess, curate, and evaluate code review datasets, structured around five research questions (RQs). The process involves an evaluation framework, two distinct curation pipelines, and downstream task validation.

1. Evaluation Framework (RQ1)

The authors employ an LLM-as-a-Judge approach using Llama-3.1-70B-Instruct to evaluate a large-scale dataset of 176,613 code review comments (Li et al., 2022b). The framework assesses comments across:

Categorical Dimensions: Type (e.g., Bugfix, Refactoring), Nature (Prescriptive, Descriptive, Clarification), and Civility (Civil vs. Uncivil).
Quality Scores: Relevance, Clarity, and Conciseness (rated 1–10).
Validation: A sanity check involving 100 manually annotated samples confirmed high agreement between human annotators and the LLM judge (Cohen's $\kappa$ up to 1.00 for Civility, near-perfect for others).

2. Curation Pipelines (RQ2)

Two pipelines are introduced to transform the original dataset:

CuREV (Systematic Reformulation):
- Filtering: Removes comments with low relevance scores (< 4).
- Reformulation: Uses an LLM to rewrite all remaining comments to improve clarity, conciseness, and civility while preserving semantic intent.
- Outcome: A fully standardized, high-quality dataset.
CuREV+ (Selective, Exemplar-Guided Reformulation):
- Filtering: Same relevance threshold as CuREV.
- Selection: Partitions the filtered data into "High-Quality" (Civil, Clarity $\ge$ 7, Conciseness $\ge$ 7) and "Low-Quality" comments. High-quality comments are preserved in their original form.
- Exemplar-Guided Reformulation: Low-quality comments are reformulated by the LLM using in-context learning. The model is provided with 10 randomly sampled high-quality comments as examples to inspire stylistic diversity and tone.
- Outcome: A dataset that improves quality while retaining natural human writing styles and diversity.

3. Diversity Analysis (RQ3)

The authors analyze linguistic diversity using Type-Token Ratio (TTR), n-gram diversity (bi- and trigram), Self-BLEU, and entropy. This compares the uniformity of CuREV against the variety in CuREV+.

4. Downstream Impact Evaluation (RQ4 & RQ5)

Three models (DeepSeek-6.7B-Instruct) are fine-tuned on the Original, CuREV, and CuREV+ datasets to evaluate:

Comment Generation: Measured via BLEU score.
Code Refinement: Measured via CodeBLEU and Exact Match (EM) to assess how well the comments guide code changes.

Key Contributions

Improved Curation Pipeline: Introduction of CuREV+, a selective, in-context exemplar-guided pipeline that balances quality improvement with the preservation of stylistic diversity.
Comprehensive Evaluation: A rigorous comparison of CuREV and CuREV+ against the original dataset across multiple quality dimensions (clarity, conciseness, civility) and diversity metrics.
Empirical Study on Downstream Tasks: Demonstration of how dataset curation impacts automated comment generation and code refinement tasks.
Diversity Analysis: Evidence showing that while systematic reformulation (CuREV) creates uniformity, selective reformulation (CuREV+) maintains a broader range of writing styles and vocabulary.

Results

Dataset Quality and Characteristics

Original Dataset: Characterized by a focus on Refactoring (80%) and Bugfix (18%), but plagued by low clarity (avg 6.89), moderate conciseness (7.71), and the presence of uncivil comments (1.23%).
CuREV: Achieved significant gains in clarity (6.89 $\to$ 8.96) and civility (98.8% $\to$ 100%). However, it introduced extreme uniformity, with 90.2% of comments becoming prescriptive and the word "consider" dominating the prefix n-grams (126k occurrences).
CuREV+: Achieved comparable clarity (8.95) and perfect civility. Crucially, it achieved superior conciseness (7.71 $\to$ 8.53) compared to CuREV (8.05). It preserved natural diversity, with a much lower Self-BLEU score (0.1431 vs. 0.2466) and a varied prefix distribution (e.g., "can we," "should we," "please") rather than a single template.

Downstream Task Performance

Comment Generation: Both curated datasets outperformed the original. CuREV achieved the highest BLEU (11.26) due to uniform phrasing, while CuREV+ scored slightly lower (11.05) but offered a more realistic evaluation of generalization beyond formulaic patterns.
Code Refinement: CuREV+ yielded the best results, achieving a CodeBLEU of 0.49 and Exact Match of 463, outperforming CuREV (0.44, 445) and the Original (0.36, 408). This suggests that the blend of clear, reformulated comments and authentic, diverse human comments provides the most effective guidance for code refinement.

Significance and Claims

The paper claims that curating code review datasets requires more than just refining quality; it necessitates balancing standardization with diversity.

Trade-off Resolution: While fully reformulating datasets (CuREV) maximizes clarity and short-term model learning (high BLEU), it risks creating synthetic, repetitive data that may not generalize well to real-world scenarios.
Optimal Strategy: The CuREV+ approach demonstrates that selectively preserving high-quality human comments and using them as exemplars for reformulating low-quality ones yields a dataset that is both high-quality and human-like.
Practical Impact: The study concludes that CuREV+ provides the strongest trade-off for downstream tasks, particularly code refinement, by offering models clearer instructions without sacrificing the stylistic richness necessary for robust generalization.

The authors position this work as a critical step toward overcoming the limitations of noisy, unrefined data in software engineering automation, suggesting that future dataset curation should prioritize preserving natural variation alongside quality improvements.