💬 NLP

Conv-to-Bench: Evaluating Language Models Via User-Assistant Dialogues In Code Tasks

Conv-to-Bench is a scalable, multi-stage framework that automatically transforms authentic multi-turn user-assistant dialogues into structured, verifiable evaluation benchmarks for code tasks, achieving near-perfect alignment with human-authored standards while significantly reducing the reliance on labor-intensive expert curation.

Original authors: Victor M. dos Santos, Andre C. Castro, Samuel L. de S. Toledo, Bruno M. L. Calura, Lisandra C. de M. Menezes, Raul C. R. Mata, Telma W. de L. Soares, Bryan L. M. de Oliveira

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Victor M. dos Santos, Andre C. Castro, Samuel L. de S. Toledo, Bruno M. L. Calura, Lisandra C. de M. Menezes, Raul C. R. Mata, Telma W. de L. Soares, Bryan L. M. de Oliveira

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to write computer code. To do this well, you need a test to see if the robot is getting better. Traditionally, creating these tests is like hiring a team of master chefs to hand-write 1,000 unique, perfect recipes. It's expensive, slow, and hard to scale. If you want a new test, you have to wait for the chefs to write it again.

This paper introduces a new way to make these tests called Conv-to-Bench. Instead of hiring chefs to write new recipes from scratch, the researchers decided to look at the "kitchen logs" of real people talking to AI assistants. They asked: Can we turn these messy, real-life conversations into a perfect test?

Here is how they did it, broken down into simple steps:

1. The Problem: The "Chef" Bottleneck

Right now, the best tests for AI (like BigCodeBench) are written by human experts. It takes them a year to make a good test. It's like trying to build a new city by hand-carving every single brick. It's high quality, but it's too slow to keep up with how fast AI is learning.

2. The Solution: Mining the "Kitchen Logs"

The researchers realized that millions of people are already talking to AI every day, asking for help with code. These conversations are goldmines.

The Scenario: A user asks, "Write a Python script." The AI writes something wrong. The user says, "No, it crashes when I run it. Fix the loop." The AI tries again. The user says, "Great, now add a comment."
The Insight: This back-and-forth isn't just a chat; it's a blueprint. The user's final request, combined with all their corrections, is actually a very detailed, perfect set of instructions for what a good code solution looks like.

3. The Process: Turning Chat into a Checklist

The team built a system (Conv-to-Bench) that acts like a super-smart editor. It takes these long, messy conversations and does three things:

Filters: It throws away the chats about cooking or weather, keeping only the ones about coding.
Synthesizes: It reads the whole conversation and writes down one single, perfect "Master Instruction" that combines the original request with all the fixes the user asked for.
Creates a Checklist: It turns that instruction into a simple "Yes/No" checklist. For example: "Does the code run without errors?" "Does it handle the loop correctly?" "Are there comments?"

4. The Experiment: Did it Work?

They tested this new system by seeing if it could rank AI models the same way the expensive, human-written tests do.

The Result: It worked incredibly well. When they compared their AI-generated tests to the human-written "Gold Standard" (BigCodeBench), the rankings matched almost perfectly. In some cases, the correlation was 1.0 out of 1.0—meaning the new system agreed with the human experts 100%.
The "Feedback" Twist: They tried two versions. One used only the final instructions, and the other used the instructions plus the user's complaints and corrections (the feedback). Surprisingly, the version with only the instructions was actually more stable and reliable. The user's feedback sometimes added "noise" (confusion) rather than clarity, like a customer changing their mind too many times.

5. The Verdict

The paper concludes that we don't need to wait for human experts to write every new test. We can use the natural, messy conversations people already have with AI to build high-quality, reliable tests automatically.

In a nutshell:
Think of traditional testing as hand-painting a masterpiece (slow, expensive, perfect).
Conv-to-Bench is like using a high-tech scanner to turn millions of rough sketches made by the public into a perfect, standardized painting. It's faster, cheaper, and surprisingly just as accurate, provided you stick to the main instructions and ignore the messy scribbles in the margins.

Technical Summary: CONV-TO-BENCH

Problem Statement

The rapid advancement of Large Language Models (LLMs) has outpaced the scalability of traditional evaluation benchmarks. Established standards for code generation and reasoning (e.g., BigCodeBench, HumanEval) rely heavily on labor-intensive, multi-stage expert curation. This human-dependent process creates a significant bottleneck, making the development of diverse, high-fidelity evaluation sets resource-heavy and time-consuming. While vast repositories of authentic user-assistant dialogues exist (e.g., LMSYS-Chat-1M, WildChat), they have primarily been used for training or behavioral analysis rather than benchmark construction. The challenge lies in transforming these unstructured, multi-turn interactions—which contain iterative refinements and implicit feedback—into structured, verifiable evaluation criteria without introducing noise or losing the nuance of user intent.

Methodology

The authors propose Conv-to-Bench, a multi-stage framework designed to automatically convert raw, multi-turn user-assistant dialogues into structured evaluation datasets consisting of consolidated instructions and binary requirement checklists. The methodology operates through the following pipeline:

Domain-Specific Filtering:
- Clustering: Conversations are grouped using unsupervised topic modeling (dense vector representations) and screened against a domain-specific keyword lexicon to identify relevant clusters (e.g., programming).
- Classification: A zero-shot LLM-based classifier performs instance-level verification on individual conversations to filter out thematic noise, ensuring the corpus contains only high-signal, in-domain interactions.
Instruction Synthesis:
- An LLM analyzes the complete conversational history to synthesize a single, comprehensive instruction. This process integrates the initial request with all subsequent clarifications and constraints, reconstructing the user's evolving intent.
- A binary classifier filters out synthesized instructions that are semantically ambiguous, too vague, or dependent on external files not present in the logs.
Feedback-Driven Requirement Synthesis:
- Feedback Identification: The framework identifies specific user messages constituting evaluative feedback (Positive or Negative) based on a strict "reaction-based" protocol, ignoring silence or neutral continuations.
- Checklist Generation: Acting as a Quality Assurance analyst, an LLM translates the dialogue, synthesized instruction, and identified feedback into a list of atomic, testable requirements formatted as binary (Yes/No) questions. Each requirement is tagged by its origin (instruction vs. feedback) to enable ablation analysis.
Scoring and Statistical Calibration:
- Hierarchical Estimation: To prevent bias from instructions with varying numbers of requirements, the framework computes an Instruction-Level Score ( $S_i$ ) by averaging the binary outcomes of its atomic requirements. The global performance ( $\theta$ ) is the mean of these instruction-level scores.
- Uncertainty Quantification: A Cluster Bootstrap method ( $B=1,000$ ) is employed to generate 95% confidence intervals, accounting for variance in task difficulty and ensuring the stability of the benchmark.

Key Contributions

Automated Benchmark Construction: The introduction of a pipeline that transforms authentic multi-turn dialogues into structured evaluation sets, reducing reliance on manual expert curation.
Instruction-Centric Extraction: The demonstration that synthesizing instructions from the entire conversational history (rather than just the initial prompt) captures the full scope of user intent.
Ablation of Feedback: A rigorous investigation into whether implicit user feedback (corrections, confirmations) serves as a useful refinement signal or introduces noise. The study isolates "Instructions-Only" vs. "Full (Instructions + Feedback)" configurations.
Reliability Validation: A comprehensive evaluation of the "LLM-as-a-judge" framework, comparing automated verdicts against human-verified ground truth using Cohen's Kappa and F1-scores.

Results

The framework was evaluated using 387 high-quality (instruction, requirement checklist) pairs derived from 2 million raw dialogues, focusing on the programming domain.

Judge Reliability: The primary evaluator (Gemini-2.5-Flash) achieved substantial agreement with human experts ( $\kappa = 0.705$ ) and a high Macro F1-score (0.852), validating the automated scoring mechanism.
Alignment with Gold Standards: Conv-to-Bench demonstrated near-perfect alignment with BigCodeBench.
- Using the Instructions-Only configuration, the framework achieved a Spearman correlation of $\rho = 1.000$ with BigCodeBench (Full Set) across both "Instruct" and "Complete" modes.
- The Full Pipeline (including feedback) showed slightly lower or inconsistent correlations in some configurations, suggesting that conversational noise can degrade evaluative integrity.
Comparison with Baselines: Conv-to-Bench (Instructions-Only) outperformed the Arena-Hard-Auto (Code) benchmark in ranking stability against BigCodeBench, achieving higher Spearman correlations ( $\rho = 1.000$ vs. $\rho = 0.943$ for Arena-Hard-Auto on the Full Set).

Significance and Claims

The paper claims that Conv-to-Bench provides a scalable, cost-effective paradigm for maintaining high-fidelity evaluation standards as user-centric AI applications diversify.

Scalability: By leveraging the "instructional evolution" found in real-world logs, the framework bypasses the bottleneck of manual task curation while producing evaluation sets that align with expert-authored standards.
Robustness of Instruction Synthesis: A key finding is that while multi-turn interactions capture the iterative nature of user intent, instruction-centric extraction (consolidating the full history into a single prompt) provides a more robust foundation for evaluation than attempting to incorporate explicit feedback signals, which often introduce noise.
Validation of Automated Evaluation: The study confirms that automated judges, when properly calibrated and validated against human ground truth, can reliably assess complex code generation tasks, offering a viable alternative to resource-intensive human verification.

The authors maintain a modest stance, acknowledging that the current analysis is limited to the English programming domain and that the accuracy of the framework remains contingent on the capabilities of the LLM judge. Future work is proposed to extend this methodology to multi-lingual datasets and diverse technical fields.