💬 NLP

CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks

CoEval is an open-source framework that generates contamination-free, attribute-controlled benchmarks and employs a diverse ensemble of judge models to reliably rank language models for custom tasks without requiring labeled data or trusting public benchmarks.

Original authors: Alexander Apartsin, Yehudit Aperstein

Published 2026-06-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alexander Apartsin, Yehudit Aperstein

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a hiring manager trying to pick the best employee for a very specific job, like "organizing a chaotic warehouse" or "writing funny jokes about gardening." You have a problem: you don't have any past performance reviews (labeled data), and the standard tests everyone else uses (public benchmarks) are useless because the candidates have already memorized the answers from studying them beforehand.

This is the exact problem CoEval solves. It's a new, open-source tool that helps you rank AI models for your specific needs without needing human experts to grade them or worrying about cheating.

Here is how it works, broken down into simple analogies:

1. The "Fresh Test" Generator (No Cheating Allowed)

Usually, AI models take the same old tests (like the SATs or GRE) that have been around for years. Because these tests are so popular, the AI models have likely seen the questions during their training and just memorized the answers. It's like a student memorizing the answer key instead of learning the math.

CoEval changes the game. Instead of using old questions, it uses a "Teacher AI" to invent brand-new questions on the spot, based only on a description of your task.

The Analogy: Imagine a teacher who creates a math test while the student is in the room. The student couldn't have memorized the answers because the questions didn't exist until that second.
The Result: The paper proves these new questions are 100% fresh. They share no 13-word phrases with any major public test banks, meaning the AI can't cheat by recalling old data.

2. The "Jury" of Judges (Diversity Over Numbers)

Once the AI models answer these fresh questions, someone has to grade them. You could ask one super-smart AI to grade them, but that's risky. That single judge might have weird biases, like preferring long answers over short, good ones, or liking answers that sound like its own family of models.

CoEval uses a "Jury" approach. It gathers a small group of judges from different companies (e.g., one from OpenAI, one from Anthropic, one from Google).

The Analogy: Think of a courtroom. If you have one judge who is known to be grumpy, the verdict might be unfair. But if you have a jury of three people from different backgrounds, their individual quirks cancel each other out. If one judge likes long answers and another hates them, the average score becomes fair.
The Big Discovery: The paper found that who is on the jury matters more than how many people are there. Adding more judges from the same company just amplifies their shared bias. But a small, diverse group of judges is incredibly reliable. In fact, a single judge can sometimes be "anti-correlated" (giving the wrong answer), but the diverse jury almost never is.

3. The "No-Human" Factory

Usually, to build a good test, you need humans to write questions and grade answers. This is slow and expensive.

The Analogy: CoEval is like a fully automated factory. You give it a blueprint (a description of your task), and it automatically:
1. Designs the test questions.
2. Has the AI candidates take the test.
3. Assembles the diverse jury to grade the answers.
4. Spits out a final ranking.
The Cost: The authors ran a massive study with nearly 8,000 evaluations for the price of a cup of coffee ($5.89). It's so cheap you could run a new test every time a new AI model comes out.

4. Why This Matters

The paper tested CoEval in three real-world scenarios where no "correct" answers existed:

Drug Interactions: Figuring out if two medicines clash.
Clinical Reasoning: Diagnosing patient symptoms.
Legal Analysis: Interpreting laws.

In these areas, there are no standard "gold standard" tests. CoEval successfully ranked the models, showing which ones were best for these specific jobs. It even caught that a smaller, cheaper model was actually worse than a larger one, something a single biased judge might have missed.

The Bottom Line

CoEval is a tool that says: "Don't trust the old, memorized tests. Don't rely on a single judge who might be biased. Instead, generate a fresh test for your specific job and have a diverse, small team of AI judges grade it."

It turns the messy, expensive process of testing AI into a cheap, repeatable, and fair process that any team can use to find the right AI for their specific needs.

Technical Summary: CoEval

Problem Statement

Practitioners face a critical bottleneck when selecting language models for specific, custom applications (e.g., summarizing clinical notes or triaging support tickets). This challenge arises from two converging conditions:

Absence of Labeled Data: Building a representative, task-specific benchmark manually is slow, costly, and often impossible for new or proprietary domains.
Untrustworthy Public Benchmarks: Standard static benchmarks are increasingly contaminated. Test items often leak into pretraining corpora, causing high scores to reflect memorization rather than generalization or fitness for the specific use case.

Existing solutions like static benchmarks suffer from rigidity (non-extensibility), contamination susceptibility, and genericity (mismatched input distributions). Meanwhile, "LLM-as-judge" approaches address cost and extensibility but introduce new biases, including positional bias, verbosity bias, and self-preference (judges favoring outputs from their own model family).

Methodology: The CoEval Framework

CoEval is an open-source, declarative framework designed to generate a fresh, contamination-free evaluation pipeline and rank candidate models without human labels. It operates through a five-phase pipeline driven by a single configuration file, separating three distinct model roles: Teacher, Student, and Judge.

1. Role Separation

Teacher: Synthesizes domain-targeted benchmark items (prompts and reference responses) conditioned on declared target attributes.
Student: The candidate models under evaluation produce responses to the generated prompts.
Judge: Scores student responses against an automatically generated rubric.
Design Principle: The judge set is explicitly disjoint in vendor family from the student under test. This structural separation precludes a model from scoring its own family, eliminating the self-preference channel by design.

2. Attribute-Stratified Generation

Instead of uniform sampling, CoEval defines a set of target attributes (e.g., difficulty, topic, reasoning type) and stratifies generation across their cross-product. The sampler guarantees a minimum number of items per stratum ( $n_s \ge \lfloor N/|S| \rfloor$ ), ensuring edge cases and minority regions of the deployment distribution are covered. Because items are synthesized fresh from the attribute specification on every run, they are structurally resistant to contamination; they cannot have appeared in any student's pretraining data.

3. Cross-Family Judge Ensemble and Aggregation

Responses are scored by a panel of judges drawn from distinct model vendors (e.g., OpenAI, Anthropic, Google, open-weight models).

Composition over Size: The framework posits that reliability is driven by vendor diversity (composition) rather than panel size. A large panel of same-family judges amplifies shared biases, whereas a small, diverse panel cancels them.
Bias Cancellation: Individual judges often carry biases of mixed signs (e.g., one judge rewards verbosity while another penalizes it). Averaging scores across a vendor-diverse panel cancels these biases.
Aggregation: The default aggregator is an unweighted mean. CoEval also offers an unsupervised reliability-weighted aggregator that weights judges by their mean agreement with the panel consensus, down-weighting outliers without requiring ground truth.

Key Contributions

Self-Generating, Contamination-Resistant Benchmarks: CoEval generates fresh, attribute-controlled benchmarks per run, ensuring items are never seen during student pretraining.
Composition-Driven Reliability: The paper establishes that judge-panel composition (vendor diversity), not size, is the decisive factor for reliability. A small, well-chosen cross-family panel is more reliable than a large, homogeneous one.
Bias-Cancelling Aggregation: The framework structurally precludes same-family self-preference and empirically cancels verbosity bias through diverse panel aggregation.
Reusable, Label-Free Framework: It provides a declarative, model-agnostic pipeline that can be regenerated for any domain or model release at low cost, replacing static leaderboards with renewable, application-specific evaluations.

Empirical Results

The framework was validated across a four-task medium-scale study (summarization, QA, code generation, reasoning) involving 7,978 evaluations.

Ground-Truth Correlation: On exact-match QA tasks (SciQ, ARC-Challenge) where ground truth exists, CoEval's label-free scores track correctness with a Spearman correlation of $\rho = 0.86$ . It successfully reproduces the true model ranking (e.g., $gpt\text{-}4o\text{-}mini > gpt\text{-}3.5\text{-}turbo > llama\text{-}3.2\text{-}3b$ ).
Ensemble Reliability: Reliability (measured by ICC(3,k)) is non-monotone with panel size. A selected two-judge cross-family panel reached ICC = 0.70, while indiscriminately adding low-agreement judges reduced reliability to 0.40.
Judge-Choice Regret: Selecting a single judge without labels is risky; candidate judges ranged from anti-correlated ( $\rho = -0.04$ ) to positively correlated ( $\rho = 0.31$ ), a regret of 0.35. The cross-family ensemble is never anti-correlated.
Bias Cancellation: Individual judges exhibited verbosity biases ranging from $r = -0.177$ to $r = +0.234$ . The ensemble reduced the mean absolute bias by 93%, resulting in a length-score correlation of $r = +0.010$ (CI includes zero).
Contamination Resistance: CoEval-generated items showed zero verbatim 13-gram overlap with five major public benchmarks. A controlled study demonstrated that static benchmarks could rank a memorizing 0.5B model above a frontier model, whereas fresh CoEval items correctly ranked the frontier model higher.
Cost Efficiency: The entire study of 7,978 evaluations cost USD 5.89 (approx. $0.00074 per evaluation), making it feasible to re-run on every model release.
Domain Case Studies: Applied to three custom verticals (drug-interaction, clinical, legal) with no labeled data, CoEval produced defensible rankings where judges either unanimously agreed or surfaced genuine disagreements that a single judge would have hidden.

Significance and Claims

The paper claims CoEval solves the "no-data, untrusted-benchmark" predicament by providing a renewable, contamination-free, and bias-hardened evaluation instrument. Its primary significance lies in shifting the paradigm from static, human-curated benchmarks to dynamic, declarative pipelines.

The authors emphasize that composition over size is a fundamental design principle for LLM judging: a diverse, cross-family panel is the low-regret default that neutralizes self-preference and cancels mixed-sign biases without human calibration. By closing the loop from item generation to scoring, CoEval ensures that both the evaluation items and the judges are robust against the specific failure modes (memorization and bias) that plague current evaluation practices. The framework is presented not as a one-off benchmark, but as a reusable tool for teams to regenerate their own application-specific leaderboards.