💻 computer science

LLM-as-a-Judge for Human-AI Co-Creation: A Reliability-Aware Evaluation Framework for Coding

This paper presents a reliability-aware, rubric-driven LLM-as-a-Judge framework for evaluating human-AI co-creation in coding, which combines rigorous multi-metric judge assessment with trajectory-level analysis to reveal that co-creation success typically concentrates early while revision behaviors remain heterogeneous.

Original authors: Md Faizul Ibne Amin, Yutaka Watanobe, Daniel M. Muepu, Haruto Suzuki, Kenta Nanaumi, Md Mostafizer Rahman

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Md Faizul Ibne Amin, Yutaka Watanobe, Daniel M. Muepu, Haruto Suzuki, Kenta Nanaumi, Md Mostafizer Rahman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a high-stakes coding competition where participants are allowed to use AI assistants to help them solve problems. This paper is like a report card for two things happening in that room: how well the humans and AI worked together, and how good the "referees" (AI judges) were at grading the work.

Here is the breakdown of what the researchers did, using simple analogies:

1. The Setup: The "AI-Allowed" Coding Contest

Usually, in coding contests, using AI is cheating. But here, the researchers ran a special track where 15 participants tried to solve 13 difficult problems using AI tools of their choice.

The Goal: They wanted to see not just if the code worked, but how the humans and AI figured it out together. Did they get stuck? Did they fix small errors, or did they throw the whole thing away and start over?
The Problem: Traditional grading only looks at the final answer (Pass/Fail). It's like a teacher only looking at the final essay and ignoring the messy drafts, the crossed-out sentences, and the notes in the margins. The researchers wanted to grade the process, not just the product.

2. The Solution: The "Rubric-Driven" AI Judge

To grade these messy, multi-step processes, the researchers couldn't just ask a human to read thousands of logs. So, they built a system using AI Judges (like OpenAI, DeepSeek, Gemini, and Claude) to act as referees.

Think of this like a panel of sports referees:

The Rules: Instead of letting the AI judges write free-form opinions (which can be messy and inconsistent), the researchers forced them to fill out a strict scorecard (a "schema"). They had to give specific scores for things like "Is the logic sound?" and "Did it handle edge cases?"
The Safety Net: Since AI sometimes makes mistakes or gives weird answers, the system had a "repair mechanism." If an AI judge forgot to fill out a box on the scorecard, the system would catch it and ask the AI to try again until the scorecard was perfect.
The Context: The judges were allowed to see the participant's entire history of prompts (what they asked the AI) before grading a specific piece of code. This is like a referee watching the whole game replay before calling a foul, rather than just looking at one split-second frame.

3. The Findings: How the Humans and AI Worked Together

The researchers analyzed the "trajectories" (the step-by-step journey) of the participants.

The "Early Bird" Effect: They found that success happens very early. Imagine a race where 85% of the runners cross the finish line within the first few steps. Once a participant and their AI figured out the right path, they usually solved it quickly. If they were still struggling after the first few tries, they rarely succeeded later.
Two Ways to Fix a Car: When the code was wrong, participants fixed it in two very different ways:
1. The Mechanic: Tweaking small parts (incremental refinement).
2. The Architect: Scraping the whole design and rebuilding it (broad restructuring).
- The Surprise: Both methods worked equally well. There was no "best" way to fix code; sometimes a small tweak worked, and sometimes a total rebuild was needed.

4. The Findings: How Good Were the AI Judges?

The researchers tested four different AI models to see which one was the best referee.

Different Strengths: Just like human referees, the AI judges had different personalities.
- DeepSeek was the best at ranking the good attempts above the bad ones (like spotting the best plays).
- OpenAI was good at being precise with its probability scores.
- Gemini and Claude had their own quirks.
The "Agreement" Problem: The judges didn't always agree with each other. If you asked two different AI judges to grade the same code, they might give different scores. The researchers found that while they agreed some of the time, they often disagreed.
The Lesson: You can't rely on just one AI judge. You need a "panel" of them, and you have to look at a bunch of different metrics (like how well they rank, how well they calibrate their confidence, and how often they agree) to get a true picture of quality.

5. The Big Takeaway

This paper introduces a new playbook for evaluating how humans and AI work together.

For the Process: It shows that in AI-assisted coding, success usually happens early, and there is no single "correct" way to fix a bug.
For the Grading: It proves that AI can be a reliable referee if you force it to follow strict rules, check its work, and use multiple judges to cross-check the scores.

In short: The paper built a better way to grade the messy, collaborative dance between humans and AI, showing us that success often happens quickly, and that we need a team of AI referees to get the score right.

1. Problem Statement

The paper addresses a critical gap in the evaluation of Human-AI Co-Creation in coding and software engineering (SE). While Large Language Models (LLMs) are increasingly used as both co-creation partners and evaluators ("LLM-as-a-Judge"), existing evaluation protocols are often insufficient for this specific context:

Limitations of Legacy Metrics: Traditional coding evaluation relies on final outcomes (e.g., pass/fail test cases), which fail to capture the value of intermediate artifacts, partial solutions, debugging hypotheses, and the iterative reasoning process inherent in co-creation.
Reliability and Bias: LLM judges are prone to biases (e.g., favoring longer responses, position bias) and inconsistencies. There is a lack of standardized, auditable frameworks that ensure judge outputs are reliable, comparable across different models, and interpretable over multi-turn interactions.
Data Leakage Risks: In co-creation settings, multiple attempts by the same user on the same problem create "trajectories." Standard random splitting can lead to data leakage, where the judge sees information from a later turn during the evaluation of an earlier turn.

2. Methodology

The authors propose a unified, rubric-driven LLM-as-a-Judge framework designed specifically for contest-style human-AI co-creation. The methodology consists of three main stages:

A. Data Processing and Dataset Construction

Source: Data was collected from the AI-permitted co-creation track of the PC Koshien (PCK) Finals, where participants solved the same problems as the official contest but were allowed to use AI tools.
Trajectory Grouping: Data is organized into trajectories defined by (participant, problem) pairs to prevent leakage.
NONBLIND Context: To simulate realistic evaluation, judges are provided with a participant-level aggregated context ( $Q_u$ ), which concatenates all prompt logs for that participant. This ensures the judge has the full history of the user's interaction when evaluating any specific attempt.
Schema Constraints: Judges are forced to output structured JSON rather than free-form text, containing:
- $p_a$ : Predicted probability of acceptance.
- $s_{algo}$ : Ordinal score (1-5) for algorithmic adequacy.
- $s_{robust}$ : Ordinal score (1-5) for robustness/constraint satisfaction.
- $r$ : Short rationale.

B. LLM-as-a-Judge Inference Pipeline

Multi-Model Evaluation: Four distinct LLM judges were evaluated: OpenAI (gpt-5.2), DeepSeek (reasoner), Gemini (2.5-pro), and Claude (sonnet-4).
Verification and Repair: The pipeline includes a robust verification step. If an API call fails or returns malformed JSON, the system retries with backoff. Only outputs satisfying strict schema constraints (e.g., probability in [0,1], valid ordinal scores) are accepted.
Splitting Strategy: A grouped split (Train/Val/Test) is applied at the (participant, problem) level.
- Validation: Used to select the optimal decision threshold ( $t^*$ ) by maximizing the Matthews Correlation Coefficient (MCC).
- Test: Used exclusively for final reporting to ensure no data leakage.

C. Evaluation Metrics

The framework employs a multi-metric protocol to assess both the judges and the co-creation dynamics:

Judge Quality Metrics:
- Discrimination: ROC-AUC and PR-AUC (ranking ability).
- Probabilistic Reliability: LogLoss, Brier Score, and Expected Calibration Error (ECE).
- Decision Quality: MCC using the threshold selected on the validation set.
- Inter-Judge Agreement: Cohen's $\kappa$ (pairwise) and Fleiss' $\kappa$ (multi-rater).
Co-Creation Dynamics Metrics:
- Success@Turn: Cumulative fraction of trajectories solved by turn $k$ .
- Time-to-Success: Analyzed via Kaplan-Meier survival analysis (handling right-censoring for unsolved trajectories).
- Revision Behavior: Measured using Normalized Edit Distance (NED) for surface-level changes and CodeBLEU for code-aware similarity/convergence.

3. Key Contributions

Reliability-Aware Framework: Introduces a rigorous pipeline for LLM judging that includes schema-constrained outputs, automated repair mechanisms, and grouped splitting to prevent trajectory leakage.
Multi-Metric Evaluation Suite: Moves beyond single-metric reporting by jointly analyzing discrimination, calibration, decision quality, and inter-judge agreement.
Trajectory-Level Analysis: Provides a novel method for analyzing human-AI interaction patterns (struggle curves, survival analysis) rather than just final outcomes.
Empirical Bridge: Connects the fields of "LLM-as-a-Judge" reliability and "Human-AI Co-Creation" dynamics within a single, reproducible empirical setting.

4. Key Results

A. Judge Performance

Discrimination: DeepSeek and Claude achieved the highest ROC-AUC (0.5937), with DeepSeek leading in PR-AUC (0.6904). Gemini performed the weakest (ROC-AUC 0.4250).
Calibration: DeepSeek showed the best calibration (lowest ECE: 0.2819), while Gemini and Claude exhibited higher errors.
Decision Quality: DeepSeek achieved the highest thresholded MCC on the test set (0.5000) with a conservative threshold (0.81).
Inter-Judge Agreement: Consistency was modest overall.
- Mean pairwise Cohen's $\kappa$ : 0.1592.
- Fleiss' $\kappa$ : 0.0696.
- Implication: Judges are not interchangeable; they exhibit distinct biases and strengths, reinforcing the need for multi-metric reporting rather than relying on a single model.

B. Human-AI Co-Creation Dynamics

Front-Loaded Success: Success is heavily concentrated in the early turns.
- Success@Turn reached 0.8533 at the first observed turn.
- By turn 6, it only increased to 0.8641.
- Kaplan-Meier analysis confirms that trajectories extending beyond the first few turns show diminishing returns.
Heterogeneous Revision Patterns: There is no single "correct" revision style.
- Both incremental refinement (small changes) and broad restructuring (large changes) led to successful outcomes.
- Code-aware similarity (CodeBLEU) showed that successful trajectories tend to converge structurally toward the accepted solution, regardless of the magnitude of intermediate changes.
Prompt-Code Dissimilarity: High NED values between prompts and code confirmed that the judge context (natural language) and the solution (code) are lexically distinct, validating the non-trivial nature of the evaluation task.

5. Significance and Implications

Beyond Final Correctness: The study demonstrates that evaluating AI-assisted coding requires looking at the process (trajectory), not just the product. Early success patterns suggest that effective co-creation relies on rapid iteration and verification rather than prolonged trial-and-error.
Judges as Instruments, Not Oracles: The results highlight that LLM judges are measurement instruments with specific "blind spots" and strengths. No single metric (e.g., accuracy) captures their full utility; a multi-dimensional view is essential for auditable research.
Human Expertise Remains Central: The findings suggest that AI does not replace human judgment but reorients it. The ability to evaluate intermediate outputs, recognize stagnation, and steer revisions remains the critical human skill in co-creation.
Reproducibility: The proposed framework offers a template for future studies to evaluate AI-assisted coding with transparency, ensuring that evaluation protocols are robust against data leakage and model-specific biases.

In conclusion, this paper provides a foundational methodology for rigorously evaluating how humans and AI collaborate to solve complex coding problems, while simultaneously establishing a reliability-aware standard for using LLMs as judges in such dynamic environments.