Technical Summary: RubberDuckBench

Problem Statement

While AI coding assistants have revolutionized software engineering workflows—particularly in code generation, repair, and search—their ability to answer contextualized questions about existing code remains understudied. Current benchmarks for Large Language Models (LLMs) primarily focus on:

• Text-to-Code Generation: Synthesizing standalone functions from natural language descriptions (e.g., HumanEval, MBPP).
• Specific Development Tasks: Secure backend generation, full-stack web development, or code translation.
• Out-of-Context Queries: Benchmarks derived from Stack Overflow, which often address general language-level questions rather than project-specific reasoning.

There is a lack of evaluation frameworks that assess an LLM's ability to reason over specific project contexts, files, and line numbers to answer nuanced developer questions. Furthermore, existing tools often fail to distinguish between correct reasoning and hallucination in these specific scenarios.

Methodology

Benchmark Curation (RubberDuckBench)

The authors curated RubberDuckBench, a multilingual benchmark consisting of 15 contextualized questions derived from real-world GitHub pull request (PR) comments.

• Source Data: The dataset was mined from the CodeReview dataset, focusing on high-quality open-source repositories (average 25.3k stars) in Java, Python, and C++.
• Filtering & Rephrasing: PR comments were not used directly, as they often contain edit suggestions or team planning rather than queries. The authors employed an LLM-human annotation pipeline:
  1. An LLM (Claude Opus 4.1) rephrased comments into concise, unambiguous questions.
  2. The LLM determined suitability for an AI assistant.
  3. Human annotators verified and further refined the questions to ensure they were grounded in specific code elements.
• Question Taxonomy: The 15 questions are evenly split across languages and categorized into four types:
  1. Project Behavior: Functionality of in-project code.
  2. Library Behavior: Functionality of library/API code within the project context.
  3. Value: Propagation and values of program variables.
  4. Performance: Efficiency and performance considerations.

Evaluation Rubrics

Recognizing that viable responses are open-ended and not unique, the authors manually curated detailed rubrics for each question.

• Scoring Mechanism: A negative-scoring scheme was used, where responses start with a perfect score, and points are deducted for errors.
• Error Types: Rubrics heavily penalize hallucinations (fabricated facts) more than omissions (missing relevant information).
• Validation: Rubrics were developed through consensus among three authors, including an Assistant Professor, and refined using a "training set" of model responses to ensure comprehensive coverage.

Experimental Setup

• Models Evaluated: 20 diverse LLMs (11 reasoning, 4 low-cost, 12 proprietary, 8 open-source) from 8 providers (Anthropic, OpenAI, Google, xAI, Alibaba, Meta, Mistral, Deepseek).
• Configuration: Models were prompted with Chain-of-Thought (CoT) reasoning and standardized HTML-style tags. Proprietary models were run via API; open-source models were run locally on NVIDIA H100 GPUs.
• Protocol: Each model was evaluated three times per question at a temperature of 0.01. Scores were averaged.
• Human Evaluation: Three authors independently applied the rubrics, achieving high inter-rater reliability (ICC3 = .991).

Key Results

1. Performance on RubberDuckBench (RQ1)

• Overall Scores: The average score across all models was 60.17% (median: 61.30%).
• Top Performers: Grok 4 (69.29%), Claude Opus 4 (68.53%), and GPT-5 (67.80%) performed best. However, pairwise statistical analysis showed no significant superiority (p < .05) between Grok 4 and the next 12 best-performing models.
• Complete Correctness: Models struggled to provide fully correct answers. Under strict criteria (full credit across all three trials), the best models (Grok 4, Claude Opus 4) answered at most 2 out of 15 questions completely correctly. Most models answered 0 or 1.
• Language Bias: Models performed significantly worse on Python (50.44%) compared to Java (66.86%) and C++ (63.21%).
• Question Type: Models performed best on Library Behavior questions (63.7%) and worst on Project Behavior questions (55.0%).

2. Resource vs. Performance (RQ2)

• Cost Correlation: There was no correlation between API pricing and performance.
  • Grok 4 (top performer) cost $0.05 per query.
  • Claude Opus 4 (second best) cost $0.597 per query.
  • Claude Opus models were high-cost outliers with minimal performance gains over cheaper alternatives.
• Model Size Correlation: For open-source models, larger parameter counts did not correlate with higher scores. The best open-source model, gpt-oss-20B (20B parameters), outperformed gpt-oss-120B (120B parameters).

3. Hallucination Rates (RQ3)

• Frequency: On average, models hallucinated (provided lies) in 58.3% of responses.
• Top Performers: Even high-performing models like o3 hallucinated in 67% of their answers (10 out of 15).
• Least Hallucinating: Grok 4 hallucinated the least (40%), though it still hallucinated in 6 out of 15 questions.

Significance and Claims

The paper positions RubberDuckBench as a target for future research into trustworthy and correct AI coding assistants. The authors claim:

1. Gap Identification: Current benchmarks fail to evaluate the critical use case of answering contextualized questions about code, a primary use case identified in the Stack Overflow 2025 Developer Survey.
2. State of the Art Limitations: Even state-of-the-art models fail to provide consistent, correct responses to contextualized code questions, often relying on partial credit and frequently hallucinating.
3. Cost/Size Inefficiency: The findings challenge the assumption that higher cost or larger model size guarantees better performance in code reasoning tasks.
4. Reproducibility: The authors provide a reproducible evaluation package, including scripts to clone the necessary project contexts and the detailed rubrics used for evaluation.

The authors acknowledge limitations, including potential bias introduced by the LLM-assisted rephrasing of PR comments and the possibility that the specific open-source projects used may exist in training sets (though the specific contextualized questions were not). They emphasize that the benchmark is small (15 questions) by design to function as a deep case study rather than a massive-scale evaluation.