Technical Summary: Tricky2 – Towards a Benchmark for Evaluating Human and LLM Error Interactions

Problem Statement

While Large Language Models (LLMs) are increasingly integrated into software development workflows (e.g., GitHub Copilot), existing benchmarks for software testing and debugging evaluate human-generated and AI-generated errors in isolation. Current datasets, such as TrickyBugs (human defects) or buggy-HumanEval (LLM defects), fail to capture the reality of modern development where human-written and LLM-generated code coexist within the same files and functions.

The paper identifies a critical gap: existing benchmarks cannot evaluate error interactions. In hybrid contexts, human fixes may mask LLM faults, LLM repairs may reintroduce human defects, and standard evaluation metrics may fail when multiple error sources compound or obscure one another. Consequently, the software engineering community lacks the ability to rigorously evaluate repair tools in realistic, mixed-origin settings or to study the reliability of debugging in hybrid human–AI collaboration.

Methodology

To address this, the authors introduce Tricky2, a hybrid dataset that extends the existing TrickyBugs corpus. The construction pipeline involves the following steps:

1. Base Dataset: The foundation is TrickyBugs, containing 3,043 human-written buggy programs and 1,361 fixed programs across C++, Python, and Java, derived from competitive programming submissions.
2. Controlled Bug Injection: The authors employ two LLMs, GPT-5 and OpenAI-oss-20b, to inject additional errors into the existing programs.
   • Prompting Framework: A structured, taxonomy-guided prompt instructs the models to inject exactly one new bug from a predefined taxonomy (Input/Output, Variable/Data, Logic/Condition, Loop/Iteration, Function/Procedure) while strictly preserving all original human defects and program structure.
   • Constraints: The models are instructed to avoid comments or formatting changes and to output the modified file with the taxonomy level and character positions of the injected code.
3. Dataset Splits: The process generates three distinct splits:
   • Human-only: The original buggy programs.
   • LLM-only: Fixed programs with LLM-injected errors (no human bugs).
   • Human+LLM: Buggy programs containing both the original human defects and the newly injected LLM errors.
4. Validation: An automated pipeline validates syntax and compilation using language-specific tools (g++, py_compile, javac). The final corpus contains 11,851 buggy programs, with all splits sharing identical test cases and fixed references.

Key Contributions

• Tricky2 Benchmark: The first controlled benchmark explicitly modeling the co-occurrence of human and LLM errors within shared program contexts.
• Taxonomy-Guided Injection: A reproducible pipeline using a specific error taxonomy to generate machine-originated bugs that interact with human defects without altering the original code structure.
• Three Evaluation Tasks: The paper defines three complementary tasks to measure reliability in hybrid settings:
  1. Origin Classification: Distinguishing between human, LLM, and mixed defects.
  2. Error Identification: Localizing bug spans and identifying the taxonomy level.
  3. Program Repair: Evaluating the success of minimal patches against provided test cases.
• Baseline Evaluations: Small-scale proof-of-concept evaluations on a stratified subset of 135 difficult problems (45 per split) across C++, Java, and Python.

Results

The baseline evaluations, while limited in scope, reveal distinct patterns regarding mixed-origin errors:

• Repair Difficulty: Both human-only and LLM-only splits demonstrated higher repair success rates compared to the Human+LLM split.
• Interaction Effects: The results suggest that mixed-origin defects introduce interaction effects that complicate reasoning and repair. Notably, in the Human+LLM C++ split, the model failed to produce a single fixed program, despite successfully correcting substantial subsets of the single-origin cases.
• Classification and Localization: The paper reports confusion matrices and heatmaps for origin classification and taxonomy identification, though specific accuracy percentages are not the primary focus of the summary; the emphasis is on the divergence in repair outcomes.

Significance and Claims

The paper positions Tricky2 as an early but necessary step toward evaluating software reliability in hybrid human–AI development environments. The authors claim that:

• Interaction Effects are Real: Mixed-origin defects are not merely the sum of their parts; they create compounding or masking effects that current single-source benchmarks cannot detect.
• Current Tools Struggle: Existing LLMs appear to struggle significantly with the complexity of debugging code where human and AI errors interact, highlighting a gap in current repair capabilities.
• Foundation for Future Work: Tricky2 serves as a foundation for studying robustness, error-aware collaboration, and the design of safer human–AI coding workflows. The authors envision future extensions involving agentic debugging protocols, longitudinal studies of model reliability, and expansion to additional languages and model families.

The paper explicitly acknowledges limitations, including the reliance on specific LLM versions (which may drift over time), the constraint of the underlying TrickyBugs domain (competitive programming vs. industrial systems), and the lack of coverage for complex semantic errors like concurrency or memory management. However, it asserts that the dataset provides a controlled baseline for investigating error interactions that were previously unmeasurable.