Technical Summary: DELULU – A Verified Multi-Lingual Benchmark for Code Hallucination Detection in Fill-in-the-Middle Tasks

Problem Statement

Large Language Models (LLMs) used for code generation frequently produce hallucinations in Fill-in-the-Middle (FIM) tasks. These are completions that are syntactically valid and semantically plausible but factually incorrect, such as invented API methods, invalid parameters, undefined variables, or non-existent imports. While these errors often pass superficial code review, they introduce latent runtime bugs that manifest only under specific execution conditions.

Current benchmarks for code generation (e.g., HumanEval, MBPP) primarily focus on function-level generation from natural language specifications or lack execution verification for FIM tasks. Existing FIM benchmarks (e.g., HumanEval Infilling, SAFIM) are often Python-only, do not distinguish between error types, and lack execution-based ground truth to verify that hallucinations actually cause runtime failures. Furthermore, there is no systematic evaluation of whether frontier LLMs can reliably detect these hallucinations when acting as code reviewers.

Methodology: The DELULU Benchmark

The authors introduce DELULU, a verified multi-lingual benchmark comprising 1,951 FIM samples across 7 programming languages (Python, Java, TypeScript, Go, Rust, C#, C++) and 4 hallucination categories. The benchmark is constructed through a rigorous, five-stage adversarial pipeline designed to ensure difficulty, correctness, and execution verification:

1. Hallucination Generation: Using CLAUDE SONNET as the sole generator, the system creates "hallucinated" variants of golden completions mined from real GitHub repositories. The generator is constrained to modify exactly one element (method name, parameter, variable, or import) while keeping the rest of the code identical, ensuring the hallucination is plausible and indistinguishable from real code on the surface.
2. Multi-Model Judge Evaluation: Four diverse frontier LLMs (GPT-5.1, GPT-5.2, GPT-5.2-CODEX, GLM-4.7) independently evaluate the pairs.
3. Iterative Adversarial Sampling: The pipeline analyzes judge disagreements and uses embedding-based clustering to mine new, semantically similar code examples that resemble the failure modes of the judges. This loop runs for three iterations, progressively surfacing harder samples as judge accuracy decreases over time.
4. Difficulty-Based Selection: Samples are scored based on their ability to fool judges. Only samples that are difficult enough to fool at least one judge (or cause over-rejection of golden code) are retained.
5. Docker-Based Execution Verification: This is the critical differentiator. Every sample is packaged into a self-contained Docker container.
   • The golden completion must compile and run without errors.
   • The hallucinated completion must produce a specific, verifiable runtime error corresponding to its category:
     • Method: AttributeError
     • Parameter: TypeError
     • Undefined Variable: NameError
     • Import: ImportError
   • A CLAUDE SONNET-assisted fix loop resolves dependency issues to ensure verification success.
6. Human Expert Review: A panel of three human experts reviews the Docker-verified samples to remove residual bias, trivial cases, or labeling errors, resulting in the final dataset of 1,951 samples.

Key Contributions

1. Four-Category Hallucination Taxonomy: A structured taxonomy where each category (Method, Parameter, Undefined Variable, Import) maps to a distinct, verifiable runtime error class, enabling execution-based ground truth.
2. Adversarial Curation Pipeline: A novel five-stage process that pairs real-world code with hallucinated counterparts, filters trivial cases via multi-model probing, and ensures execution-based verification.
3. Execution-Verified Multi-Lingual Dataset: 1,951 samples across 7 languages, each packaged as a self-contained Docker container with three invocation modes (verify golden, verify hallucinated, verify patch).
4. Comprehensive Evaluation: An evaluation of 11 open-weight FIM models from 5 families (Qwen2.5-Coder, CodeLlama, DeepSeek-Coder-V2, StarCoder2) spanning 0.5B to 32B parameters.
5. Frontier-Judge Detection Study: An evaluation of 8 frontier LLMs acting as code reviewers to detect hallucinations, revealing significant gaps in detection capabilities.

Results

Generation Performance

• Scaling Limits: The strongest model, Qwen2.5-Coder-32B, achieved only 84.5% pass@1 on the 950 unique FIM contexts. This indicates that even at 32B parameters, models fail on roughly one in seven unique contexts.
• Cross-Family Consistency: The difficulty is intrinsic to the task, not specific to a model family. No cross-family model exceeded 0.77 Edit Similarity. Even the best cross-family model (DeepSeek-Coder-V2-Lite-Instruct) trailed the best Qwen model by 1.8 pass@1 points.
• Hallucination Types: Import hallucinations were consistently the most difficult category across all model sizes and languages (e.g., 77.1% pass@1 for Qwen-32B on imports vs. >85% on other types), suggesting that detecting fake imports requires specific ecosystem knowledge rather than just local pattern completion.
• Instruction Tuning: Instruction tuning significantly improved "stop" behavior (Exact Match) but had minimal impact on the underlying knowledge required to avoid hallucinating identifiers (Edit Similarity and CodeBLEU remained similar between base and instruct models).

Detection Performance

• Unsolved Detection: Even the strongest judge model, Claude-4.5-Opus, achieved only 92.1% both-correct accuracy (correctly accepting golden and rejecting hallucinated).
• Detection Gap: Smaller judges performed significantly worse (e.g., GPT-4o-mini at 52.3%), establishing a 40-point capability range.
• Import Bottleneck: Detection of import hallucinations remained the hardest task for all judges (83.4% for Opus), confirming that distinguishing real from fake package names is a persistent capability gap.
• False Negatives: The primary bottleneck for judges was rejecting hallucinated completions, not accepting golden ones.

Significance and Claims

The paper claims that DELULU addresses a critical gap in code evaluation by providing the first benchmark that simultaneously offers:

1. Systematic hallucination categorization with fine-grained types.
2. Multi-lingual coverage beyond Python.
3. Execution-based verification ensuring hallucinations produce real runtime errors.
4. Adversarial difficulty calibration ensuring the benchmark is genuinely challenging.

The authors emphasize that the difficulty exposed by DELULU is task-intrinsic rather than family-specific, as evidenced by the consistent failure rates across diverse model architectures. The results suggest that simply scaling model parameters is insufficient to solve code hallucination; targeted interventions such as training-data curation and verifier-style fine-tuning are likely necessary.

The paper positions DELULU not just as a generator benchmark, but as a reusable tool for evaluating code-review and verifier models, highlighting that the ability to detect hallucinations is currently an unsolved problem even for frontier models. The benchmark is released as a test-only artifact to prevent contamination, with the authors explicitly discouraging fine-tuning on the dataset.