Hallucination Inspector: A Fact-Checking Judge for API Migration

1. Problem Statement

The paper addresses a critical failure mode in Large Language Model (LLM)-assisted software engineering: Scaffolding Hallucination during API migration.

• Context: While LLMs are effective at identifying target APIs for migration (e.g., moving from a deprecated Android API to a new one), they frequently fail to generate the correct "glue code" required to instantiate objects, chain method calls, or invoke constants.
• The Issue: LLMs often invent Phantom Symbols—imports, constructors, constants, or methods that do not exist in the target API's official documentation.
• Detection Failure:
  • Standard Metrics: Metrics like CodeBLEU are unreliable because hallucinated code often looks syntactically and textually similar to the ground truth, yielding high similarity scores despite being un-compilable.
  • LLM-as-a-Judge: Using another LLM to verify code is prone to "optimism bias" (accepting broken code) and "over-correction bias" (rejecting valid code), making it an unreliable evaluator.

2. Methodology: The Hallucination Inspector

The authors propose Hallucination Inspector, a lightweight, deterministic, static analysis framework designed to fact-check LLM-generated code against a trusted API specification (Oracle).

Core Architecture

The system operates in three sequential modules:

1. Parser: Converts the LLM-generated code snippet into an Abstract Syntax Tree (AST) using the Tree-sitter library.
2. Symbol Extractor: Isolates testable units from the AST, including atomic symbols (constants, static fields) and method chains (fluent API usage).
3. Verification Engine: Cross-references extracted symbols against a Deterministic API Oracle.
   • The Oracle: A ground-truth symbol table derived directly from official Android SDK XML documentation. It contains a pre-computed hash table of valid classes, methods, signatures, and inheritance hierarchies.

Two-Phase Verification Strategy

The tool detects two distinct categories of hallucinations:

• Phase 1: Atomic Hallucinations (Direct Lookup)
  • Checks for the existence of atomic symbols (e.g., constants, static methods) directly in the API namespace.
  • Example: Detecting a constant like CONTENT_TYPE_STREAM that is not defined in the documentation.
• Phase 2: Scope-bound Hallucinations (Type Propagation)
  • Analyzes method chains to ensure methods are called on the correct object types.
  • The engine resolves the type of the receiver object (e.g., determining if a variable is an AudioAttributes.Builder or an AudioManager) and verifies if the subsequent method call exists for that specific type.
  • Example: Detecting a call to setStreamType() on an AudioAttributes.Builder object, when that method only exists on the legacy AudioManager.

Algorithmic Logic

The verification follows a sequential validation algorithm (Algorithm 1):

1. Resolve the root object type.
2. Iterate through the method chain.
3. Check if the method exists in the Oracle for the current object type.
4. If missing, flag as a Phantom Member.
5. If the method returns void but is followed by another call in the chain, flag as a Broken Chain.
6. Update the current type to the method's return type for the next iteration.

3. Key Contributions

1. Definition of Scaffolding Hallucination: A formal definition distinguishing between Atomic Hallucinations (invented symbols) and Scope-bound Hallucinations (invented relationships/methods on specific object types).
2. Hallucination Inspector Tool: A static analysis tool that uses an API Oracle to deterministically verify code, eliminating the probabilistic biases of LLM judges.
3. Empirical Evidence: A demonstration that standard similarity metrics (CodeBLEU) fail to detect these hallucinations, while the proposed tool achieves high precision.

4. Experimental Results

The authors evaluated the tool on a dataset of 51 Android API migration pairs using two LLMs (Qwen-2.5-32b and Codestral-v1-22b).

• RQ1: Blind Spot of Metrics:
  • CodeBLEU scores for hallucinated patches were nearly identical to valid patches (Median scores: ~0.84 for hallucinated vs. ~0.82 for valid in Qwen-2.5).
  • Hallucinated code could achieve a similarity score of 0.96, proving that high similarity does not guarantee correctness.
• RQ2: Detection Efficacy (Inspector vs. LLM-as-a-Judge):
  • Hallucination Inspector: Achieved 100% Precision (0 false positives) and high Recall (0.73–0.90) across both models. It successfully identified all phantom symbols.
  • LLM-as-a-Judge (gpt-5-mini): Suffered from severe reliability issues.
    • False Positives: High over-correction bias (e.g., rejecting valid code 50% of the time for Codestral).
    • False Negatives: Failed to detect actual hallucinations (Recall ~0.56–0.67).
  • Conclusion: The deterministic Oracle is significantly superior to probabilistic LLM judges for this specific task.

5. Significance and Future Work

• Significance: The paper establishes that for API migration, deterministic verification is superior to probabilistic evaluation. It provides a practical solution to a "silent killer" in automated software engineering where code looks correct but fails to compile or run due to invented symbols.
• Future Directions:
  • Extending the Oracle to handle argument type mismatches and callback signatures more robustly.
  • Integrating the tool as a "gatekeeper" in iterative repair loops (e.g., using static linting results as reward signals for Reinforcement Learning).
  • Expanding the evaluation to multi-language benchmarks and diverse ecosystems beyond Android/Java.

In summary, Hallucination Inspector offers a rigorous, documentation-grounded approach to ensuring the fidelity of LLM-generated code, solving a critical gap where current evaluation metrics and LLM judges fail to detect subtle but fatal structural errors.