When LLMs Lag Behind: Knowledge Conflicts from Evolving APIs in Code Generation

1. Problem Statement

Large Language Models (LLMs) are trained on static corpora with fixed knowledge cutoffs. As software libraries evolve continuously (e.g., API deprecation, modification, or addition), the parametric knowledge encoded in LLM weights becomes stale.

• The Core Conflict: When developers provide up-to-date API specifications via Retrieval-Augmented Generation (RAG) or prompt context, a "context-memory conflict" arises. The model must choose between following the external, up-to-date instructions or relying on its internal, memorized (but outdated) parametric knowledge.
• The Gap: It is unclear how reliably LLMs can suppress their internal knowledge to adopt new API specifications, particularly when the external context contradicts their training data. Existing benchmarks often assume the required knowledge exists within the model's training distribution, failing to test post-training knowledge gaps.

2. Methodology

The authors conducted a systematic empirical study involving three main phases:

A. Dataset Construction (The Benchmark)

• Scope: 270 real-world API updates across 8 popular Python libraries (NumPy, Pandas, scikit-learn, SciPy, JAX, Keras, TensorFlow, PyTorch).
• Temporal Constraint: All updates were introduced after December 2023, ensuring they fall outside the knowledge cutoff of the evaluated models.
• Update Patterns:
  1. P1 (Deprecation): APIs removed/deprecated with a recommended alternative.
  2. P2 (Modification): Changes to function names, parameters, or return types.
  3. P3 (Addition): Newly introduced APIs.
• Data Curation: Automated extraction using GPT-5-mini followed by rigorous manual curation to ensure high-quality, unambiguous specifications.

B. Experimental Setup

• Models Evaluated: 11 models across 4 families (DeepSeek-Coder, CodeLlama, DeepSeek-R1-Qwen, GPT-4o-mini) with varying parameter sizes (1.3B to 33B+).
• Prompting Strategies:
  1. UD (Update Description): Text-only description of the change.
  2. UD + Doc: Description + Full structured API documentation.
  3. Reasoning Techniques:
     • Chain-of-Thought (CoT): Explicit reasoning before code generation.
     • Self-Reflection (SR): Post-generation verification and correction loop.
• Evaluation Metrics:
  1. API Adoption Rate: Measures if the model attempts to use the new API (coarse filter).
  2. Executable Rate: Measures if the generated code actually runs in the target environment (strict filter). Note: Executable Rate is calculated only on code that passed the Adoption filter.

3. Key Contributions

1. First Systematic Benchmark: Created a dataset of 270 post-training API updates to evaluate LLMs under context-memory conflict.
2. Comprehensive Evaluation: Tested 11 models across different scales and families to analyze the impact of model size and architecture on handling evolving APIs.
3. Reasoning Strategy Analysis: Investigated the efficacy of CoT and Self-Reflection in resolving knowledge conflicts.
4. Failure Taxonomy: Categorized specific failure modes at both the adoption and execution levels.

4. Key Results

RQ1: Compliance with External Specifications

• Without Documentation: LLMs struggle significantly. Average Adoption Rate is 74.64%, and Executable Rate is only 42.55%. Models often ignore updates or hallucinate interfaces.
• With Documentation: Adding structured API docs significantly boosts performance. Adoption rises to 92.87%, and Executable Rate improves to 66.36%.
• Pattern Difficulty: API Modification (P2) is the most challenging pattern (lowest adoption and execution rates), even with documentation, likely due to subtle parameter changes conflicting with strong prior knowledge.

RQ2: Model Family and Scale

• Performance Variance: GPT-4o-mini performed best overall. Among open-source models, DeepSeek-Coder led in executability, while CodeLlama and DeepSeek-R1 struggled more with execution despite high adoption.
• Scaling Laws: Increasing parameter size generally improves Adoption Rate but does not consistently resolve Executability issues. Larger models still fail to generate runnable code for complex modifications.

RQ3: Impact of Reasoning Techniques

• Marginal Adoption Gain: CoT and Self-Reflection (SR) provided only slight improvements to Adoption Rate (~1.5% - 2%).
• Significant Execution Gain: These techniques substantially improved Executable Rate by ~11.33%.
• Self-Reflection Dominance: SR contributed more than CoT, particularly for P2 (Modification) and P3 (Addition). SR acts as a verification loop, catching structural errors and hallucinations that CoT misses.

RQ4: Failure Analysis

• Adoption Failures (L1):
  • Omission (42.1%): Ignoring the update entirely.
  • Old API Usage (16.4%): Reverting to deprecated methods.
  • Hallucination (P3): Inventing non-existent APIs.
• Execution Failures (L2):
  • Wrong Parameters (26.6%): Using incorrect arguments or types.
  • Hallucinated Behavior (16%): Correctly calling the API but assuming wrong return types or method chains.
  • Insight: Over half of execution failures are due to incorrect API usage, not unrelated syntax errors.

5. Significance and Implications

• Documentation is Critical but Insufficient: While providing API documentation is the single most effective factor for improving performance, it does not guarantee executable code.
• Self-Reflection is Essential: For tasks involving API evolution, Self-Reflection prompting is highly recommended to bridge the gap between "adopting" an API and "correctly using" it.
• Tooling Recommendations: IDE plugins and RAG systems must automatically inject structured, up-to-date documentation. Relying on developer manual input or model memory is insufficient.
• Future Benchmarks: The community needs evolution-aware benchmarks that track API changes post-training cutoffs to accurately assess LLM reliability in dynamic software ecosystems. Current benchmarks (e.g., HumanEval) fail to capture these knowledge conflicts.

Conclusion: LLMs exhibit a persistent tendency to rely on outdated parametric knowledge even when explicitly provided with correct, external specifications. While documentation and reasoning strategies (specifically Self-Reflection) mitigate this, they do not fully eliminate the risk of generating non-executable code, highlighting a critical need for evolution-aware development practices.