Technical Summary: LibEvoBench

Problem Statement

Modern software development relies heavily on third-party libraries that evolve independently, often resulting in production codebases pinned to older versions while APIs continue to change. This creates a "version-specific development context" where the correctness of an API call depends not on the library in general, but on the exact version specified in the project's dependency manifest.

Current Large Language Models (LLMs) are trained on temporally mixed corpora that aggregate code and documentation from multiple library versions simultaneously. The training objective compresses this heterogeneous material into a single set of weights without explicit mechanisms for temporal stratification. Consequently, LLMs struggle with version-specific reasoning, leading to anachronistic errors: generating code that is valid for a different library version than the one requested. While prior work has addressed temporal knowledge in general factual domains, version-aware code generation poses a distinct challenge: models must distinguish among multiple coexisting versions of the same API to avoid producing code that is syntactically valid but semantically incorrect for the target environment.

Methodology

LibEvoBench Benchmark

The authors introduce LibEvoBench, a multi-task benchmark designed to evaluate version-aware API knowledge across widely used Python libraries (PyTorch, NumPy, and SciPy). The benchmark spans 29 library versions and includes over 125,000 evaluation instances.

The benchmark construction pipeline is fully automated:

1. Data Collection: It mines public GitHub repositories with pinned requirements.txt files to gather real-world code contexts.
2. API Surface Reconstruction: It reconstructs the public API surface for each version using two complementary sources: official Sphinx documentation inventories and direct runtime introspection. An API is considered present only if both sources agree, creating a deterministic ground truth.
3. API Classification: Based on compatibility matrices, APIs are classified as Stable (unchanged signature across versions) or Evolving (added, removed, or modified).

Evaluation Tasks

LibEvoBench evaluates three distinct modalities of knowledge:

1. API Calling (API-C): Given an incomplete code snippet, the model must generate the missing API call. This task uses four noise-reduction levels (L0–L3) to isolate different capabilities:
   • L0: Raw code context only.
   • L1: Code + redacted docstring (isolates recall from retrieval).
   • L2: Code + docstring + explicit version constraint (tests version awareness).
   • L3: API name + version (tests fine-grained parameter recall).
2. API Identification (API-I): Given a module path and a redacted description, the model must identify the correct API name. This isolates declarative knowledge from compositional reasoning.
3. Signature Recall (SR): Given only the fully qualified API name and a target version, the model must reproduce the complete function signature from memory. This probes parametric knowledge of arguments, defaults, and ordering.

Hallucination Taxonomy

A novel deterministic classification system is introduced to categorize model errors based on the version-indexed API surface:

• Correct: Matches the ground truth at the target version.
• Unknown: The predicted symbol never appeared in any observed version (fabrication).
• Invalid: The symbol exists at the target version but is not the intended one (retrieval confusion).
• Anachronistic: The symbol exists in the library's history but not at the requested version (temporal confusion). This category specifically isolates errors where the model confuses versions rather than hallucinating non-existent symbols.

Software Evolution Understanding Score (SEUS)

To aggregate performance, the authors propose SEUS, a composite metric that penalizes inconsistency and anachronism. It is calculated as the harmonic mean of scores on Stable and Evolving APIs, adjusted by:

• Stability Penalty: Penalizes variance in performance across different versions of Evolving APIs.
• Anachronism Penalty: Penalizes the rate of anachronistic predictions.

Key Results

The authors evaluated frontier proprietary models (OpenAI GPT-4.1 through GPT-5.5, Anthropic Claude Sonnet 4/4.6, Google Gemini 2.0/2.5/3 Flash) and the open-weight Qwen-3.5 family (35B, 122B, 397B).

1. Version-Obliviousness: State-of-the-art models are largely version-oblivious. Performance on Evolving APIs degrades significantly as library versions become newer, while performance on Stable APIs remains flat. Even the strongest models (e.g., GPT-5.5, Sonnet 4.6) exhibit a persistent 7–10 point accuracy gap between Stable and Evolving APIs.
2. Structural Limitation: The degradation is a structural property of the training process rather than a limitation of model scale. Within the Qwen-3.5 family, increasing model size improves absolute performance but does not alter the downward trend on Evolving APIs.
3. Ineffectiveness of Explicit Constraints: Providing an explicit library version constraint in the prompt (L1→L2) yields no measurable improvement in accuracy. This suggests that while the version tag triggers a response, the model lacks the underlying version-differentiated knowledge to back it up.
4. Benefit of Documentation: Providing relevant documentation (L0→L1) results in a consistent 10–20 point improvement, indicating that models can retrieve and use information when made explicit, but fail to internalize it during training.
5. Error Analysis: Errors are primarily driven by confusion rather than fabrication. Models frequently produce valid but incorrect symbols from the same library (Invalid) or symbols from different versions (Anachronistic). Anachronistic errors constitute a substantial fraction of parameter-level errors (6–14% in Signature Recall), intensifying with newer target versions.

Significance and Contributions

The paper makes three primary contributions:

1. LibEvoBench: A novel, large-scale benchmark for evaluating how well LLMs navigate evolving APIs, featuring a deterministic, version-indexed ground truth that avoids the heuristic limitations of prior work.
2. SEUS Metric: A unified metric designed to track progress on software evolution understanding, explicitly penalizing inconsistent version-specific knowledge and anachronistic errors.
3. Empirical Evidence of a Capability Gap: The evaluation establishes that current LLMs are fundamentally version-oblivious. The findings highlight that neither model scale nor recent training data has proven sufficient to close the gap in temporal knowledge stratification.

The authors conclude that temporal knowledge stratification in code generation is a persistent, first-class capability gap. The results motivate the need for new approaches to training and retrieval that can ground code generation in temporally specific contexts, moving beyond static code generation to master the continuous, version-differentiated evolution of real-world software ecosystems.