Technical Summary: Diagnosing Knowledge Gaps in LLM Tool Use

Problem Statement

Code Large Language Models (LLMs) increasingly function as agents that must invoke software libraries and APIs to solve complex tasks. A critical challenge arises when these agents encounter novel APIs—functions or interfaces absent from their pretraining data due to library version updates, breaking changes, or emerging packages. Successfully utilizing these APIs requires more than recalling a function name; it demands coordinating heterogeneous knowledge types:

• Declarative: Signatures, parameter types, and module paths.
• Procedural: Correct composition within workflows and import resolution.
• Grounded: Concrete execution examples and implementation details.

Existing evaluation infrastructure for novel API acquisition is fundamentally limited. Current benchmarks are often static (rapidly degrading as models absorb new data), rely on coarse aggregate metrics (e.g., pass@k) that obscure the root causes of failure, and frequently synthesize artificial tasks that do not reflect real-world library evolution. Consequently, it remains poorly understood which specific knowledge components are learned by different adaptation paradigms (e.g., Retrieval-Augmented Generation vs. Fine-Tuning) and which missing components lead to specific failure modes.

Methodology: NOVELAPIBENCH

To address these limitations, the authors introduce NOVELAPIBENCH, a fully automated, dynamic, and execution-based benchmark designed to diagnose knowledge gaps in code LLMs. The framework operates through a four-stage pipeline:

1. Novel API Discovery: Given a target library $L$ and a base model $M$, the system identifies APIs introduced in library versions released after $M$'s knowledge cutoff. It diffs public API surfaces between the pre-cutoff version ($v_{old}$) and the post-cutoff version ($v_{new}$) using introspection and AST parsing, filtering out private, deprecated, or source-less entries.
2. Knowledge Extraction: For each novel API, the system decomposes knowledge into four distinct components:
   • $S$ (Surface): Fully-qualified names and parameter signatures (types, defaults, descriptions).
   • $E$ (Exemplars): Canonical usage examples mined from documentation and GitHub, validated in a sandbox.
   • $M_{prose}$ (Mechanism Prose): Natural-language descriptions of the underlying algorithm or design.
   • $M_{code}$ (Mechanism Code): AST-extracted implementation source code.
3. Task Generation: The system generates executable coding tasks with difficulty-controlled test harnesses. Crucially, every harness includes an auto-injected target-API spy that patches all aliases of the target API. This ensures that passing the test requires a genuine call to the target API, preventing models from bypassing the task via structural mimicry.
4. Quality & Novelty Filtering: Tasks are filtered to ensure the reference solution is valid, the base model cannot solve them unaided (empirical novelty), and the task is solvable with the full knowledge bundle.

Failure Taxonomy:
A key contribution is an automated failure taxonomy that classifies execution failures into six mutually exclusive categories, linking specific errors to missing knowledge:

1. WrongAPISelection: Target API never invoked (hallucinated or sibling symbol used).
2. WrongImport: Correct symbol but wrong/missing import path.
3. WrongSyntax: Invalid Python syntax.
4. WrongParam: Correct call but incorrect arguments.
5. WrongShapeDtype: Contract violation inside the API body (e.g., shape mismatch).
6. WrongLogic: Surrounding program structure is incorrect despite correct API usage.

Key Contributions

1. Dynamic Benchmarking: A regenerable benchmark that adapts to any base model and evolving library, ensuring tasks remain genuinely novel.
2. Granular Diagnosis: An automated failure taxonomy that connects specific execution errors to missing knowledge components across retrieval, fine-tuning, and knowledge-editing paradigms.
3. Systematic Empirical Study: A large-scale evaluation spanning ~800 novel APIs across 5 domains, 4 backbone models, and multiple adaptation methods (RAG, SFT, RAFT, GRACE, MEMIT, AlphaEdit).

Experimental Results

1. Effectiveness of Knowledge Components (RQ1)

• Usage Examples ($E$) are the strongest standalone component: Providing examples alone increased pass@1 from 3.1% to 18.2%, primarily by resolving WrongAPISelection errors.
• Signatures ($S$) provide the most reliable anchor: The surface cell ($S$) achieved 21.7% pass@1, outperforming all singletons, by reducing API selection errors.
• Mechanism Prose ($M_{prose}$) aids semantic disambiguation: When combined with signatures ($S+M_{prose}$), it further reduced logic errors but had minimal impact as a standalone component.
• Source Code ($M_{code}$) introduces noise: Adding implementation source code often degraded performance. It increased WrongImport errors significantly, suggesting that raw source code introduces module-path noise that confuses models rather than resolving residual errors.
• Optimal Configuration: The best two-component configurations paired signatures ($S$) with either mechanism prose or examples, depending on the domain and backbone. Full bundles ($S+E+M_{prose}+M_{code}$) often underperformed due to the negative interference of source code.

2. Impact of Task and Backbone (RQ2)

• Task Regularity: Examples ($E$) were highly effective for templatable, regular tasks (e.g., data science fixtures) but detrimental for heterogeneous tasks (e.g., AI4Science) where examples diverged from the specific requirement. Mechanism prose ($M_{prose}$) was more stable across task types.
• Backbone Preferences: The optimal secondary component depended on the backbone's inherent strength. Models where signatures ($S$) were already strong benefited more from prose; models relying on examples benefited from explicit call structures.
• Reasoning-Oriented Models: Reasoning-oriented backbones (e.g., R1-Distill) were more resistant to the import-path noise introduced by source code, likely due to their ability to re-resolve identifiers rather than copying paths.

3. Parametric Adaptation vs. Retrieval (RQ3)

• No Internalization of Volatile Facts: No parametric adaptation paradigm (SFT, RAFT, GRACE, MEMIT, AlphaEdit) fully internalized novel API knowledge. When the external knowledge bundle was removed at inference, performance dropped drastically (e.g., SFT dropped from 85.5% to 3.3%), failing to close the gap with retrieval-based methods.
• Learning Procedural Meta-Skills: Fine-tuning did not memorize API details (signatures, paths) but rather learned a transferable procedural skill: how to effectively utilize provided knowledge bundles.
  • SFT/RAFT/AlphaEdit significantly reduced WrongAPISelection and WrongImport errors when a bundle was present, shifting failures to WrongLogic.
  • This "bundle utilization" skill transferred to held-out libraries (OOD), but the specific module-path knowledge did not. Models trained on one library could identify the correct API in a new library but often failed to resolve the correct import path, leading to a spike in WrongImport errors.

Significance and Claims

The paper argues that content acquisition (identifying the API and its contract) and procedural realization (integrating it into code with correct imports and logic) are distinct sub-skills.

• Retrieval is essential for supplying volatile, low-structure facts (API names, signatures, paths) that are difficult for models to infer or memorize.
• Parametric Tuning (Fine-tuning) is effective for improving the procedural integration of such facts—teaching the model how to use retrieved evidence correctly.

The authors conclude that the optimal architecture for adapting coding LLMs to novel APIs is not a choice between RAG and fine-tuning, but a combination: compact, high-value retrieval units (e.g., signatures + mechanism prose) paired with adapters trained across heterogeneous API-use tasks to master procedural integration. Current "locate-and-edit" methods do not yet achieve this combined frontier. The work suggests that future progress may require process-level supervision (e.g., execution-grounded reinforcement learning) to better bridge the gap between identifying an API and correctly implementing it.