Technical Summary: Correct Code, Vulnerable Dependencies

Problem Statement

Large Language Models (LLMs) have become integral to software development workflows, routinely generating code that includes third-party library (TPL) imports annotated with specific version identifiers. While prior research has extensively studied the logical security of generated code (e.g., CWE vulnerabilities) and the existence of hallucinated package names, the version dimension remains largely unexplored. Specifically, it is unknown whether the specific release versions of correctly named libraries recommended by LLMs are safe, compatible, or even valid.

This paper identifies a critical risk surface: LLMs may generate syntactically correct code that pins a library version carrying known Common Vulnerabilities and Exposures (CVEs) or one that is incompatible with the generated API usage. Because these choices are made implicitly at scale during code generation, they can silently propagate systemic security risks and installation failures into downstream projects.

Methodology

The authors present PinTrace, the first large-scale measurement study of LLM-specified library versions. The study evaluates 10 diverse LLMs (including closed-source models like GPT-5.4 and open-source models like Qwen3 and Llama-4) on a curated benchmark of 1,000 real-world Stack Overflow programming tasks.

Experimental Design

The study employs two prompting modes to reflect common developer interactions:

1. Inline Mode: The LLM annotates imports with version comments (e.g., import lib # version==x.y.z).
2. Explicit Mode: The LLM generates a requirements.txt manifest file alongside the code.

The analysis pipeline consists of five stages:

1. Generation: Producing code for each task under both modes.
2. Extraction: Parsing library references and version specifiers.
3. Validation: Verifying against PyPI to ensure versions are published releases (filtering out hallucinations).
4. Vulnerability Check: Cross-referencing valid versions against the Open Source Vulnerability (OSV) database to identify CVEs and CVSS severity.
5. Compatibility Check:
   • Static: Attempting installation in isolated virtual environments and running static type checking (ty).
   • Dynamic: Executing test suites from the BigCodeBench benchmark to verify runtime correctness.

The study also includes robustness checks across Python versions (3.8–3.14), a diagnosis experiment substituting neighboring versions to isolate failure causes, and mitigation probes testing safety instructions and external version anchoring.

Key Contributions

1. Measurement Study: A comprehensive evaluation of version specification behavior, validity, vulnerability exposure, and compatibility across 10 models and 1,000 tasks.
2. Systemic Bias Evidence: Empirical proof that LLMs converge on a small set of risky, outdated versions across different model families, driven by training signal bias rather than isolated model errors.
3. Root Cause Diagnosis: A controlled experiment establishing that installation failures are primarily attributable to version selection (choosing incompatible/old versions) rather than defects in the generated code logic.
4. Mitigation Analysis: A characterization of prompt-level interventions, demonstrating that natural language safety instructions are ineffective, while external version anchoring significantly reduces risks.
5. Open Resources: Release of the PinTrace dataset and evaluation pipeline for reproducible research.

Key Results

1. Version Specification Behavior (RQ1)

• Format Affordance: Version annotation is driven by the output format, not engineering intent. In Inline mode, models specify versions for 26.83%–95.18% of references. In Explicit mode (manifest generation), this drops to 6.45%–59.19%, with many models omitting versions entirely.
• Convergence: Models converge on a narrow band of popular releases. For example, requests==2.31.0 and django==6.0.1 are specified by nearly all models.
• Temporal Lag: Specified versions lag behind model knowledge cutoffs by 9 to 31 months. Models preferentially select older versions that have accumulated more training signal.
• Validity: While most inline-specified versions are valid, 7%–16% are hallucinated (invalid PyPI releases), often due to suffix omissions (e.g., 4.8.0 instead of 4.8.0.74).

2. Vulnerability Exposure (RQ2)

• High Exposure: Under inline prompting, 36.70%–55.70% of tasks contain at least one version with a known CVE.
• Severity: 62.75%–74.51% of these vulnerable versions are rated Critical or High severity.
• Avoidability: In 72.27%–91.37% of cases, the associated CVEs were publicly disclosed before the model's knowledge cutoff, indicating the exposure is avoidable but not recognized by the model.
• Structural Risk: 95.11% of vulnerable (library, version) pairs have no CVE-free alternative within the same major version branch, meaning simple patch updates cannot resolve the risk.

3. Compatibility (RQ3)

• Static Compatibility: Rates range from 19.70% to 63.20% in inline mode. The dominant cause of failure is dependency installation error, often due to missing distutils in Python 3.12+ environments (a consequence of selecting older versions).
• Dynamic Compatibility: Pass rates collapse to 6.49%–48.62% in inline mode. The majority of tasks fail before execution due to installation errors.
• Diagnosis: A substitution experiment confirmed that swapping the LLM-specified version with a neighboring release resolves installation failures in 17.19%–55.00% of cases, proving the code logic is often sound but the version choice is flawed.

4. Mitigation

• Prompting: Adding natural language safety instructions ("avoid vulnerable versions") yields negligible improvement (<2.2% reduction in exposure).
• External Anchoring: Injecting pre-computed safe versions into the prompt reduces vulnerability exposure significantly (down to ~10–27%) and improves static compatibility to 80%–93%.

Significance and Claims

The paper argues that LLM version selection is a first-class, previously overlooked risk surface. The findings reveal that:

1. The risk is systemic, not idiosyncratic: The convergence on the same vulnerable versions across all model families indicates a shared training-signal bias, making it impossible to mitigate by simply switching models.
2. The problem is a grounding gap, not a safety alignment gap: Since most CVEs were known before training, the failure lies in the model's inability to map version identifiers to their security status at inference time.
3. Prompting is insufficient: Natural language instructions cannot bridge the gap between version selection and security knowledge.
4. External tooling is required: The authors conclude that addressing this risk requires external mechanisms (e.g., SCA tools, version anchoring) rather than relying on the LLM's internal knowledge.

The authors have disclosed these findings to the evaluated model providers (including OpenAI, Alibaba, and Moonshot AI), several of whom have acknowledged the issue. The paper calls for the integration of version-level security and compatibility assessment into standard LLM evaluation benchmarks.