💻 computer science

Stdlib or Third-Party? Empirical Performance and Correctness of LLM-Assisted Zero-Dependency Python Libraries

This paper presents "zerodep," an open-source collection of LLM-assisted, single-file Python standard library re-implementations of popular third-party modules, demonstrating that while C-extension-heavy tasks remain a performance bottleneck, stdlib-only alternatives often achieve comparable speed or even significant speedups by eliminating architectural overhead, thereby validating the feasibility of dependency-free software engineering with high correctness.

Original authors: Peng Ding, Rick Stevens

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Peng Ding, Rick Stevens

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a house. In the world of Python programming, the "Standard Library" (stdlib) is like a high-quality, pre-packed toolbox that comes free with every Python installation. It has hammers, screwdrivers, and saws. However, for specific, fancy jobs, programmers often buy "Third-Party Libraries." These are like specialized, branded toolkits you download from the internet. They are powerful, but they come with a catch: they often require you to buy other tools just to use them (dependencies), they might break if the brand changes its rules, and they can introduce security risks if the brand is compromised.

This paper, titled "Stdlib or Third-Party?", asks a simple but profound question: How much of that fancy, specialized toolkit can we actually rebuild using just the free, basic tools we already have?

To answer this, the authors created a project called zerodep. Think of zerodep as a "DIY Workshop" where they took 44 popular, complex Python tools and tried to rebuild them from scratch using only the standard toolbox. They didn't do this alone; they used an AI assistant (an LLM) to help write the code, but they put the AI on a very strict leash:

No new tools allowed: You can't import anything that isn't already in the standard Python box.
One file only: The whole tool must fit in a single sheet of paper (a single .py file).
Drop-in replacement: It must work exactly like the original tool so you can swap them without breaking your house.
Proof of work: The AI's creation must pass a rigorous test against the original to prove it works correctly.

The Three Main Findings

The researchers tested these 44 rebuilt tools against the originals and found three distinct "zones" of performance:

1. The "Lightweight Win" Zone (The AI did great here)

For many common tasks, the original third-party tools were actually over-engineered. They were like a Swiss Army knife with 50 gadgets when you only needed a screwdriver. The AI, forced to use only the basic tools, built a simple, streamlined screwdriver that was actually faster.

The Analogy: Imagine a restaurant that serves a complex, multi-course meal just to get you a glass of water. The zerodep version is just a glass of water. It's much faster to get.
The Result: In categories like reading configuration files, handling retries (trying again if something fails), and parsing text, the AI-built tools were often 5 to 115 times faster than the originals because they cut out all the unnecessary "bloat."

2. The "Parity" Zone (Good enough)

For about two-thirds of the tools, the AI-built versions were just as good as the originals. They might be slightly slower or slightly faster, but they were within a "safe margin" (less than 2x difference).

The Analogy: It's like driving a reliable sedan versus a luxury sports car. The sports car (third-party) might be slightly faster, but the sedan (stdlib) gets you to the same destination in almost the same time, without needing a special mechanic to fix it.
The Result: For tasks like basic networking or data validation, the standard library is perfectly capable of doing the job without needing extra downloads.

3. The "Hard Wall" Zone (The C-Extension Cliff)

There was one place where the AI and the standard library hit a brick wall: heavy math and low-level processing.

The Analogy: Imagine trying to paint a massive mural. The third-party tools use a team of professional painters with industrial spray guns (compiled C or Rust code). The AI, restricted to the standard library, is forced to use a tiny paintbrush and a bucket of paint. No matter how hard the AI tries, it cannot compete with the speed of the industrial spray gun.
The Result: For things like processing images (pixels), heavy cryptography (encryption), or complex binary data, the standard library was significantly slower (sometimes 300x slower).
The Workaround: The authors found a clever trick: instead of painting the mural themselves, they built a small door in the wall and asked the professional painters (the system's built-in C libraries) to come in and do the heavy lifting. This "subprocess" trick allowed them to get the speed of the pros without needing to download new tools.

The Role of the AI (LLM)

The paper also looked at how well the AI did the work.

Simple Tasks: For small, simple tools, the AI was a wizard. It could write a working version in one or two tries.
Complex Tasks: For huge, complicated systems (like a full web server), the AI got confused. It needed a human architect to draw the blueprints first. Once the human set the structure, the AI could fill in the details.
The Safety Net: The most important part of the process was the "Correctness Test." The AI would guess, the test would say "Wrong," and the AI would try again. This loop ensured that the final product actually worked, preventing the AI from "hallucinating" (making up) fake tools that don't exist.

The Bottom Line

The paper concludes that for most everyday programming tasks, you don't need the fancy third-party tools. You can often build a faster, safer, and lighter version using just the standard library, especially with the help of an AI.

However, if you are doing heavy-duty work like image processing or high-speed encryption, the standard library hits a speed limit that only specialized, compiled code can break. In those cases, you either need the third-party tools or a clever workaround to borrow power from the system's native code.

In short: For 2/3 of your programming needs, the "free toolbox" is all you need, and it might even be faster. For the other 1/3, you still need the specialized gear, but you now know exactly where the line is drawn.

Technical Summary: Empirical Performance and Correctness of LLM-Assisted Zero-Dependency Python Libraries

Problem Statement

The Python package ecosystem, while abundant, introduces significant overhead through transitive dependencies, supply chain risks, and deployment friction. Empirical studies indicate that a single declared dependency often triggers a median of 8–10 additional transitive packages, with over 70% of code in a dependency closure never executed. Furthermore, code-generating Large Language Models (LLMs) frequently hallucinate non-existent package names, creating latent security vulnerabilities.

While developers often resort to "copy-paste" reuse for small utilities to avoid dependencies, this approach lacks systematic correctness guarantees, versioning, and performance baselines. The central research question is: To what extent can popular third-party utility libraries be replaced by single-file, standard-library-only (stdlib) implementations without sacrificing correctness or performance, and can LLMs effectively generate such code under strict constraints?

Methodology

The authors introduce zerodep, a curated testbed of 44 single-file Python modules. Each module is a reimplementation of a popular third-party library (e.g., PyYAML, httpx, Pillow, tenacity) using only the Python standard library.

Design Constraints

Zero Dependencies: No external imports; all functionality must rely on the Python standard library.
Single File: Each module is contained within a single .py file to simplify auditing and embedding.
API Compatibility: Modules provide drop-in API compatibility with their reference libraries.
Correctness Validation: Every module is paired with a test suite that compares its behavior against the reference library using identical inputs.

Development Workflow

The modules were developed through an iterative human-LLM collaboration process:

Specification: Defining the API surface and intentional simplifications.
Generation: LLMs (primarily Claude and o-series models) generate initial implementations under strict constraints.
Oracle Validation: Generated code is tested against the reference library; failures are fed back to the LLM for refinement.
Performance Assessment: Benchmarks are run to identify regressions, with human guidance used to optimize critical paths (e.g., selecting specific stdlib primitives).

Evaluation Scope

The study covers 44 modules across 12 categories (Network, Protocol, Serialization, Text, Crypto, etc.) and three complexity tiers (Simple, Medium, Subsystem). Evaluation focuses on:

Correctness: Behavioral equivalence against reference libraries across hundreds of test vectors.
Performance: Benchmarking execution time against reference libraries, defining "parity" as a speed ratio within a factor of 2 ( $0.5 \le r \le 2.0$ ).

Key Contributions

The zerodep Testbed: A versioned, benchmarked collection of 44 stdlib-only modules, each with a correctness test suite and performance baseline.
Empirical Correctness Evaluation: A systematic analysis of where behavioral equivalence holds and where intentional simplifications are applied.
Performance Characterization: Identification of three distinct performance regimes:
- Parity: Performance within 2× of the reference.
- Architectural Win: Significant speedups (5–115×) due to the elimination of reference library overhead.
- C-Extension Cliff: Severe performance degradation when the reference relies on compiled C/Rust extensions.
LLM-Assisted Development Analysis: Observations on the efficacy of LLMs across complexity tiers, highlighting the necessity of human architectural guidance for complex subsystems.

Results

Correctness

All 44 modules pass their correctness test suites. The study found that behavioral divergence is often subtle (e.g., YAML null representations, JSON comment edge cases) rather than API mismatches. Modules with intentional API simplifications (e.g., omitting rarely used validators) explicitly document these omissions.

Performance Regimes

Architectural Wins (Faster): 19 modules are consistently faster than their references, often by orders of magnitude.
- Examples: jsonc is 75–115× faster than commentjson (avoiding eval); httpclient is 18–32× faster than httpx (removing middleware layers); yaml is 6–7× faster than PyYAML's pure-Python path.
- Cause: Many third-party libraries carry architectural overhead (plugin systems, intermediate representations) that targeted stdlib implementations avoid.
Parity: 11 modules achieve performance parity (within 2×).
- Examples: dotenv, cache, and retry (in decorator overhead) show near-exact parity or minor gains.
C-Extension Cliff (Slower): 6 modules are significantly slower (often >10×) because their references rely on compiled extensions.
- Examples: protobuf (6–72× slower than google-protobuf), png (7–45× slower than Pillow), and pure-Python aes (300–17,000× slower than PyCryptodome).
- Mitigation: The aes module demonstrates a hybrid approach, delegating to system OpenSSL via subprocess, achieving 1.5–8× speedups over the reference while remaining zero-dependency in packaging.

LLM Effectiveness

Simple/Medium Tiers: LLMs produce near-correct drafts in 1–5 iterations, requiring mostly edge-case refinement.
Subsystem Tiers: Complex modules (e.g., full HTTP clients, protocol stacks) require significant human architectural guidance before LLM assistance becomes productive. The LLM accelerates implementation details but does not drive high-level design.

Significance and Claims

The paper claims that stdlib-only implementations are viable substitutes for the majority of utility and infrastructure libraries, challenging the assumption that third-party dependencies are necessary for performance or correctness.

Scope of Sufficiency: Roughly two-thirds of the studied modules achieve performance parity or better. The primary limitation is not the overhead of Python itself, but the reliance of third-party libraries on C/Rust extensions for compute-intensive tasks (image processing, binary serialization, hardware-accelerated crypto).
Architectural Overhead: The study highlights that "bloat" in reference libraries (abstraction layers, plugin systems) often imposes a performance penalty that targeted stdlib reimplementations can eliminate.
LLM Role: LLM-assisted development is effective for constrained, single-file generation but requires human oversight for complex subsystems. The use of a reference library as an "oracle" is critical for iterative refinement.
Practical Implication: For environments where dependency management is costly (e.g., constrained deployments, supply chain security), zerodep offers a curated, benchmarked path to zero-dependency software without meaningful loss in correctness or performance, provided the workload does not require native code acceleration.

The authors conclude that while pure-Python implementations cannot compete with native code in raw throughput for specific domains, the combination of stdlib-only constraints, LLM-assisted iteration, and subprocess offloading (where applicable) closes the gap sufficiently for a wide range of practical engineering tasks.