PyPI API Drift: A Release-Level Record of Public API Changes in 1,872 Popular Python Libraries, with a Checker and a Benchmark for Version-Aware Code Generation

In the vast, shifting landscape of modern software, Python stands as a dominant language, powering everything from scientific research to web applications. Its strength lies in a massive ecosystem of pre-built tools, known as libraries, which developers can plug into their projects to handle complex tasks without reinventing the wheel. However, these tools are not static monuments; they are living entities that evolve constantly. Developers frequently update them to fix bugs, improve speed, or add new features. In doing so, they sometimes quietly remove or alter the specific commands and instructions that users rely on. For human programmers, keeping track of these changes is a manageable challenge, often solved by reading release notes or asking colleagues. But for the new generation of artificial intelligence assistants that write code, this evolution presents a unique and persistent problem. These AI models are trained on vast archives of code from many different eras, learning to write instructions that were correct years ago but may now be broken. Without a way to know exactly which version of a tool a user has installed, an AI might confidently suggest a command that no longer exists, leading to errors that halt a project.

To solve this, a researcher named Pushkar Devanahalli Gopalakrishna has created a massive, detailed record of how these tools change over time. The project, called PyPI API Drift, acts as a historical archive for 1,872 of the most popular Python libraries. Instead of trying to run and test every single piece of code, which would be impossibly slow and expensive, the researcher developed a method to look directly at the digital files that make up these libraries. By examining only the necessary parts of these files, the system can map out exactly what commands, functions, and settings were available in every recent release of each library. This process covers over 55,000 specific versions of software released between 2021 and late 2026. The result is a precise ledger that tracks when a specific feature was added, when it was renamed, and when it was completely removed. Crucially, the system distinguishes between things that are truly gone and things that have simply moved or changed their name, marking each entry with a confidence level to indicate how certain the record is.

The scale of this change is significant. The analysis revealed that roughly 13.4 percent of minor software updates, which are often assumed to be safe and backward-compatible, actually contain at least one change that breaks existing code. Even more striking, about 64 percent of major version jumps and nearly 30 percent of minor ones include changes that would cause older code to fail. This data challenges the common assumption that software updates follow a strict, predictable pattern of safety. The researcher found that only a tiny fraction of removed features were officially marked as "deprecated" or "coming soon to be removed" in a way that automated tools could easily spot. Often, the warnings were hidden in text documents or conditional messages that a computer program could not read, leaving developers and AI assistants in the dark until the code broke.

To prove that this record is useful, the researcher built a set of tools to test its accuracy and utility. One tool acts as a checker that scans code to see if it uses commands that are no longer valid for a specific software version. Another tool serves as a lookup system for AI assistants, allowing them to query the history of a command before writing it. Most importantly, a series of experiments tested whether giving an AI access to this historical record actually helped it write better code. In tests where the AI had to fix code that was broken by a software update, the AI performed significantly better when it could consult this record. Without help, the AI managed to fix the code correctly less than half the time. When provided with the historical record, its success rate jumped to nearly 70 percent, outperforming even when it was given the official documentation for the new software version. This suggests that having a precise, version-specific history of changes is more valuable to an AI than reading general instructions.

The project also produced a benchmark, a standardized test set of thousands of code snippets, to help other researchers measure how well their own AI models handle these version changes. The findings confirm that while AI models are powerful, they struggle with the rapid pace of software evolution unless they are equipped with a reliable, up-to-date map of what exists and what has vanished. The record shows that the landscape of Python libraries is far more fluid than previously understood, with breaking changes appearing frequently even in small updates. By providing a clear, high-precision map of these changes, this work offers a foundation for building smarter coding assistants that can navigate the evolving world of software without getting lost in the drift. The data is now open for anyone to use, ensuring that the community can continue to build tools that keep pace with the constant motion of the digital world.