Evaluating Inference-Time Defenses Against Package Hallucination in LLM-Generated Code

In the modern world of software development, programmers often rely on artificial intelligence assistants to write code. These systems, known as large language models, act as tireless partners that can suggest entire functions or fix errors in seconds. To make their suggestions work, these models frequently recommend adding external software packages—pre-written collections of code that handle specific tasks like connecting to a database or creating a graph. The problem arises when the artificial intelligence invents a package name that sounds perfectly real but does not actually exist in any official software library. This phenomenon is called a package hallucination. If a developer blindly trusts the suggestion and tries to install this non-existent package, they may inadvertently download a malicious file created by a hacker who registered the fake name. This creates a dangerous backdoor into the software supply chain, allowing attackers to inject harmful code into applications that millions of people might use.

A team of researchers set out to understand how often these hallucinations occur and whether specific techniques could stop them before the code is even finished. They focused on smaller, open-source artificial intelligence models, which are widely used because they are less expensive to run, even though they are more prone to making mistakes than their larger counterparts. The researchers tested these models across four different programming languages: Python, JavaScript, Ruby, and Rust. They discovered that previous methods for measuring these errors were flawed. Many earlier studies counted standard, built-in tools that come with a programming language as hallucinations simply because those tools are not listed in the external package libraries. By correcting this counting error, the team found that the rate of hallucinations for Python was actually lower than previously thought, though still significant.

The core of their work involved testing seven different strategies to see if they could reduce the number of fake package names the models generated. Some of these strategies involved changing how the model selects its next word, while others asked the model to check its own work or look up information in a verified database before answering. The researchers found that no single method worked best in every situation. A technique called Retrieval-Augmented Generation, which forces the model to consult a real database of existing packages before speaking, proved highly effective for most languages, cutting the error rate significantly. However, this same technique sometimes made things worse for JavaScript, suggesting that the solution depends heavily on the specific language being used. Another approach, where the model is asked to critique and rewrite its own suggestions, worked well for larger models but failed for the smallest ones, which often could not recognize their own mistakes.

The team also introduced a new way to measure whether the model's suggestions were actually useful, not just correct. They found that some strategies that successfully stopped hallucinations also stopped the model from suggesting any packages at all, leaving the developer with nothing to use. The most balanced approach, which reduced errors while still providing helpful suggestions, was a straightforward method where the model simply picks the most likely next word every time, rather than taking a chance on less probable options. This "greedy" approach offered the best trade-off between safety and usefulness for the models they tested.

Perhaps the most striking finding emerged when the researchers tested these defenses against a hostile environment. They created prompts that deliberately tried to trick the models into recommending fake packages by embedding the fake names directly into the instructions. Under these adversarial conditions, the error rates skyrocketed, rising by as much as 45 percentage points compared to normal requests. In this hostile setting, the simple tricks of changing how the model picks words failed completely. Only the methods that relied on checking against a real database or forcing the model to double-check its own work could resist the attack. The researchers concluded that while simple adjustments can help in normal use, protecting software from determined attackers requires a system that can verify facts against the outside world or rigorously check its own logic. The study highlights that the best defense is not a one-size-fits-all solution, but a choice matched carefully to the specific threat and the programming language involved.