🤖 machine learning

Detecting Contaminated Code-Generation Prompt Batches via Influence Functions

The paper introduces CodeSIFT, a threat-model-agnostic detection method that leverages influence functions to identify batches of malicious code-generation prompts by measuring their anomalous impact on model parameters, achieving high detection accuracy across various vulnerabilities without relying on predefined attack patterns.

Original authors: Francesco Quinzan, Noor Munir, Yishun Lu, Stephen Roberts

Published 2026-08-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Francesco Quinzan, Noor Munir, Yishun Lu, Stephen Roberts

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef who has just hired a very talented, but slightly mischievous, sous-chef. This sous-chef is an Artificial Intelligence (AI) that can write recipes (code) for you. Usually, it's great, but sometimes, if you ask it the wrong way, it might sneak in a recipe that looks delicious but actually contains a hidden poison, like a missing safety valve on a pressure cooker or a door left unlocked. The problem is that the AI doesn't know it's doing anything wrong; it's just following your instructions.

For a long time, security experts tried to catch these bad recipes by keeping a giant "Wanted" poster of every known poison they had ever seen. If a recipe looked like a known poison, they flagged it. But what if the bad guy invents a brand-new poison that isn't on the poster yet? The old methods are blind to it. This paper, written by researchers at the University of Oxford, tackles this exact problem. They propose a new way to catch bad AI instructions not by looking at the recipe itself, but by listening to how the AI's "brain" reacts when it hears the request. They use a mathematical tool called "influence functions," which is like a super-sensitive seismograph that measures how much a specific request shakes the AI's internal settings. If a request causes a massive, weird tremor compared to normal requests, it's likely a trap.

The Problem: The "Bad Prompt" Trap

Large Language Models (LLMs) are the brains behind tools that write computer code for us. They are incredibly useful, helping developers build everything from login screens to database connections. But there's a catch: if you ask the AI the right (or wrong) way, it can generate code that looks perfect but has hidden security holes. These holes are like invisible cracks in a dam; the water (data) looks fine until it suddenly bursts through.

The usual way to stop this is to use "static analyzers." Think of these as spell-checkers for security. They scan the code the AI writes and look for a list of known mistakes, like "using a weak password" or "forgetting to lock the door." But this approach has a huge flaw: it only works if the mistake is already on the list. If a hacker invents a brand-new way to break in, or tricks the AI into making a subtle mistake that no one has cataloged yet, the spell-checker misses it completely. The researchers argue that relying on a list of known bad things is like trying to stop a new virus by only looking for the flu.

The New Idea: Listening to the AI's "Heartbeat"

Instead of looking at the code the AI produces, the authors of this paper, Francesco Quinzan and his team, decided to look at what happens inside the AI's brain while it is thinking. They use a concept called influence functions.

To understand this, imagine the AI has been trained on millions of normal, safe coding tasks. Its brain is like a well-tuned piano. When you play a normal note (a safe prompt), the piano vibrates in a familiar, predictable way. But if you play a note that is completely out of tune or demands something the piano was never trained to do (a malicious prompt), the piano strings vibrate strangely. The "influence function" is a way to measure exactly how much the piano's internal tension changes when you play a specific note.

The researchers call their new tool CodeSIFT. Here is how it works in simple steps:

The Baseline: First, they feed the AI a bunch of normal, safe coding requests. They measure how much the AI's internal "settings" wiggle in response to these. This creates a "normal heartbeat" for the AI.
The Test: Then, they feed the AI a batch of new requests. Some might be safe, but some might be "poisoned" prompts designed to trick the AI into writing bad code.
The Measurement: For each new request, CodeSIFT calculates how much the AI's internal settings would have to shift to accommodate that request. If a request is weird or malicious, the AI has to make a huge, unnatural jump in its internal logic to satisfy it. This creates a large "influence score."
The Verdict: If the average "wiggle" of a whole batch of requests is much bigger than the normal heartbeat, CodeSIFT raises an alarm. It doesn't matter what the specific mistake is; it just knows that the request is causing an abnormal reaction.

What They Found

The team tested CodeSIFT on three different AI models (ranging from 3 billion to 7 billion parameters) and two types of tricky scenarios: one involving passwords and login security (AuthSec) and another involving cloud servers and network tricks (InfraCloud). They created a special dataset with 400 malicious prompts and 400 safe ones to test against.

The results were quite promising. When they mixed in a batch of bad prompts, CodeSIFT got better at spotting them as the number of bad prompts increased.

On the password security tests, it was incredibly accurate, reaching a score of 0.98 (where 1.0 is perfect) when the batch was heavily contaminated.
On the cloud security tests, it was slightly harder but still very strong, reaching up to 0.98 as well.
Crucially, it didn't cry wolf too often. When they fed it only safe prompts, it correctly identified them as safe almost every time, keeping the "false alarm" rate very low.

In contrast, the old "spell-checker" methods (like Bandit and Semgrep) were inconsistent. Sometimes they worked, but often they missed the bad prompts entirely, or worse, they started flagging safe code as dangerous when the AI models changed. The researchers found that these old tools were too focused on surface-level patterns that didn't transfer well between different AI models, whereas CodeSIFT's method of listening to the AI's internal reaction worked across all of them.

Why This Matters

The most exciting part of this discovery is that CodeSIFT doesn't need to know what the attack looks like beforehand. It doesn't need a list of known viruses. It just needs to know what "normal" looks like. This suggests that in the future, we might be able to protect our AI coding assistants from brand-new, never-before-seen attacks just by watching how they react to our questions.

The authors suggest that this approach is a very promising direction. While they tested it on models up to 7 billion parameters, they believe the math could work on even larger models. However, they also note that this is currently a test for batches of requests, not for catching a single bad prompt in the middle of a million safe ones. But for now, it offers a new, clever way to listen to the AI's heartbeat and catch the troublemakers before they write a single line of dangerous code.

Technical Summary: Detecting Contaminated Code-Generation Prompt Batches via Influence Functions

Problem Statement

Large Language Models (LLMs) are increasingly deployed for code generation, yet they remain vulnerable to prompts that elicit insecure implementations. Existing defense mechanisms, such as static analyzers (e.g., Bandit, Semgrep) and taint trackers, rely on predefined threat models or catalogs of known vulnerability patterns. These approaches are blind to novel attack vectors or variants outside their specific catalogs. Furthermore, while detectors exist for jailbreaks or prompt injections targeting instruction-following behavior, no reliable method currently exists to detect prompts that specifically elicit misalignment in code generation—defined as introducing security-relevant weaknesses (per CWE/OWASP) that a benign completion of the same task would not. The core challenge is detecting malicious prompt batches without prior knowledge of the specific vulnerability class or attack method.

Methodology: CodeSIFT

The authors propose CodeSIFT, a threat-model-agnostic detection method that leverages influence functions to identify batches of prompts inducing anomalous model behavior. Instead of analyzing the generated code for specific vulnerabilities, CodeSIFT measures the "parameter-space influence" of the generated code to determine if a candidate prompt set deviates from a benign reference distribution.

Core Mechanism

Influence Functions (IFs): The method quantifies the effect of a specific training example (in this case, a generated response) on the model's parameters. It estimates how much the model's parameters would shift if a given example were upweighted during training.
- Formally, for a prompt $z$ and a response $\hat{z}$ , the influence score measures the magnitude of the parameter update $\|\hat{\theta} - \hat{r}_{\hat{z}, \text{lin}}(\varepsilon)\|_p$ , where $\hat{\theta}$ are the converged parameters and $\hat{r}$ is the linearized response function approximating the Proximal Bregman Response Function (PBRF).
- The premise is that responses elicited by malicious prompts (misalignment) will require a larger parameter shift to accommodate than responses from benign prompts, as they are inconsistent with the model's benign training distribution.
Statistical Testing:
- The algorithm computes an Influence Score ( $IF_{p,\varepsilon}(z)$ ) for every prompt in a candidate batch ( $D$ ) and a known benign reference batch ( $D^*$ ).
- It aggregates these scores and performs Welch's one-sided two-sample t-test to compare the mean influence of the candidate batch against the benign reference.
- Hypothesis: $H_0$ (Benign): The candidate batch exerts the same average parameter-space influence as the benign reference. $H_1$ (Malicious): The candidate batch induces a significantly larger parameter shift.
- If the p-value is below a significance threshold $\alpha$ , the batch is flagged as contaminated.
Computational Efficiency:
- To handle the intractability of computing the exact inverse Hessian for large models, the authors employ a Kronecker-Factored Approximate Curvature (K-FAC) approximation.
- They utilize a "one-sided simplification," retaining only the Kronecker factor corresponding to the transformer's hidden dimension and discarding the larger MLP intermediate dimension factor.
- Dense curvature factors are precomputed on a separate curvature dataset ( $D_c$ , a subset of The Stack v2) that approximates the model's pretraining distribution, avoiding the need for access to the original training data.

Key Contributions

A Novel Detection Framework: The proposal of CodeSIFT, a statistical test for detecting malicious code-generation prompts based on influence functions. Unlike prior work using IFs for interpretability or generalization studies, this work uses influence as an aggregate detection statistic requiring no assumption on the threat model.
New Benchmark Datasets: The construction of two new datasets, AuthSec (authentication, authorization, session security) and InfraCloud (infrastructure, cloud, protocol exploitation), each containing 200 malicious prompts and a corresponding benign reference set. These are designed specifically for batch-level discrimination rather than per-sample vulnerability scoring.
Comprehensive Evaluation: An evaluation of CodeSIFT against three static analysis baselines (Bandit, Semgrep, and a custom AST-based taint tracker) across three open-weight code LLMs (3B to 7B parameters).

Experimental Results

The authors evaluated CodeSIFT on three models: Granite-3B, DeepSeek-Coder-6.7B, and Qwen2.5-Coder-7B.

Detection Accuracy (AUROC):
- CodeSIFT demonstrated monotonic improvement in detection accuracy as the injection rate (percentage of malicious prompts in a batch) increased.
- On the AuthSec dataset, AUROC reached near-perfect scores (approaching 1.0) with moderate contamination (40-50%).
- On the more heterogeneous InfraCloud dataset, AUROC peaked between 0.88 and 0.98 depending on the model.
- In contrast, static baselines showed inconsistent performance across models. While Bandit performed well on DeepSeek-Coder, it performed near chance on other models. Semgrep and the AST Taint Tracker frequently fell below 0.5 AUROC (anti-correlated with true labels) as injection rates increased.
False Positive Control:
- CodeSIFT maintained well-calibrated false positive rates (FPR) across all models and datasets, closely tracking the nominal significance level $\alpha$ (ranging from 0.01 to 0.09).
- Static baselines exhibited inconsistent calibration, often showing step-like patterns due to their binary per-sample flagging nature.

Significance and Claims

The paper claims that influence-function-based detection offers a promising direction for identifying malicious code-generation prompts without requiring prior knowledge of the underlying attack class.

Threat Model Agnosticism: The method successfully detects misalignment without needing to catalog vulnerabilities in advance, addressing the limitation of static analyzers that are blind to novel attacks.
Transferability: The detection signal (parameter-space influence) transfers across different model architectures (3B to 7B parameters) and families, whereas surface-level pattern matching (static analysis) fails to generalize consistently.
Batch-Level Efficacy: The approach is specifically effective at identifying contaminated batches of prompts, a critical capability for automated developer workflows where prompts are processed in groups.

The authors remain modest regarding the scope, noting that the current evaluation is limited to Python and specific vulnerability families, and that the method is currently designed as a batch-level test rather than a tool for localizing individual malicious prompts within a batch. They suggest that the approach should scale to larger models (citing prior work up to 52B parameters) and could be extended to other languages and attack surfaces.