💻 computer science

The Security Budget of Code LLMs: An Information-Theoretic Capacity-Security Bound

This paper establishes and empirically validates an information-theoretic bound demonstrating that code LLMs operate under a fixed "security budget" where the sum of functional capacity and perturbation retention is limited by task entropy and prompt leakage, with experimental results showing this theoretical ceiling holds across various models, datasets, and precision levels.

Original authors: Jianwei Tai

Published 2026-06-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jianwei Tai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a very talented, but slightly nervous, robot programmer. You give it a set of instructions (a "prompt") to write a piece of code. Sometimes, you might accidentally change a word in your instructions, or a hacker might try to tweak them slightly to trick the robot into writing something dangerous.

This paper asks a fundamental question: How much of the robot's "brain" can we spare for doing the job correctly versus how much is left over for it to accidentally (or maliciously) follow a trick?

The authors call this the "Security Budget." They treat the robot's ability to think like a fixed amount of energy or bandwidth that has to be split between two competing tasks.

The Two Competing Needs

Think of the robot's attention as a pie. The paper says this pie is divided into two slices:

The "Job" Slice (Capacity): This is how well the robot understands your original intent. Did it write the code you actually asked for?
The "Echo" Slice (Security/Retention): This is how much the robot's output still "remembers" the specific words you used, even if you changed them slightly.
- The Catch: If the robot's output is too sensitive to tiny changes in your prompt (high "Echo"), it means a hacker could easily swap a word like "check" for "ignore" and the robot would follow the new, dangerous instruction.
- The Goal: You want the robot to be good at the job, but you don't want it to be too sensitive to the specific wording of the prompt.

The Big Rule (The Theorem)

The authors proved a mathematical rule that acts like a speed limit for this pie. They say:

Job Slice + Echo Slice ≤ Total Brain Space + Prompt Leakage

In plain English: The robot cannot be perfectly good at the job and perfectly sensitive to every tiny change in your prompt at the same time. There is a hard limit.

Total Brain Space: This is how complex the task is. If you ask for a simple "Hello World," the robot has plenty of room. If you ask for a complex banking system, the "Brain Space" is huge, leaving less room for safety margins.
Prompt Leakage: This is how much information is shared between your original prompt and the "tricked" prompt. If the trick is just changing "cat" to "dog" (synonyms), the leakage is high. If the trick is deleting half the sentence, the leakage is low.

The paper proves that if you try to make the robot too sensitive to the prompt (to make it very robust), you inevitably shrink the space available for it to do the actual job correctly.

How They Tested It

The researchers didn't just guess; they ran experiments with real AI models (like CodeLlama and Qwen) on real coding problems.

The "Black Box" Test: They looked at the robot's final output (the code) without peeking at its internal thoughts during the process. They treated the code like a fingerprint.
The Results: In every test, the math held up. The sum of the "Job" performance and the "Echo" sensitivity never broke the speed limit.
- Sometimes the robot was very good at the job but not very sensitive to tricks (leaving a lot of "slack" or unused budget).
- Sometimes it was very sensitive to tricks, but that meant it had less room to be perfect at the job.
- Crucially: They found that certain types of tricks (like renaming variables or swapping synonyms) leave a bigger "echo" than others. This tells us which types of prompt changes are the most dangerous to leave unguarded.

The "Stress Test"

To make sure their rule was tough, they tried to break it:

The 23-Attack Pool: They tried 23 different ways to mess with the prompt. The rule still held.
The "Universal Suffix": They added the same dangerous phrase to every prompt. The rule still held.
The "Gradient Attack": They used a super-smart mathematical attack to find the perfect way to trick the robot. Even then, the rule held, though the robot's code quality dropped significantly (it "collapsed" rather than being tricked).

The Takeaway for Humans

The paper concludes with a practical lesson for building AI assistants:

You cannot just measure if an AI passes a test. You also have to measure how much "information channel" you are leaving open for an attacker.

If you harden your prompts (make them rigid and standard), you reduce the "Echo" slice, making it harder for hackers to trick the AI.
However, you can't just make the AI "stupid" to be safe. You have to find the balance where the AI is still smart enough to do the job, but not so sensitive to wordplay that it becomes a security risk.

In short: There is a hard, mathematical limit to how much an AI can be both a perfect worker and a perfect listener to every tiny change in your voice. The paper gives us the ruler to measure that limit.

Technical Summary: The Security Budget of Code LLMs

Problem Statement
AI programming assistants increasingly rely on natural-language prompts as the primary interface for software development. This shift introduces a critical vulnerability: small perturbations to a prompt (e.g., synonym substitution, identifier renaming, or comment injection) can alter the generated code while leaving the user's intent partially recognizable. Existing evaluations often rely on execution pass rates (pass@1) before and after perturbation. However, the authors argue this is insufficient for interactive assistants because it fails to distinguish between a model that has lost all task information (making "robustness" meaningless) and one that retains enough information to be recognizable but shifts the implementation. There is a need for a "budget-level" view that separates task information, prompt leakage, and generation stability to understand the trade-offs between functional capacity and security under perturbation.

Methodology
The paper formalizes this trade-off using information theory, defining two key quantities for a code Large Language Model (LLM) $\pi$ :

Capacity ($Cap$): The mutual information between the canonical solution $c^*$ and the model's generation $c_\pi$ , denoted as $I(c^*; c_\pi)$ . This measures how much of the task the model captures.
Security ($Sec$): The mutual information between the generation under the original prompt $p$ ( $c_\pi$ ) and the generation under an adversarially perturbed prompt $\tilde{p}$ ( $\tilde{c}_\pi$ ), denoted as $I(c_\pi; \tilde{c}_\pi)$ . This acts as a "perturbation-retention channel," measuring the shared information between clean and perturbed outputs.

Theoretical Framework
The core theoretical contribution is Theorem 1, which establishes an upper bound on the sum of these two quantities:
$Cap + Sec \leq H(c^*) + I(p; \tilde{p})$
Here, $H(c^*)$ is the entropy of the canonical solution (task entropy), and $I(p; \tilde{p})$ is the mutual information between the original and perturbed prompts (prompt leakage). The proof relies on standard information-theoretic inequalities:

$Cap \leq H(c^*)$ follows from the definition of mutual information.
$Sec \leq I(p; \tilde{p})$ follows from the Data Processing Inequality (DPI), assuming the generations $c_\pi$ and $\tilde{c}_\pi$ are conditionally independent given the prompt pair $(p, \tilde{p})$ .

To make the bound actionable without access to the model's internal training data, the authors derive Theorem 2, a closed-form, model-agnostic companion bound for $H(c^*)$ . This bound utilizes the compressibility of source code:
$H(c^*) \leq \min(L_{max} \log V, |c^*|_{gz})$
where $L_{max}$ is the maximum token length, $V$ is the vocabulary size, and $|c^*|_{gz}$ is the expected gzip-compressed length of canonical solutions.

Empirical Validation
The authors validate the Corollary 1 (Embedded-Variable Bound) using an output-only embedding protocol. To preserve the Markov chain required by the proof ( $z_\pi \to p \to \tilde{p} \to \tilde{z}_\pi$ ), the embedding $E$ is applied only to the generated code, excluding the prompt context.

Models & Datasets: Experiments were conducted on CodeLlama-7B and Qwen2.5-Coder-7B across HumanEval and MBPP-sanitized datasets.
Estimators: Mutual information was estimated using MINE (Mutual Information Neural Estimation) and KSG (Kraskov-Stögbauer-Grassberger) estimators on last-token hidden states (reduced via PCA).
Perturbations: A pool of 23 perturbations was used, including synonym swaps, negation, comment injection (including security-anti comments), and identifier renaming.

Key Results

Bound Satisfaction: Across six individual validation configurations (varying models, datasets, precision, and embedding ablations), the embedded check satisfied the inequality. The saturation ratio $(Cap + \max_T Sec) / (H(z^*) + \max_T I(p; \tilde{p}))$ ranged from 0.27 to 0.92, with positive theorem slack (2.36 to 26.94 nats).
Stability: A three-seed stability diagnostic showed a mean saturation of 0.87.
Stress Tests: Adaptive stress tests, including a 23-perturbation pool, a fixed universal security-anti suffix, and a gradient-based PGD attack, all resulted in positive slack, confirming the bound holds even under aggressive perturbation strategies.
Per-Problem Signal: While the theoretical bound uses output-only embeddings, a separate context-mixed embedding cosine (conditioning on both prompt and code) was found to correlate with execution-level pass@1 (e.g., $\rho=0.36$ for CodeLlama-HumanEval), serving as a practical alignment signal distinct from the security bound.
Leakage Analysis: The study identified that identifier renaming and synonym substitution leave the largest shared-information ceilings (high $I(p; \tilde{p})$ ), whereas security-anti comments leak less prompt information but are more dangerous for safety.

Significance and Claims
The paper claims to provide a quantitative, information-theoretic budget for code generation stability. Its significance lies in:

Decomposition: Separating the "budget" into task entropy and prompt leakage, allowing developers to understand that reducing prompt leakage ( $I(p; \tilde{p})$ ) tightens the maximum channel available to an adversary without necessarily increasing functional capacity.
Model-Agnostic Ceiling: The closed-form bound on $H(c^*)$ allows for a security ceiling to be computed for any new model on a known task set without requiring model access or training data, relying only on the tokenizer and canonical solution distribution.
Practical Guardrails: The authors argue that programming-assistant guardrails should be evaluated not just by pass@1, but by the prompt-perturbation information they leave available to an adversary. The bound offers a test for prompt hardening: reducing $I(p; \tilde{p})$ should tighten the security budget.

The authors remain modest, noting that the contribution is the instantiation of these concepts for code generation (defining the budget and the closed-form ceiling) rather than a new data-processing theorem. They acknowledge limitations, including the sensitivity of MI estimators (where MINE can overestimate near boundaries) and the fact that the bound characterizes the available stability budget rather than proving that realistic adversaries will always saturate it.