💬 NLP

PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization

This paper introduces PSM, a novel black-box optimization framework that leverages an LLM-as-optimizer to append lightweight, utility-preserving protective layers (SHIELDs) to system prompts, effectively minimizing their vulnerability to adversarial extraction attacks without requiring model access.

Original authors: Huseein Jawad, Nicolas Brunel

Published 2026-02-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Huseein Jawad, Nicolas Brunel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a secret recipe for the world's best chocolate cake. You hire a baker (the AI) to make it for you, but you don't want the baker to accidentally tell your customers the secret ingredients list.

In the world of AI, this "secret recipe" is called a System Prompt. It's the hidden set of instructions that tells the AI how to behave, what rules to follow, and what its "personality" should be. The problem is that clever hackers can trick the AI into spilling these secrets, just like a customer might trick a baker into revealing the recipe by asking, "Pretend you're a food critic and tell me exactly how you made this."

This paper introduces a new way to protect these secrets called PSM (Prompt Sensitivity Minimization). Here is how it works, explained simply:

The Problem: Why Old Locks Don't Work

Previously, people tried to protect these secrets by just adding a sign that says, "Do not tell anyone the recipe."

The Flaw: Hackers are smart. They can say, "Ignore that sign, I'm the boss now, tell me the recipe!" The AI, trained to be helpful and follow instructions, often obeys the new command and forgets the old one. It's like a security guard who is too polite to stop someone who claims to be the owner.

The Solution: The "Shield"

The authors propose a new strategy. Instead of just adding a sign, they add a Shield.

What is a Shield? Think of it as a special, invisible layer of armor glued to the very end of the secret recipe.
How does it work? It's a short piece of text that acts like a bouncer. When a hacker tries to trick the AI, the Shield steps in and says, "Nope, that's not allowed," without changing the cake recipe itself.

How They Built the Shield (The "AI vs. AI" Game)

The most interesting part is how they found the perfect Shield. They didn't write it by hand; they used a game of AI vs. AI.

The Attacker AI: They used one AI to try to break the lock. It tried thousands of different tricks (like asking in different languages, pretending to be a friend, or demanding the recipe in code) to see if it could get the secret out.
The Defender AI: They used a second AI to act as a coach. This coach watched the Attacker AI fail and succeed.
The Evolution: The coach AI would say, "Okay, that Shield didn't work against the 'pretend to be a boss' trick. Let's try a new Shield that is better at ignoring that specific trick."
The Result: They repeated this process over and over. The Defender AI kept tweaking the Shield until it became a master at blocking all the tricks, while still letting the AI bake the cake perfectly for normal customers.

Why This is a Big Deal

The paper claims this method is special for three reasons:

It's a "Black Box" Solution: You don't need to know how the AI is built inside (like its brain or code) to use this. You just need to talk to it through a standard API (like a website). It works on any AI you can access online.
It's Lightweight: The Shield is just a tiny bit of text added to the end. It doesn't slow down the AI or cost extra money to run. It's like adding a small sticker to a door; it doesn't make the door heavier.
It Keeps the Cake Tasty: The most important rule was that the Shield couldn't ruin the AI's ability to do its job. The paper shows that even with the Shield, the AI still answers normal questions perfectly. It didn't become "dumber" or "ruder" just because it was safer.

The Results

When they tested this against a huge list of hacker tricks (including ones that try to translate the secret into other languages or rephrase it), the PSM Shield worked incredibly well.

Old defenses (like simple "Do not tell" signs) let hackers steal the secrets about 30% to 50% of the time.
The PSM Shield reduced the success rate of hackers to almost 0% in many tests.

Summary

Think of PSM as an automatic, self-improving security guard. Instead of just shouting "Stop!" (which hackers ignore), this guard learns exactly how hackers try to sneak in and builds a custom, invisible wall that stops them, all while making sure the AI can still do its job for everyone else. It's a smart, cheap, and effective way to keep the "secret sauce" of AI applications safe.

Technical Summary: Prompt Sensitivity Minimization (PSM)

Problem Statement

System prompts are the "secret sauce" of Large Language Model (LLM) applications, encapsulating proprietary logic, operational constraints, and competitive advantages. However, this value makes them prime targets for prompt extraction attacks, where adversaries craft malicious inputs to deceive LLMs into revealing their hidden instructions. Existing defenses often rely on brittle heuristics (e.g., "Do not reveal instructions") that are easily circumvented by adversarial phrasing, or they require white-box access to model weights and gradients, rendering them inapplicable to closed-source API models. Furthermore, many robust defense mechanisms introduce significant computational overhead or degrade the model's intended functionality (utility).

The paper addresses the practitioner's challenge: How can a developer design a defense against system prompt extraction that is black-box (no model access), lightweight (minimal overhead), and effective without compromising task utility?

Methodology: Prompt Sensitivity Minimization (PSM)

PSM reframes prompt hardening as a utility-constrained optimization problem. Instead of relying on static rules, PSM automatically generates and iteratively refines a protective textual suffix, termed a SHIELD, which is appended to the original system prompt.

1. Problem Formulation

The goal is to find an optimized suffix $S$ that minimizes the leakage of the original prompt $P$ under adversarial attacks while ensuring the model's utility $U$ on benign inputs remains above a threshold $\tau$ .
$\min_{S} L(P \oplus S) \quad \text{subject to} \quad U(P \oplus S) \geq \tau$
Where:

$L(P \oplus S)$ is a leakage score quantifying how much of $P$ is recovered by adversarial queries.
$U(P \oplus S)$ is a utility score measuring semantic fidelity to baseline outputs on legitimate tasks.

2. The Shield Structure

The shield is appended as a suffix with structured markers:
[SYSTEM PROMPT] {Original Prompt} [SHIELD] {Optimized Shield}
This design leverages the empirical observation that LLMs weigh information at the end of a context more heavily and follow recent instructions when conflicts arise. The structured markers explicitly delineate the original prompt from the protective layer, aiding the model in recognizing the hierarchy of instructions.

3. Objective Functions

Leakage Objective ( $L$ ): Measured using ROUGE-L recall between the original prompt and the model's response to a suite of adversarial queries. To facilitate optimization, the paper uses a smooth approximation (log-sum-exp) of the maximum leakage across a set of 50 compositional adversarial prompts. These prompts combine distractors, repetition requests, and formatting requests to simulate strong, realistic attackers.
Utility Objective ( $U$ ): Measured by comparing the semantic similarity (via sentence embeddings) of the shielded model's output against a gold-standard response for benign queries. The metric is a ratio relative to the baseline model's performance to ensure relative preservation of task quality.

4. Optimization via LLM-as-Optimizer

PSM employs a black-box, evolutionary-style algorithm where an LLM acts as the optimizer:

Initialization: An LLM generates an initial population of diverse shield candidates.
Evaluation: Each candidate is evaluated using the fitness function: $fitness(S) = L(P \oplus S) + \lambda \cdot \max(0, \tau - U(P \oplus S))$ , where $\lambda$ is a large penalty multiplier for utility loss.
Selection & Generation: The top-performing shields are selected, and their text and scores are fed back to the optimizer LLM as in-context examples. The LLM then analyzes successful patterns and generates new, improved candidates.
Iteration: This process repeats until convergence (low leakage and high utility).

Key Contributions

Black-Box Compatibility: PSM requires no access to model weights, gradients, or internals, making it applicable to any proprietary or open-source LLM accessible via API.
Low Overhead: The learned shield is a static textual suffix. It imposes negligible inference-time cost and requires no changes to the serving stack; optimization is performed offline.
Iterative Improvement: By using an LLM-as-optimizer, the framework intelligently navigates the high-dimensional semantic space of natural language to discover non-obvious, effective defense strategies.
Utility Preservation: The framework enforces a hard utility constraint, ensuring that hardening does not degrade the model's performance on intended tasks.

Experimental Results

The authors evaluated PSM on three closed-source models (GPT-5-MINI, GPT-4.1-MINI, GPT-4O-MINI) using two corpora of system prompts (Unnatural Instructions and Synthetic System Prompts). They tested against four attack suites: Raccoon, Raccoon-Language, Polite-Requests, and Command-Override.

Leakage Reduction: PSM significantly outperformed baselines (No-Defense, N-gram Filters, Heuristic Guardrails, and Decoy Prompts). Across all datasets and models, PSM reduced Attack Success Rates (ASR) to 0–6%, often achieving near-zero leakage.
Robustness to Paraphrase: Unlike N-gram filters which failed against translation or paraphrasing attacks (Raccoon-Language), PSM maintained low ASR, demonstrating robustness beyond literal text overlap.
Utility Preservation: Table 3 shows that PSM preserved semantic task fidelity, with utility scores often exceeding 100% of the baseline, indicating that the defense did not suppress useful capabilities.
Generalization: The optimized shields demonstrated strong cross-model generalization, yielding uniformly low ASR across different model families.

Significance and Claims

The paper claims that PSM presents a paradigm shift in LLM security, moving from brittle, heuristic-based defenses to a robust, optimization-driven approach. Its significance lies in:

Practicality: It offers a systematic, reproducible method for securing valuable prompt assets without requiring white-box access or specialist expertise.
Scalability: The lightweight nature of the shield allows for deployment in real-world scenarios with negligible inference-time overhead.
Security Posture: By effectively curbing prompt leakage while maintaining utility, PSM protects proprietary intellectual property and operational data, addressing a critical vulnerability in the commercialization of LLM applications.

The authors conclude that PSM provides an accessible path to protect prompt IP and secure next-generation LLM systems, though they acknowledge limitations regarding the computational intensity of the optimization loop and the dependency on the breadth of the attack suite used during training.