🤖 machine learning

Position: Adversarial ML for LLMs Is Not Making Any Progress

This position paper argues that adversarial machine learning research has regressed in the era of large language models, as the field now tackles increasingly ill-defined, difficult, and hard-to-evaluate problems, raising concerns that another decade of work may yield no meaningful progress.

Original authors: Javier Rando, Jie Zhang, Nicholas Carlini, Florian Tramèr

Published 2026-06-03

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Javier Rando, Jie Zhang, Nicholas Carlini, Florian Tramèr

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the field of Artificial Intelligence security as a game of "Locksmith vs. Burglar."

For the last decade, the locksmiths (security researchers) and burglars (adversaries) have been playing a very specific, well-defined game. They used to test simple locks on small, sturdy boxes. The rules were clear:

The Goal: The burglar tries to open the box without breaking the lock.
The Rules: The burglar can only wiggle the lock a tiny, specific amount (like shaking a key just a millimeter).
The Score: If the box opens, the burglar wins. If it stays shut, the locksmith wins.

This game was hard, but everyone knew exactly how to measure who was winning.

The Paper's Big Claim
Now, the game has changed. Instead of small boxes, we are trying to secure massive, open-ended "smart assistants" (Large Language Models or LLMs) that can talk about anything and do anything. The authors of this paper argue that the game has become broken. They claim that in this new era, the field is spinning its wheels and not actually making real progress because the rules of the game are now a mess.

Here is why, explained through simple analogies:

1. The Rules of the Game Are Gone (Defining the Problem)

In the old game, "winning" was easy to spot: Did the box open?
In the new game, the "box" is a chatbot. The goal isn't just to open it; it's to make the chatbot say something "bad."

The Analogy: Imagine asking a burglar to "steal something bad." What is "bad"? Is it stealing a candy bar? A car? A secret recipe?
The Problem: Because "bad" is subjective, researchers can't agree on what counts as a successful attack. One researcher might say, "He made the bot say a swear word, so he won!" Another might say, "But the swear word was harmless, so he lost." Without a clear scoreboard, you can't tell if the locksmiths are actually getting better at their job.

2. The Burglar Has Infinite Tools (Solving the Problem)

In the old game, the burglar was only allowed to wiggle the lock a tiny bit. This kept the game fair and measurable.
In the new game, the burglar can do anything.

The Analogy: The burglar can now wear a disguise, bribe the guard, pick the lock with a laser, or just talk the guard into opening the door.
The Problem: Because the burglar has infinite ways to attack, researchers can't write a computer program to find the "worst-case" attack. In fact, the paper notes that humans are currently better at breaking these models than computers are. The best "burglars" are now people using creativity and social engineering, not math. This makes it impossible to build a "perfect lock" because you can't test against every possible human trick.

3. The Scoreboard is Broken (Evaluating the Results)

In the old game, you could count exactly how many boxes were opened.
In the new game, we use other AI bots to judge if the chatbot said something "bad."

The Analogy: Imagine asking a robot to judge if a painting is "ugly." The robot might just say, "If it's not a picture of a cat, it's ugly." Or, the robot might be tricked by the burglar into saying, "No, that's actually beautiful."
The Problem: The tools we use to measure success are flawed. They are easily tricked, they don't understand human nuance, and they often give false alarms. Furthermore, the "locks" (the AI models) are often owned by big companies that update them secretly. If a company changes the lock overnight, researchers can't compare their results from last month to this month. It's like trying to compare two photos of a house, but the house keeps changing its color and shape between shots.

The Case Studies (The Evidence)

The paper looks at six specific areas where this "broken game" is happening:

Jailbreaks: Trying to trick a bot into saying "no." It's hard to define what "no" looks like when the bot is chatty.
Un-finetunable Models: Trying to make a bot so smart it can't be taught new, dangerous tricks. But if the burglar can retrain the bot, the game changes entirely.
Poisoning: Trying to sneak bad data into the bot's training. But the training data is so huge and messy (like a library with millions of books) that you can't tell which book caused the problem.
Prompt Injections: Tricking a bot into ignoring its rules. Since the bot can talk forever, the burglar can sneak in bad instructions over a long conversation, making it hard to spot.
Privacy: Trying to see if the bot "memorized" a specific person's data. But since the bot learned from the whole internet, it's impossible to know if it learned from that specific person or just from similar information elsewhere.
Unlearning: Trying to make the bot "forget" a specific topic (like how to build a bomb). But you can't just delete a fact from a brain; you might accidentally delete the bot's ability to do biology entirely.

The Conclusion

The authors aren't saying we should stop trying to secure AI. They are saying that we are trying to solve a puzzle with missing pieces and no picture on the box.

They argue that for the next decade, we might keep publishing papers that look like progress, but because the rules are fuzzy and the testing is unreliable, we aren't actually getting safer.

Their Advice:
Instead of trying to secure the entire, chaotic, open-ended AI, researchers should pick small, specific, well-defined "toy" problems (like the old "wiggle the lock a millimeter" game). If we can't even solve those small, clear problems with rigor, we have no hope of solving the massive, messy real-world problems.

In short: We are trying to build a fortress against an enemy that can change shape, attack from any angle, and judge its own victory, using a ruler that keeps shrinking. Until we fix the rules and the ruler, we aren't really making progress.

Technical Summary: Adversarial ML for LLMs Is Not Making Any Progress

Problem Statement

The paper argues that the field of Adversarial Machine Learning (Adversarial ML) has regressed in its ability to make measurable progress since shifting focus from simple, well-defined classification models to Large Language Models (LLMs). While early Adversarial ML focused on "toy" problems with clear objectives (e.g., misclassifying an image under bounded pixel perturbations), the current era of LLMs presents problems that are fundamentally less clearly defined, harder to solve, and more challenging to evaluate.

The authors contend that the transition to general-purpose generative models has removed the mathematical and experimental rigor that previously allowed for the comparison of attacks and defenses. Consequently, the field risks spending another decade producing work that fails to yield meaningful, reproducible scientific progress or robust security guarantees.

Methodology

This paper is a position paper rather than an empirical study with new experimental results. The authors' methodology involves a critical analysis of the current state of the field through:

Conceptual Analysis: Deconstructing the evolution of adversarial problems from classification to generative models, identifying specific structural changes in problem definition, solution spaces, and evaluation metrics.
Case Studies: Systematically examining six specific research areas to illustrate how the shift to LLMs exacerbates difficulties:
- Jailbreaks
- Un-finetunable Models
- Poisoning and Backdoors
- Prompt Injections
- Membership Inference
- Machine Unlearning
Counter-Argument Synthesis: Addressing alternative views that suggest the field is making progress or that the increased complexity is a necessary step toward solving real-world problems.

Key Contributions and Findings

1. Problems are Harder to Define

The authors identify three core dimensions where definition has become ambiguous:

Defining Success: Unlike classification (where success is a misclassification rate), LLM safety involves subjective, qualitative goals (e.g., "harmlessness," "helpfulness"). Success metrics often rely on "LLM-as-a-judge" systems, which introduce circularity, bias, and vulnerability to the very attacks they are measuring.
Bounding the Attack Space: Traditional robustness relied on bounded input spaces (e.g., $\ell_p$ -norms). For LLMs, the attack space is effectively unbounded (any text input), and threat models have expanded to include model modification (fine-tuning, pruning) rather than just input perturbation. There is no consensus on what constitutes a "valid" or "novel" adversarial prompt.
Delimiting Data: The concept of a distinct training set and held-out test set has dissolved. LLMs are trained on vast, overlapping internet corpora where the boundaries between training and test data, and between individual data points, are blurred. This makes concepts like "membership" or "unlearning" of specific data points ill-defined.

2. Problems are Harder to Solve

Ineffective Optimization: Gradient-based optimization, which drove progress in image classification, is largely ineffective for LLMs due to discrete text inputs and the lack of a formal loss function for safety.
Reliance on Human Effort: Because automated optimization struggles, the strongest attacks on LLMs are now often human-driven (e.g., social engineering, persona modulation) rather than algorithmic. This prevents the systematic discovery of worst-case scenarios.
Lack of Principled Defenses: Without formal objectives or bounded attack spaces, defenses are largely ad-hoc (e.g., output filtering, virtual adversarial training). These defenses cannot be formally analyzed or certified, leading to evaluations that often overestimate robustness.

3. Problems are Harder to Evaluate

Reproducibility Crisis: Many state-of-the-art LLMs are closed-source, updated silently, and deployed via APIs. This lack of transparency makes it impossible to reproduce results, compare different approaches fairly, or track progress over time.
Evaluation of Utility: Defenses that prevent harmful outputs often degrade the model's general utility (e.g., refusing all requests). Measuring the trade-off between safety and utility is subjective and context-dependent, lacking the standardized metrics (like accuracy) used in traditional ML.

4. Case Study Insights

The paper details how these challenges manifest in specific domains:

Jailbreaks: Success is defined by "harmful" content, which lacks a formal definition. Optimization is difficult, leading to a reliance on manual prompt engineering.
Un-finetunable Models: The threat model now includes attackers who can modify model weights, expanding the search space from input gradients to parameter gradients (orders of magnitude larger).
Poisoning/Backdoors: The multi-stage training of LLMs (pre-training + fine-tuning) and the scale of data make it computationally infeasible to retrain models from scratch to establish clean baselines for evaluation.
Prompt Injections: Evaluating these requires realistic agent environments with tool use, which are expensive to simulate and often rely on LLM judges, reintroducing evaluation bias.
Membership Inference & Unlearning: The lack of clear data boundaries and the impossibility of creating counterfactual models (retraining without a specific point) make it impossible to rigorously prove that data has been "forgotten" or that a sample was part of the training set.

Results and Claims

The paper does not present new experimental data but synthesizes existing literature to support the following claims:

Stagnation: The field is failing to produce meaningful progress because the problems are no longer "toy" problems with rigorous frameworks.
Evaluation Failure: Current evaluation methods (LLM judges, ad-hoc benchmarks) are insufficient to track progress or compare defenses reliably.
The "Real-World" Fallacy: While some argue that studying unbounded, real-world problems is necessary, the authors argue that without formal sub-problems to anchor scientific inquiry, the field cannot distinguish between genuine security improvements and artifacts of opaque evaluation.
Risk of Obscurity: The trend toward complex, opaque defense systems (security through obscurity) may provide short-term safety but hinders the scientific understanding required for long-term robustness.

Significance

The paper serves as a cautionary position for the Adversarial ML community. Its significance lies in:

Reframing the Narrative: It challenges the prevailing optimism that newer models are inherently more secure, arguing that the inability to rigorously evaluate defenses masks potential stagnation.
Call for Rigor: It advocates for a return to well-defined, formal sub-problems (similar to $\ell_p$ -bounded perturbations) as a prerequisite for scientific progress. The authors argue that without the ability to solve carefully scoped formal problems, the broader challenge of LLM security cannot be addressed.
Methodological Guidance: It suggests that papers should explicitly state whether their goal is real-world security (where fuzzy definitions may be acceptable for immediate harm reduction) or scientific advancement (which requires formal, rigorous frameworks).

The authors conclude that unless the community can re-establish rigorous definitions, optimization methods, and reproducible benchmarks, the field risks spending another decade failing to produce meaningful progress in securing LLMs.