🤖 machine learning

Risk Under Pressure: Compute-Aware Evaluation of Adversarial Robustness in Language Models

This paper proposes a compute-aware evaluation framework that measures adversarial risk using cumulative FLOPs rather than fixed query budgets, revealing that alignment training and model scaling have non-monotonic and category-dependent effects on the computational effort required to jailbreak large language models.

Original authors: Malikeh Ehghaghi, Boglárka Ecsedi, Marsha Chechik, Colin Raffel

Published 2026-06-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Malikeh Ehghaghi, Boglárka Ecsedi, Marsha Chechik, Colin Raffel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: It's Not Just About If You Can Break In, But How Much It Costs

Imagine you own a high-security bank vault. A security company comes to test it. They report: "We broke into the vault 100% of the time."

In the world of Large Language Models (LLMs), that's usually the end of the story. Researchers say, "The model is unsafe because an attacker succeeded."

But this paper argues that's like saying a bank is unsafe just because a thief eventually got in, without asking how hard they had to work.

Scenario A: The thief picks the lock in 5 seconds with a paperclip.
Scenario B: The thief spends 10 days, uses a laser cutter, and hires a team of engineers to drill through the concrete.

Both scenarios result in a "broken" vault. But Scenario B is much less likely to happen in the real world because it's too expensive and difficult.

This paper introduces a new way to measure safety called "Risk Under Pressure." Instead of just counting how many times an AI failed, it measures how much computer power (effort) an attacker had to spend to make the AI say something bad.

The New Tools: Measuring "Computer Sweat"

The authors created a framework that treats computer processing power like a budget. They measure the "sweat" an attacker has to exert using FLOPs (floating-point operations), which is basically a count of how many math calculations the computer had to do.

They use two main tools to visualize this:

The "Risk-Compute Curve" (The Hill): Imagine a hill. The bottom is "easy to break," and the top is "hard to break."
- Some models are like a small hill; you can climb to the top (break the safety) with a few steps.
- Other models are like Mount Everest; you have to spend a massive amount of energy just to get halfway up.
The "Price Tag" (C@τ): This answers: "How much computer power does it take to break this model 50% of the time?"
- If the price tag is low, the model is vulnerable.
- If the price tag is huge, the model is robust, even if it can theoretically be broken.

What They Discovered: The Surprising Results

The researchers tested many different AI models and attack methods. Here is what they found, translated into everyday terms:

1. Training Doesn't Always Make Things Safer (The "Over-Training" Trap)

You might think that the more you train an AI to be safe, the safer it gets. The paper found this isn't always true.

The Analogy: Imagine teaching a child to say "no" to strangers.
- Stage 1 (Base): The child says "yes" to everything.
- Stage 2 (SFT): You teach them politely. They become very good at saying "no."
- Stage 3 (DPO/RL): You try to fine-tune them even more with rewards. Surprisingly, they sometimes get worse at saying "no" to tricky questions.
The Finding: Sometimes, the "middle" version of the model was actually the hardest to break. The final, most "aligned" versions sometimes became easier to trick, or at least didn't get much harder to break.

2. Bigger Models Aren't Always Safer (The "Big Target" Problem)

Making a model bigger (more parameters) is like building a bigger castle.

The Finding: If the attacker uses a "smart" method (like a gradient attack that calculates the perfect path), a bigger castle is much harder to break. It's like trying to climb a 100-story tower vs. a 1-story house.
The Catch: If the attacker uses a "dumb" method (like just trying random templates or copy-pasting bad prompts), the size of the castle doesn't matter. They can still break in just as easily.
Takeaway: Bigger models stop the "smart" hackers, but they don't stop the "lazy" hackers.

3. The "Surrogate" Trick (Stealing the Keys)

Attackers often don't have access to the secret, closed-source model they want to hack.

The Analogy: Imagine you want to break into a specific bank, but you can't get close to it. So, you go to a similar-looking bank down the street, pick its lock, and find a master key that works on both banks.
The Finding: The researchers showed that attackers can train their "lock-picking" tools on a small, open, free AI model. Once they figure out the trick, they can use that same trick on a huge, expensive, closed AI model. This saves the attacker a massive amount of money and effort.

4. Safety is Uneven (The "Swiss Cheese" Effect)

Even a model that seems generally safe has holes.

The Finding: It might take a lot of effort to make an AI talk about "bullying," but it might take very little effort to make it talk about "cybercrime" or "illegal drugs."
The Analogy: Imagine a fortress with a thick stone wall on the north side (hard to break) but a flimsy wooden door on the south side (easy to break). If you only measure the average wall thickness, you think the fortress is safe. But a smart attacker will just go through the wooden door.

Why This Matters

The paper argues that we need to stop just asking, "Did the AI break?" and start asking, "How much did it cost to break it?"

If an AI requires a supercomputer running for a week to be tricked, it's effectively safe for most people. If it takes 5 seconds, it's a disaster. By measuring the "cost" of the attack, we get a much clearer picture of real-world safety.

In short: The paper says we need to stop looking at the "Success Rate" scoreboard and start looking at the "Effort Required" scoreboard to truly understand how safe our AI models are.

Technical Summary: Risk Under Pressure: Compute-Aware Evaluation of Adversarial Robustness in Language Models

Problem Statement

Current adversarial robustness evaluations for Large Language Models (LLMs) predominantly rely on Attack Success Rate (ASR) measured under fixed query budgets. This approach implicitly treats all attack steps as having equal cost, ignoring the fact that different attack strategies (e.g., gradient-based optimization vs. template-based prompting) incur computational expenses that can vary by orders of magnitude. Consequently, a model that requires 100 queries of a cheap template attack to fail is scored identically to a model that fails immediately, obscuring the true "work factor" or adversarial effort required to breach the system. This limitation hinders the ability to determine if an attack's computational cost justifies its payoff for an adversary and makes it difficult to compare models under equal attacker spending.

Methodology

The authors propose a compute-aware evaluation framework that replaces query counts with computational pressure, measured in cumulative floating-point operations (FLOPs), as a proxy for adversarial effort.

Core Framework

Budgeted Iterative Refinement: Attacks are modeled as iterative processes where an attack policy $\pi$ proposes prompts, a target model $M$ generates responses, and a safety judge $E$ evaluates outcomes. This loop continues until a success is found or a query budget $\lambda$ is exhausted.
FLOPs Accounting: The framework calculates the cumulative FLOPs for each step, including:
- Forward passes through the target model (generation and evaluation).
- Backward passes for gradient-based attacks (white-box).
- Forward passes through auxiliary attacker models (black-box refinement).
- Forward passes through the safety judge.
- Costs are derived using standard transformer FLOP approximations ( $C_{fwd} \approx 2NL$ ), ensuring hardware-agnostic comparability.
Risk-Compute Curves: Instead of a single ASR point, the framework generates curves mapping cumulative FLOPs (x-axis) to empirical risk (y-axis).
Summary Metrics:
- C@ $\tau$ (Compute to $\tau$ -risk): The average cumulative FLOPs required for an attack to first reach a specific risk threshold (e.g., 50%). Higher values indicate greater robustness.
- Average Efficiency (AE): The normalized expected risk per FLOP ( $CAURC / \bar{C}_{max}$ ). Lower values indicate a model is harder to exploit per unit of compute.

Experimental Setup

The framework was applied across:

Models: Ten models spanning three families (Tulu3, Qwen2.5, Qwen3) and various training stages (Base, SFT, DPO, RLVR, SafeRL).
Attacks: Three distinct strategies:
- JailBroken: Template-based (random obfuscation).
- PAIR: Black-box iterative refinement using an attacker LLM.
- GCG: White-box gradient-based coordinate gradient optimization.
Benchmarks: HarmBench (200 behaviors) and JailbreakBench (100 behaviors).

Key Results

The compute-aware analysis revealed patterns invisible to standard ASR metrics:

Non-Monotonic Effects of Alignment: Alignment training does not linearly increase robustness. In the Tulu3 family, the Supervised Fine-Tuning (SFT) stage often exhibited the highest robustness (e.g., GCG and PAIR failed to reach 50% risk within budget). Subsequent alignment via Direct Preference Optimization (DPO) and Reinforcement Learning with Verifiable Rewards (RLVR) often reduced robustness, particularly against template-based attacks, despite higher ASR in some cases.
Asymmetric Scaling Benefits: Increasing model size (e.g., Qwen2.5 from 0.5B to 7B) significantly increased the compute cost for gradient-based attacks (20 $\times$ increase in C@0.5) but offered minimal protection against cheap template-based attacks (only 2.8 $\times$ increase). Standard ASR metrics masked this disparity, showing similar success rates across scales.
Surrogate Attack Transfer: Gradient-based attacks (GCG) optimized on a small, open-weight surrogate model (Qwen2.5-0.5B) successfully transferred to a larger, closed-weight target (Qwen3-8B) at a fraction of the original cost. While the transfer did not always reach high-risk regimes, it demonstrated that attackers can reduce costs by optimizing on surrogates.
Heterogeneity Across Harm Categories: Within a single model, the compute cost to breach safety varied by up to $\approx 5\times$ across different harm categories (e.g., harassment vs. cybercrime). Safety-aligned RL increased aggregate costs but left certain categories (e.g., cybercrime, illegal activities) disproportionately accessible or even more efficiently exploitable (higher AE) than the base model.
Safety-RL Trade-offs: While Safety-RL generally raised the aggregate adversarial cost, it introduced specific vulnerabilities. For instance, under GCG, the SafeRL model was strictly less robust than its base counterpart, suggesting a mismatch between the distribution of RL training prompts and the token sequences discovered by gradient optimization.

Significance and Claims

The paper argues that robustness is a function of computational cost, not just possibility. By adopting a security-centric "work factor" perspective, the authors claim:

Operational Relevance: A vulnerability is only operationally relevant if it is exploitable within a realistic adversary budget. Raising the computational cost floor can be as effective as lowering success rates.
Limitations of Static Metrics: Standard ASR scores under fixed query budgets systematically overestimate robustness by ignoring the varying efficiency of different attack classes and the non-linear effects of model scaling and alignment.
Unified Comparison: The proposed framework enables fair comparisons across diverse attack strategies (white-box vs. black-box) and model configurations by normalizing them against a shared compute axis.
Nuanced Safety Assessment: Safety tuning is not a monolithic improvement; it creates uneven protection landscapes where some categories become significantly harder to breach while others remain vulnerable or become more efficient to attack.

The authors conclude that cost-conscious adversarial assessment is necessary to accurately gauge the true risk of LLM deployment, as models appearing robust under query-count metrics may fail catastrophically when evaluated under compute-aware pressure.