💬 NLP

Capability-Based Scaling Trends for LLM-Based Red-Teaming

This paper investigates the "weak-to-strong" challenge in LLM red-teaming by demonstrating that attack success rates decline sharply as the target model's capability surpasses that of the attacker, leading to the proposal of a "jailbreaking scaling curve" to predict vulnerability based on the capability gap.

Original authors: Alexander Panfilov, Paul Kassianik, Maksym Andriushchenko, Jonas Geiping

Published 2026-02-10

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Alexander Panfilov, Paul Kassianik, Maksym Andriushchenko, Jonas Geiping

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a game of "Hide and Seek" between two players: the Seeker (the Attacker) and the Hider (the Target).

In this game, the Hider is a high-tech vault (an AI model) designed to keep secrets and follow rules. The Seeker is a clever locksmith (a red-teaming AI) trying to find a way to trick the vault into opening.

This research paper looks at what happens to this game as both players get "smarter" (more capable). Here is the breakdown of their findings:

1. The "Smartness" Arms Race

The researchers found that as the Seeker gets smarter, they get much better at finding cracks in the vault. It’s not just a little bit better; it’s a massive leap. If you give a locksmith a better set of tools and a sharper brain, they don't just find one more way into the vault—they find dozens.

The takeaway: To keep AI safe, we can't just build better vaults; we have to realize that the "locksmiths" are getting exponentially more dangerous every day.

2. The "Wall of Intelligence" (The Capability Gap)

This is the most important part of the paper. They discovered that the success of the attack depends on the gap between the two players.

Think of it like a climbing competition:

If the Seeker is a professional athlete and the Hider is a toddler, the Seeker will scale the wall instantly. (High Attack Success).
If the Seeker is an athlete and the Hider is also a professional athlete, the Seeker will struggle, and might never get over the wall. (Low Attack Success).

The researchers found a "tipping point." As soon as the Hider (the AI) becomes significantly smarter than the Seeker, the Seeker’s success rate doesn't just drop—it plummets. It’s like the wall suddenly grows from 5 feet to 50 feet tall.

3. The "Human Problem" (The Scary Forecast)

Here is the "spooky" part of the paper. The researchers used the data to predict the future.

They modeled Humans as Seekers. Currently, humans are very good at "social engineering"—tricking AI by being persuasive, emotional, or clever. But the researchers' math suggests that as AI models continue to get smarter, they will eventually become so "intelligent" that they can see through any human trick.

The Metaphor: Imagine a master magician trying to trick a super-computer. Eventually, the computer will be so good at logic and pattern recognition that the magician’s "sleight of hand" will look like a slow-motion movie to the computer. The human will become "too simple" to trick the machine.

4. The "Psychology" Secret

The researchers noticed something strange: the attackers weren't just good at math or coding; they were good at social science.

The most successful "locksmiths" were the ones that understood psychology, persuasion, and manipulation. They didn't just try to break the lock; they tried to convince the lock that it wanted to be open.

The takeaway: When we build AI, we shouldn't just teach it math and science; we need to teach it how to resist being "gaslit" or manipulated by clever psychological tricks.

Summary in a Nutshell:

Smart Attackers are much more dangerous than weak ones.
Smart Targets are much harder to break, but only if they are significantly smarter than the attacker.
Humans might eventually become "too dumb" to jailbreak the super-intelligent AIs of the future.
Persuasion is the ultimate weapon in the world of AI hacking.

Technical Summary: Capability-Based Scaling Trends for LLM-Based Red-Teaming

Published at: ICLR 2026
Authors: Alexander Panfilov, Paul Kassianik, Maksym Andriushchenko, Jonas Geiping

1. Problem Statement

As Large Language Models (LLMs) evolve from simple text generators into agentic systems with advanced reasoning, traditional safety evaluations (red-teaming) face a looming "weak-to-strong" challenge. Current red-teaming often relies on human ingenuity or prompt engineering to "outsmart" models. However, as target models become significantly more capable than the attackers (including humans), the effectiveness of these traditional methods may collapse. The authors seek to quantify this shift by asking: At what capability gap does human-like red-teaming become ineffective?

2. Methodology

The researchers framed red-teaming as a function of the capability gap between the attacker and the target.

Experimental Framework: They evaluated over 600 attacker-target pairs using four state-of-the-art LLM-based jailbreaking attacks: PAIR, TAP, PAP, and Crescendo.
Capability Proxy: Model capabilities were measured using MMLU-Pro scores. To ensure a fair comparison, they used "unlocked" (safety-untuned) versions of open-source models to serve as attackers, preventing the attacker's own safety guardrails from confounding the results.
Target Diversity: The study included 29 target models across various families (Llama, Mistral, Qwen, Vicuna) and closed-source frontier models (Gemini, GPT-4.1, o-series, Claude-3.7 Sonnet).
Modeling Approach:
- They modeled the Attack Success Rate (ASR) using a sigmoid function in logit space.
- The Capability Gap ( $\delta$ ) was formally defined using the difference in logit-transformed MMLU-Pro scores: $\delta_{a \to t} = \text{logit}(a_{\text{MMLU-Pro}}) - \text{logit}(t_{\text{MMLU-Pro}})$ .
- They used Bayesian and Bootstrapped linear regression to derive scaling curves and forecast future trends.

3. Key Contributions

The Capability-Based Scaling Curve: The paper introduces a predictive mathematical model that relates the ASR to the capability gap between the attacker and the target.
Identification of Scaling Trends: They established that red-teaming success is not just about the absolute strength of the attacker, but the relative difference in intelligence between the two models.
Social Science Correlation: They discovered that an attacker's ability to jailbreak is more strongly correlated with social science capabilities (psychology, philosophy, etc.) than with STEM knowledge, suggesting that "persuasion" is a core component of successful red-teaming.
Human Red-Teaming Forecast: They provided a quantitative forecast showing the inevitable decline of human red-teaming effectiveness as models surpass human-level reasoning.

4. Key Results

Dual Scaling Trends:
- Stronger models are better attackers: ASR increases almost linearly with the attacker's general capability ( $\rho > 0.88$ ).
- Stronger models are harder targets: ASR drops sharply once the target's capability exceeds the attacker's.
Predictability: For most model families (notably Qwen2.5), the ASR follows a highly predictable sigmoid curve based on the capability gap. The Qwen2.5 scaling curve generalized well even to closed-source frontier models.
The "Human Ineffectiveness" Forecast: By modeling a human red-teamer (with an assumed MMLU-Pro of ~0.898), the authors forecast that as target models move toward "superhuman" capabilities (e.g., hypothetical GPT-6 or o5), the ASR for human attackers will trend toward zero.
Attack Comparison: TAP was identified as the most effective attacking scaffold, while PAP was the least effective but showed more resilience against highly safeguarded models.

5. Significance and Implications

For Model Providers: The findings emphasize that safety tuning must go beyond technical "hard science" robustness and specifically address persuasive and manipulative abilities. Furthermore, providers must benchmark the attacking potential of new models before release.
For AI Safety/Red-Teaming: The paper signals the end of the era where human-led, manual red-teaming can keep pace with model advancement. It advocates for a shift toward scalable, automated, LLM-based red-teaming to probe future frontier models.
Security Risks: The research highlights a growing risk: as open-source models become increasingly capable, they will be able to jailbreak existing, deployed LLM systems more effectively, creating a "security arms race."