Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attacks

This paper introduces a comprehensive 4×6 Target × Technique taxonomy and benchmark coverage audit framework, derived from 932 security studies, which reveals that current LLM attack benchmarks collectively cover at most 25% of the threat surface and suffer from significant evaluation gaps and naming fragmentation.

The Gist

The Big Picture: The "Safety Map" Problem

Imagine the world of Large Language Models (LLMs) as a massive, bustling city. The people building these models are like city planners trying to make sure the city is safe from criminals (hackers).

For a few years, researchers have been finding new ways for criminals to break into the city. But here's the problem: everyone is using different maps, different names for the same crimes, and different ways of measuring how safe the city is. Some researchers say, "We checked the banks!" while others say, "We checked the power plants!" But nobody has checked if the whole city is safe.

This paper is like a team of cartographers who decided to stop arguing about maps and instead build one giant, master map of every possible way to attack an AI. Then, they checked the security guards' (the "benchmarks") current patrol routes to see if they are actually covering the whole city.

The shocking finding? The security guards are only patrolling a tiny, crowded neighborhood (about 25% of the city), while huge, dangerous districts are completely empty and unguarded.

---

1. The Master Map (The Taxonomy)

The researchers looked at 932 scientific papers published between 2023 and 2026. They found over 6,300 mentions of different attacks.

The Naming Chaos:
Imagine if one type of car theft was called "Hotwiring" in one city, "Keyless Entry Theft" in another, and "The Silent Heist" in a third. That's what was happening with AI attacks.

• The Analogy: The famous "GCG" attack (a way to trick AI) was called 29 different names across 376 papers.
• The Fix: The team created a standard dictionary (a taxonomy) with 507 specific "leaves" (categories). They cleaned up the mess, merging the 29 names into one, so everyone speaks the same language.

The Structure:
They organized these attacks into a 4x6 Grid (Matrix):

• The Rows (The Goal): What does the criminal want?
  1. Safety Bypass: Making the AI say something bad (like hate speech).
  2. System Hijacking: Making the AI do something bad (like delete files or send money).
  3. Information Theft: Stealing secrets from the AI's memory.
  4. Service Disruption: Making the AI so slow or expensive to use that it crashes (like a traffic jam).
• The Columns (The Method): How do they do it?
  • Using confusing words, lying to the AI, hiding code, or attacking the AI's internal brain.

---

2. The Security Check (The Benchmark Audit)

The researchers took the three most famous "security tests" used by the industry (HarmBench, InjecAgent, and AgentDojo) and plotted them onto their Master Map.

The Result:

• Zero Overlap: The three tests don't even check the same things. They are like three different fire departments: one only checks kitchens, one only checks garages, and one only checks basements. None of them check the whole house.
• The Coverage Gap: Together, these tests only cover 25% of the Master Map.
• The Blind Spots:
  • The "Service Disruption" District: This is where attacks try to make the AI crash or cost a fortune to run. Zero of the major tests check this.
  • The "Model Internals" District: This is where hackers tweak the AI's internal brain directly. Zero tests check this either.

Why this matters:
The paper found that attacks in these "blind spot" areas are actually very effective. Some can make the AI run 46 times slower or cost 100 times more to use, yet no one is testing for them. It's like having a fire alarm that only beeps when you burn toast, but stays silent when the whole house is on fire.

---

3. The "Catch-All" Problem

The researchers noticed that because the field is moving so fast, many scientists just throw new attacks into a "Miscellaneous" bucket because they don't have a specific name yet.

• The Analogy: It's like a library where 60% of the new books are just shoved into a box labeled "Stuff."
• The Impact: This makes it hard to know how many real new attacks are happening. The paper suggests we need better names so we can actually count the threats.

---

4. What They Did With This

Instead of just pointing out the holes, the researchers released their tools to the public:

1. The Master Map: A downloadable list of all 507 attack types.
2. The Audit Report: A clear chart showing exactly which parts of the map are being tested and which are ignored.
3. A Blueprint for New Tests: They showed how to use their map to build a new test specifically for the "blind spots" (like the Service Disruption district).

The Bottom Line

The paper argues that we are currently "testing for the wrong things." We are very good at testing if an AI will say something rude, but we are terrible at testing if an AI will crash, steal data, or get hijacked to do real-world damage.

The Takeaway: Just because a security test says an AI is "safe" doesn't mean it is. It just means it passed the specific, narrow test it was given. To truly be safe, we need to expand our testing to cover the whole city, not just the neighborhood we're currently comfortable with.

Technical Summary

Technical Summary: Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attacks

Problem Statement

The rapid proliferation of Large Language Model (LLM) security research has created a fragmented landscape where the threat surface is poorly understood and evaluation infrastructure is misaligned with actual risks. The authors identify three interconnected problems:

1. Lack of Quantitative Taxonomy: Existing surveys cover only 50–150 papers manually, failing to scale with the field's growth (now exceeding 932 unique papers with 6,300+ attack mentions in the authors' corpus).
2. Naming Fragmentation: Attacks are frequently rediscovered with slight variations, leading to multiple surface names for the same technique (e.g., the GCG attack appears under 29 distinct names across 376 papers), which impedes reproducibility and meta-analysis.
3. Evaluation Blind Spots: Current benchmarks lack "external validity." While individual benchmarks may be internally consistent, it is unclear whether the collective evaluation infrastructure covers the full attack surface. Without a clear map of the threat landscape, defense frameworks may leave users vulnerable to undefined degrees.

Methodology

The authors constructed a data-informed framework to audit the coverage of LLM attack benchmarks against the actual threat surface.

Corpus and Extraction

• Data Source: A corpus of 932 arXiv papers published between H1 2023 and H1 2026, sourced from the Promptfoo LLM Security Database.
• Extraction Pipeline: Utilized Gemini 3.1 Pro to extract inference-time attacks (excluding training-phase attacks like data poisoning). The process yielded 6,366 initial attack mentions.
• Normalization: Through case normalization, alias merging, and deduplication, the mentions were reduced to 2,521 unique attack groups. This process involved manual curation of over 50 variant clusters to resolve naming fragmentation.
• Validation: A three-tier classification system (Direct lookup, Fuzzy matching, Semantic matching) was employed. Human evaluation of 200 stratified samples yielded 92.0% accuracy, and inter-annotator agreement at the matrix-cell level was 86.1%.

Taxonomy and Matrix Construction

The authors developed two complementary structures:

1. 507-Leaf Taxonomy: A hierarchical structure organizing attacks into five inference-time categories (Jailbreaking, Prompt Injection, Information Disclosure, Service Disruption, Decoding Manipulation) and one orthogonal category for Attack Generation methods. 401 leaves are populated by the corpus; 106 are placeholders derived from threat modeling.
2. 4×6 Target × Technique Matrix: Grounded in the STRIDE threat model, this matrix maps attacks based on:
   • Target Rows (4): Safety & Alignment Bypass, System & Tool Hijacking, Information Exfiltration, and Service Disruption.
   • Technique Columns (6): Instructional, Persuasion, Obfuscation, Cross-Modal, Indirect Injection, and Model Internals.
   • Classification Logic: Attacks are mapped to cells based on the specific defense layer they bypass (e.g., input filters vs. alignment training) and the adversarial objective.

Benchmark Coverage Audit

The authors mapped six public benchmarks (HarmBench, InjecAgent, AgentDojo, AdvBench, JailbreakBench, StrongREJECT) onto the matrix by matching evaluated methods to their corresponding taxonomy leaves and matrix cells.

Key Results

1. Extreme Concentration and Fragmentation

• Research Imbalance: Safety & Alignment Bypass dominates the literature, accounting for 74.4% of all taxonomy leaves (377/507).
• Naming Chaos: The GCG attack is the most referenced yet inconsistently named, appearing in 29 forms. Sixty taxonomy leaves function as "catch-alls," absorbing 10+ distinct techniques each, inflating apparent novelty and complicating comparison.
• Temporal Trends: While Safety Bypass remains dominant, its share of unique attack groups declined from 92% to 74% between H2 2023 and H1 2026, with a corresponding rise in System Hijacking and Information Exfiltration. However, the novelty rate stabilized at ~18%, suggesting the field is maturing but still expanding.

2. Critical Evaluation Gaps

The audit reveals that the three primary benchmarks (HarmBench, InjecAgent, AgentDojo) occupy non-overlapping cells, collectively covering at most 25% of the 24-cell matrix.

• Zero Overlap: HarmBench focuses exclusively on Safety Bypass (using Obfuscation, Persuasion, Instructional, and Cross-Modal techniques), while InjecAgent and AgentDojo focus on Indirect Injection for System Hijacking and Information Exfiltration.
• Major Blind Spots:
  • Service Disruption (Entire Row): No benchmark evaluates attacks targeting availability (e.g., resource exhaustion, denial-of-wallet). Despite published attacks achieving 46× token amplification and 100× latency increases, this category has zero standardized evaluation.
  • Model Internals (Entire Column): No benchmark evaluates mechanisms like activation steering, logit suppression, or side-channel probing. Attacks in this category (e.g., logit suppression) have demonstrated 96% attack success rates on standard behaviors, yet remain untested by existing infrastructure.
  • Safety Bypass × Indirect Injection: No benchmark tests whether indirect injection via external data can bypass safety alignment to produce harmful content.

3. Structural Properties

The paper identifies that the field suffers from "dataset misalignment" (benchmarks sampling too narrowly from the technique space) and "metrics misalignment" (lack of metrics for cost, latency, or white-box access required for certain attacks). The concentration of research on Safety Bypass creates systematic blind spots where attacks are most costly in production environments.

Significance and Claims

The paper claims its primary contribution is not merely the finding of gaps, but the provision of a reusable framework for benchmark-external validation. By shifting from assessing whether a single benchmark measures what it claims (internal validity) to auditing whether the field's evaluation infrastructure collectively covers the threat surface (external validity), the authors provide a mechanism to track evaluation gaps over time.

The authors assert that:

• The current evaluation landscape is structurally incomplete, leaving entire STRIDE threat categories (Service Disruption, Model Internals) without standardized public evaluation.
• The released taxonomy, attack records, and coverage mappings serve as extensible artifacts. As new benchmarks emerge, they can be mapped to the same matrix, allowing the community to objectively determine if coverage gaps are closing.
• Addressing these gaps requires new measurement infrastructure (e.g., per-request resource accounting for Service Disruption) that current prompt-level benchmarks cannot provide.

The work concludes that the concentration of benchmarks on a narrow slice of the attack surface is a structural issue, not a coincidence, and that closing these gaps is essential for the maturity of LLM security evaluation.