Technical Summary: AMT-X

1. Problem Statement

Current safety evaluation for Large Language Models (LLMs) relies heavily on single-turn attack datasets and single-judge scoring. This approach systematically underestimates risk from adaptive multi-turn adversaries and fails to distinguish between "nominal" successes (outputs that pass a threshold but lack operational utility) and "fully actionable" harm (outputs containing complete, real-world operational details).

The authors identify three specific structural limitations in existing automated red-teaming frameworks (e.g., PAIR, TAP, Crescendo, X-Teaming):

1. Lack of Reproducibility and Attribution: Attacks are often improvised turn-by-turn or rely on free-form plans, making trajectories non-reproducible and preventing the isolation of which specific components drive success.
2. Oversimplified Success Metrics: Reporting a single scalar success rate obscures the difference between partial and complete harm. A model might provide a partial answer that passes a judge's threshold but lacks the specific parameters or steps required for real-world exploitation.
3. Single-Judge Bias: Relying on a single LLM judge introduces documented biases (positional, verbosity, and self-preference) that can be exploited by adaptive attackers.

2. Methodology: AMT-X Framework

The authors propose AMT-X (Adaptive Multi-Turn Exploitation), a phase-structured red-teaming framework designed to address the limitations above. The system operates as a black-box attacker interacting with a target LLM via a text interface.

2.1 Core Architecture

AMT-X casts the attack as an explicit, reproducible multi-phase state machine driven by semantic signals from the victim, rather than ad-hoc escalation.

• Phase Controller (P0–P4): The attack progresses through five distinct phases, each with a specific objective, a configurable turn budget, and a transition rule:
  • P0 (Reconnaissance): Maps the victim's domain, capabilities, and terminology.
  • P1 (Boundary Probing): Elicits a clear refusal to establish a baseline and record the rationale.
  • P2 (Contradiction Identification): Highlights the gap between the victim's claimed capabilities (from P0) and its refusal (from P1).
  • P3 (Exploit Reframing): Reframes the harmful goal as a necessary component of a legitimate task (e.g., a compliance risk assessment).
  • P4 (Target Extraction): Requests the complete, structured, and actionable output.
• Response Analyzer: Converts the victim's free-text response into a deterministic feature vector (refusal level, cooperation level, disclosure level, phase indicators) to drive phase transitions and technique selection.
• Technique Library: A fixed library of 31 concrete attack techniques (e.g., role-play, encoding, contradiction pointing). Selection is dynamic, using a Multi-Armed Bandit (UCB1) algorithm to balance exploration of new tactics with exploitation of proven ones, conditioned on the victim's current stance.

2.2 Evaluation Backend

To mitigate judge bias, AMT-X replaces single-judge scoring with a Multi-Role Debate Jury.

• Jury Composition: Three distinct model families (Grader, Critic, Defender) deliberate before voting.
• Checklist-Gated Scoring: The jury uses phase-conditioned binary checklists. Crucially, the evaluation distinguishes between two success metrics:
  1. Overall ASR (Lenient): Triggered if at least one critical actionability item passes (e.g., a partial answer).
  2. Full ASR (Strict): Triggered only if all critical items pass, requiring the output to be complete, contain real values (not placeholders), and provide specific operational details (steps, parameters, quantities) that a user could act upon.

3. Key Contributions

1. Reproducible State Machine: AMT-X transforms red-teaming from an improvised process into a structured, reproducible program where success can be attributed to specific phase depths and techniques.
2. Dual-Metric Evaluation: The framework introduces a "gap" metric between Overall ASR and Full ASR, explicitly quantifying how many "successful" attacks are merely partially actionable versus fully operational.
3. Phase-Depth Ablation: The state-machine structure allows for controlled ablation studies to determine how much conversational depth is required to achieve actionable harm.
4. Bias-Mitigated Scoring: The use of a multi-role, multi-model jury with phase-conditioned rubrics addresses the positional and self-preference biases inherent in single-judge evaluations.

4. Experimental Results

The authors evaluated AMT-X against six frontier victim models (including GPT-5.2, GPT-4.1, Gemini-2.5-Pro, Gemma-3-12B, Grok-4, and Mistral-Large) across seven Moderation sub-categories (e.g., Hate, CBRN, Violent Crime), creating a 42-attack grid.

• High Success Rates: Under a lenient score threshold, AMT-X achieved an Overall ASR of 97.6–100% across the victim models.
• Significant Actionability Gap: When applying the stricter gate requiring complete, real, and operational detail, the Full ASR dropped to 66.7–78.6%. This reveals a gap of up to 33 percentage points between attacks that merely "pass" and those that are fully actionable.
• Depth Dependence: A phase-depth ablation showed that success is strongly dependent on later phases:
  • Capping the attack at P1 (Boundary Probing) resulted in an Overall ASR of only 28.6%.
  • Capping at P2 (Contradiction) raised it to 35.7%.
  • Allowing P3 (Reframing) recovered the Overall ASR to 97.6%, indicating that the "face-saving reframe" is the critical trigger that unlocks extraction-grade scoring, even if the final extraction happens in P4.

5. Significance and Claims

The paper claims that AMT-X provides a more honest and reproducible assessment of LLM safety. Its primary significance lies in demonstrating that:

• Current benchmarks may overstate safety: A single success rate is insufficient; a large portion of "successful" jailbreaks in existing literature may be partially actionable but lack the operational detail to cause real-world harm.
• Adaptive depth is critical: The ability to engage in multi-turn, phase-structured dialogue is a dominant factor in bypassing safety alignment. The "foot-in-the-door" effect, facilitated by the P3 reframing phase, is a key vulnerability.
• Evaluation methodology must evolve: Safety evaluation must move beyond single-turn prompts and single-judge scores to include multi-phase state machines, multi-role juries, and metrics that distinguish between partial and full actionability.

The authors explicitly state that AMT-X does not claim to be a novel zero-day exploit but rather a systematic measurement of a known vulnerability (susceptibility to adaptive multi-turn attacks). They emphasize that the framework is designed for defensive purposes to inform safety-evaluation methodologies and that no real-world harm was generated or used.