Technical Summary: CryptanalysisBench

Problem Statement

Cryptanalysis—the systematic analysis and breaking of cryptographic schemes—sits at the intersection of mathematical reasoning and cybersecurity. While Large Language Models (LLMs) have shown rapid advancement in both domains, their capability to perform genuine cryptanalysis on real-world primitives remains an open question. Unlike theoretical proofs or toy ciphers, practical cryptanalysis carries high stakes: a single successful break can invalidate systems built on a primitive and retroactively compromise past security. Furthermore, cryptanalysis offers a unique benchmarking opportunity because its results are strictly verifiable: a practical attack either wins a formally specified security game or it does not. The paper asks whether frontier LLMs can independently discover vulnerabilities, design flaws, and end-to-end attacks against cryptographic primitives that have undergone sustained expert scrutiny.

Methodology

The authors introduce CryptanalysisBench, a benchmark comprising 191 tasks across six families of cryptographic primitives (block ciphers, hash functions, AEADs, KEMs, PKEs, and digital signatures). The tasks are drawn primarily from four NIST standardization competitions (AES, SHA-3, Lightweight Cryptography, and Post-Quantum Cryptography) and include widely studied production ciphers.

The benchmark is structured into three tiers:

1. Tier 1 (Known Breaks): 49 algorithms with known practical breaks (e.g., FEAL-4, eliminated NIST candidates). This tier calibrates current capabilities and helps distinguish between memorization of known attacks and genuine rediscovery.
2. Tier 2 (No Known Practical Breaks): 142 algorithms for which no practical attack is known. Since full-strength attacks are often infeasible to verify, the authors construct scaled-down variants by reducing key sizes, round counts, or other parameters to bring potential attacks within a feasible computational range.
3. Challenge Set: A collection of production-grade schemes (e.g., 7-round AES, 7-round ChaCha) representing the current frontier of cryptanalytic knowledge, where non-trivial attacks are known but remain computationally infeasible to run in practice.

Evaluation Framework:
The benchmark utilizes the Harbor agent evaluation framework. Agents operate in an isolated sandbox with read access to the algorithm's source code. They interact with a separate "challenger" container via an HTTP API, which holds all secret material and answers oracle queries according to formal security games (e.g., Target Key Recovery, IND-CPA/CCA, forgery, collision).

• Verification: Success is determined by running the agent's submitted attack script against fresh game instances. For statistical games (IND-CPA/CCA), a submission is successful only if it wins at least 17 out of 20 independent runs (approx. 0.1% false-positive rate under random guessing).
• Constraints: Agents are subject to fixed time and token budgets (typically 2 hours for exploration and 2 hours for verification). Internet access is permitted for tool installation but explicitly forbidden for searching the target algorithm or known attacks.

Key Contributions

1. CryptanalysisBench: The first systematic benchmark of 191 cryptanalysis tasks against real cryptographic primitives with automatic, game-based verification.
2. Systematic Evaluation: The first evaluation of five frontier models (Claude Opus 4.8, Sonnet 5, Mythos 5, GPT-5.5, and open-weights GLM-5.2) on primitive-level cryptanalysis.
3. Novel Cryptanalysis: The discovery of previously unreported attacks, including a full 128-bit key-recovery attack on the unmodified SpoC AEAD and an error in the published CCA-security proof of KINDI.
4. Trace-Level Audit: A methodology to distinguish between "paper recall" (memorization), "source-level rediscovery," and "novel attack routes" by analyzing agent traces and using control cases like AIMer (whose efficient attack was published after the training cutoffs of several evaluated models).
5. Roadmap: A framework for tracking AI cryptanalysis capabilities, including a procedure for tightening Tier 2 as models improve.

Results

The evaluation of five frontier models yielded the following findings:

• Tier 1 Performance: All models solved the majority of Tier 1 tasks. Success rates ranged from 65.3% (GLM-5.2) to 85.7% (Claude Mythos 5). Failures were predominantly due to implementation errors (e.g., failing to convert a correct analysis into working code) rather than a lack of identifying the correct vulnerability.
• Novel Discoveries: Models produced genuinely novel cryptanalysis.
  • SpoC: Two models (Mythos 5 and Sonnet 5) independently discovered a full key-recovery attack on the unmodified SpoC AEAD using only two oracle queries, exploiting a design flaw in how empty messages are handled.
  • KINDI: Mythos 5 identified an error in the scheme's published CCA-security proof, exploiting a decryption-reaction vulnerability to recover the secret key.
  • LIMA & DAGS: Models exploited underspecifications in the security definitions that allowed insecure but faithful implementations.
• Tier 2 Performance: Success rates were significantly lower, ranging from 24 to 61 distinct algorithms broken across all scaled-down variants. Genuine cryptanalysis (design-level flaws) was concentrated in the "easy" scaled-down variants and thinned out for "hard" variants and full-strength schemes. Most full-strength breaks were attributed to implementation bugs or specification gaps rather than fundamental design breaks.
• Memorization vs. Rediscovery: Trace-level analysis suggests that while models often recall known attack families, they frequently re-derive them from source code. In the AIMer case study, no model reproduced the specific elimination framework from the post-training-cutoff paper, instead relying on elementary linearization techniques.
• Impact of Compute: Increasing the time budget from 2 hours to 16 hours significantly improved performance (e.g., Mythos 5's Tier 1 success rose from 85.7% to 95.9%), indicating that the primary bottleneck is often the implementation and execution of the attack rather than the initial cryptanalytic insight.

Significance and Claims

The paper claims that LLMs are already capable of performing cryptanalysis that matches, and in specific instances exceeds, the published state of the art. The significance of this work is twofold:

1. Defensive Utility: AI with moderate cryptanalytic ability can assist the cryptography community in stress-testing standardization candidates against large numbers of automated attacks before deployment, potentially identifying flaws that human experts might miss.
2. Offensive Risk: AI that matches or exceeds human experts against deployed standards represents a serious offensive capability against infrastructure that cannot be quickly patched.

The authors position CryptanalysisBench not as a final verdict, but as a "scaffold" and a "forecasting tool." They argue that the attacks surfaced by the benchmark are an "early snapshot" of a fast-moving frontier. The benchmark is designed to track when AI cryptanalysis becomes a serious factor and to provide a mechanism for tightening security evaluations as model capabilities evolve. The paper remains modest about the saturation of Tier 2, noting that many algorithms may simply lack practical attacks, and that the challenge set remains far from saturated, serving as a long-horizon target for future capability growth.