Technical Summary: SRE-Bench

Problem Statement

While AI agents are rapidly advancing in cybersecurity tasks involving source code, a critical gap remains in their ability to analyze software distributed solely as binaries. This is the reality for high-value defensive targets (proprietary enterprise software, firmware) and offensive threats (obfuscated malware), where source code is unavailable. Analyzing these binaries requires Reverse Engineering (RE): recovering high-level program semantics from opaque byte representations.

Current benchmarks for agentic RE suffer from two fundamental flaws that prevent them from accurately measuring real-world capability:

1. Data Contamination: Many benchmarks rely on public source code or CTF challenges. If a model has seen the target in its pretraining data, it can bypass the actual analysis by recognizing the program's purpose, inflating performance metrics. In RE, even coarse knowledge of a target's intent is damaging as it provides top-down guidance that shortcuts the recovery process.
2. Lack of Realistic Scale and Protection: Existing benchmarks often use "toy" programs or CTF-style challenges that lack the complexity of real-world software (thousands of lines of code) and the sophisticated anti-analysis protections (obfuscation, packing, anti-debugging) found in the wild.

Consequently, there is no rigorous testbed to determine if strong source-code security capabilities transfer to binary analysis.

Methodology: SRE-Bench Construction

To address these gaps, the authors introduce SRE-Bench, the first realistic, contamination-free RE benchmark. The construction methodology prioritizes "clean-room" development and realistic complexity over convenience.

1. Clean-Room Development

• Zero Contamination: All 19 target programs, the evaluation infrastructure, and the protection suite were developed from scratch by RE experts with an average of six years of experience. No code was derived from public projects.
• Private Specifications: Each program was implemented based on private design specifications that are never released.
• Adversarial Auditing: The authors employed iterative "reward-hacking" audits, pitting agents against in-progress instances to identify and patch shortcuts (e.g., readable strings revealing hidden behaviors) that allowed credit without genuine RE.

2. Real-World Scale and Domains

• Scale: The benchmark comprises 19 programs averaging 16,915.8 Lines of Code (LoC), totaling over 320K LoC. This is 30–470× larger than prior RE benchmarks.
• Domains: The programs span five representative RE domains identified from industry discussions:
  • Network Protocols: Proprietary encrypted client-server stacks requiring state machine reconstruction.
  • Games: A 20-floor terminal roguelike with hidden "Easter egg" behaviors triggered by specific logic.
  • File Formats: A custom file compressor/encoder requiring the agent to reverse-engineer a multi-stage pipeline to decode archives.
  • Malware: A synthetic, safe Linux implant simulating six malware families (e.g., ransomware, C2) where the task is defensive cleanup.
  • Firmware: Bare-metal firmware for a fictional microcontroller requiring control via a JTAG-style emulator.

3. Anti-Analysis Protection Suite

To mimic real-world defenses, the authors built a custom protection suite (27K LoC) featuring 44 distinct primitives across nine families. Notably, more than half of these primitives have no public implementations. Key features include:

• Per-page Authenticated Encryption: Pages are encrypted with keys derived from execution context, preventing memory dumps.
• On-Demand Decryption: Code is decrypted page-by-page only when executed, minimizing plaintext residency.
• Measurement-Keyed Anti-Debugging: Decryption keys are corrupted if debugging signals (e.g., ptrace, breakpoints) are detected.
• Loader Virtualization: Critical loading logic is executed in a custom VM with obfuscated opcodes.
• Anti-Re-hosting: Prevents attackers from extracting decryption routines to run in a separate process.

4. Evaluation Pipeline

• Instances: The 19 programs were compiled into 262 binary instances (varying by optimization, symbol stripping, linking, and protection presets).
• Tasks: Each instance defines 6 deterministically graded tasks (e.g., recovering a protocol handshake, triggering a hidden game behavior), resulting in 1,572 total tasks.
• Models: Five frontier LLMs (GPT-5.6-sol, Claude-Opus-5, GPT-5.5, Grok-4.5, GLM-5.2) were evaluated using a standardized harness with a bash-only toolset and standard RE tools (Ghidra, radare2, etc.). The total evaluation cost was $31.4K.

Key Results

The evaluation reveals that agentic RE remains largely unsolved, even for the most advanced models.

• Performance Ceiling: The strongest model, GPT-5.6-sol, achieved a mean score of 3.69/6 (61.4% of the maximum possible score) and fully solved only 31.5% of instances. The weakest model (GLM-5.2) fully solved 0% of instances.
• Impact of Protection: The in-house protection suite was the primary bottleneck. It halved the performance of GPT-5.6-sol (from 4.69 to 2.50) and drove all other models to near-zero performance.
• Divergence from Human Analysts:
  • Optimization & Linking: Unlike human analysts, who struggle with compiler optimizations and static linking, current agents are relatively insensitive to these factors.
  • Symbol Stripping: Agents rely heavily on lexical anchors (function/variable names). Stripping symbols caused a significant performance drop (e.g., GPT-5.6-sol dropped 0.48 points), suggesting agents prioritize naming over instruction-level reasoning.
• Contamination vs. Scale: Ablation studies confirmed that both factors are essential.
  • A clean-room toy program was solved easily by all models.
  • A publicly derived program (even with modifications) was solved easily by all models.
  • Only the combination of private provenance and real-world scale created a difficulty gap that separated the models.

Significance and Claims

The paper claims SRE-Bench establishes a new standard for evaluating agentic cybersecurity by proving that:

1. Source-code capability does not transfer to binary analysis: Strong performance on source-code benchmarks does not guarantee competence in reverse engineering binaries.
2. Contamination control is non-negotiable: Benchmarks must strictly prevent models from recognizing targets; otherwise, metrics reflect memory retrieval rather than analysis.
3. Scale matters: Real-world complexity (thousands of lines of code) is a prerequisite for meaningful evaluation; toy programs fail to predict performance on real targets.
4. Current agents are brittle: While agents can handle some structural analysis, they fail significantly when faced with realistic anti-analysis protections and lack the robust instruction-level reasoning required for fully obfuscated binaries.

SRE-Bench serves as a rigorous, contamination-free testbed to measure progress in this critical frontier, highlighting that the gap between source-code understanding and binary analysis remains a significant challenge for the field.