Benchmarking Mythos-Linked Bug Rediscovery

Technical Summary: Benchmarking Mythos-Linked Bug Rediscovery

Problem Statement
Following Anthropic's April 2026 "Mythos" release, which paired benchmark improvements with specific claims of finding vulnerabilities in OpenBSD, FreeBSD, Linux, FFmpeg, and browsers, a polarized public debate emerged. One side viewed these anecdotes as proof of broad offensive autonomy, while the other dismissed them as marketing due to incomplete disclosure. Neither position offered a scientific null hypothesis. The core problem addressed by this paper is the lack of a controlled, reproducible experiment to determine whether frontier models can independently rediscover these specific, public, Mythos-linked system bugs when the search space is constrained to the vulnerable source files. The authors seek to isolate the difficulty of intra-file vulnerability discovery from the difficulty of file localization or repository search.

Methodology
The authors conducted a controlled "target-file rediscovery" experiment involving three models: GPT-5.5 xhigh, Claude Opus 4.7, and Kimi K2.

• Corpus Construction: The study selected six high-confidence, Mythos-linked system vulnerabilities. The selection criteria required public evidence (patches, advisories, or commits) to construct a hidden "answer key" without leaking the solution to the model. The corpus included:

  1. OpenBSD SACK protocol state bug (tcp_input.c)
  2. FreeBSD RPCSEC_GSS stack overflow (kernel + user)
  3. Linux futex requeue use-after-free (syscalls.c)
  4. FFmpeg H.264 sentinel collision (h264_slice.c)
  5. FFmpeg MPEG-TS descriptor accounting overflow (mpegts.c)
  6. FFmpeg JPEG-XS stale buffer use-after-free (mpegts.c)
     Note: Two tasks shared the same source file (mpegts.c) but targeted different bugs.

• Experimental Setup:

  • Target-File Scaffold: Models were provided only with the specific vulnerable source file(s) for each task. They were not given the repository context, CVEs, patch hashes, or advisory text.
  • Constraints: Models operated with read-only source tools, no internet access, and no live steering. Each model attempted each of the six tasks three times (54 total attempts).
  • Prompting: Prompts were fixed and systems-focused, instructing models to audit for memory safety, integer errors, and invariant violations.
  • Validation Pipeline: A "worker" model generated candidates, a "triage" stage filtered them, and a "validator" model attempted source-grounded confirmation. Final scoring was performed manually by comparing model transcripts against the hidden answer key derived from public patches.

Key Contributions

1. Reproducible Diagnostic Benchmark: The paper establishes a clean, cross-model diagnostic that removes the variable of repository/file search. It isolates the capability to reconstruct specific vulnerability invariants within a known file.
2. Granular Failure Analysis: By distinguishing between "target rediscovery" (finding the specific public bug) and "distinct core bug rediscovery" (finding a different real bug in the same file), the study provides a nuanced view of model performance.
3. Resource Accounting: The study reports detailed metrics on token usage, cost, and runtime, demonstrating that high failure rates persist even when models are given the correct file and significant computational budgets.

Results
The experiment yielded the following outcomes across 54 counted attempts:

• GPT-5.5 xhigh: Achieved 5/18 target rediscoveries.
  • Successfully found the FreeBSD RPCSEC_GSS bug in all 3 attempts.
  • Found the FFmpeg JPEG-XS bug in 2/3 attempts.
  • Found the FFmpeg MPEG-TS descriptor bug once, but only while assigned to the JPEG-XS task (a "wrong-target" finding).
  • Total distinct core bugs found: 3/6.
• Claude Opus 4.7: Achieved 1/18 target rediscoveries.
  • Found the FreeBSD RPCSEC_GSS bug once.
  • Total distinct core bugs found: 1/6.
• Kimi K2: Achieved 0/18 target rediscoveries.
  • Generated the highest number of candidates (21) but none matched the target vulnerabilities.
  • Total distinct core bugs found: 0/6.

Dominant Failure Mode:
The primary failure mode observed was "early commitment to plausible alternate candidates." Models frequently identified source-grounded, security-shaped hypotheses within the assigned files (e.g., incorrect TCP reassembly theories or futex allocation issues) that were validated by the internal validator but did not match the specific invariant fixed by the public Mythos-linked patch.

Significance and Claims
The paper explicitly states that its results do not refute Anthropic's undisclosed workflow, which may involve larger search processes, ranking, multiple workers, and repeated attempts. Instead, the study claims to isolate an "easier subproblem" (file-localized discovery) and demonstrates that even under these favorable conditions, current systems-specific prompting yields limited success (only 6 target matches across 54 attempts).

The authors conclude that the public evidence does not yet isolate model capability from the "elicitation architecture and search budget." The difficulty lies not in finding the file, but in reconstructing the specific vulnerability invariant among many plausible candidates. The paper advocates for future benchmarks that report success alongside cost, candidate volume, validation behavior, and false-positive rates to provide a more auditable picture of agentic vulnerability discovery.