🤖 AI

STRIATUM-CTF: A Protocol-Driven Agentic Framework for General-Purpose CTF Solving

This paper introduces STRIATUM-CTF, a modular agentic framework leveraging the Model Context Protocol to enable autonomous, multi-step cybersecurity reasoning, which achieved first place in a live 2025 Capture-the-Flag competition by outperforming 21 human teams.

Original authors: James Hugglestone, Samuel Jacob Chacko, Dawson Stoller, Ryan Schmidt, Xiuwen Liu

Published 2026-03-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: James Hugglestone, Samuel Jacob Chacko, Dawson Stoller, Ryan Schmidt, Xiuwen Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a massive, high-stakes escape room. But instead of a single room, you have to hack into a digital fortress, find hidden keys (called "flags"), and unlock the final door before time runs out.

This is what a Capture The Flag (CTF) competition is for cybersecurity experts. It's a sport where hackers compete to find and exploit weaknesses in computer systems. Usually, this requires a human team working together, reading code, running tools, and guessing the right moves.

The paper you shared introduces STRIATUM-CTF, a new AI system that didn't just play this game; it won first place, beating 21 human teams.

Here is the story of how it works, explained simply.

1. The Problem: The "Smart but Clumsy" Robot

For a while, people tried to use AI (Large Language Models, or LLMs) to hack things. Think of these AIs as incredibly smart librarians who have read every book in the world. They can write code and explain concepts perfectly.

However, when you ask a librarian to physically break into a building, they struggle.

They hallucinate: They might confidently say, "I just used the Super-Unlock-9000 tool!" even though that tool doesn't exist.
They get lost: In a long game, they forget what they did five minutes ago.
They are slow: They need a human to read their instructions, type them into a computer, and then read the computer's messy answer back to them.

2. The Solution: The "Strict Foreman" (The Protocol)

The authors built a system called STRIATUM-CTF. To understand it, imagine a construction site.

The Brain (The AI): This is the architect. It has a big idea: "We need to break down that wall." It's creative and strategic.
The Hands (The Tools): These are the actual tools (like Nmap, Ghidra, or GDB) that do the heavy lifting.
The Foreman (The MCP Layer): This is the magic ingredient.

In the past, the Architect (AI) would shout instructions directly to the Hands. If the Architect yelled, "Use the Super-Unlock-9000," the Hands would try to find it, fail, and crash the whole site.

In STRIATUM-CTF, there is a Foreman standing between them.

The Architect says: "I want to use a tool to check port 80."
The Foreman checks the rulebook. "Okay, 'check port' is a valid tool, and '80' is a valid number. Go ahead."
If the Architect tries to say, "Use the Super-Unlock-9000," the Foreman stops them immediately. "That tool doesn't exist. Try again."

This "Foreman" is called the Model Context Protocol (MCP). It forces the AI to speak in a strict, structured language (JSON) that the computer tools can actually understand. It prevents the AI from making up fake tools or giving bad commands.

3. How They Won the Competition

The team put this system into a live university competition in late 2025. Here's how it played out:

Speed: Humans have to type commands, wait for results, read the text, and type again. The AI does this in milliseconds. It's like a human trying to run a race against a Ferrari.
No Fatigue: Humans get tired after 4 hours. They start guessing or making mistakes because they are stressed. The AI never gets tired. It methodically tries one option, sees it failed, logs the error, and tries the next one without getting frustrated.
Self-Correction: When the AI tried a command that was slightly wrong (like using a tool parameter that didn't exist), the Foreman caught it, told the AI "That's invalid," and the AI immediately fixed its plan. Humans often keep trying the same wrong thing because they are stubborn or tired.

4. The Result

The AI system, named "Graveyard," scored 215 points.

The second-place human team scored 205.
The third-place team scored 200.
Everyone else was way behind.

The AI didn't just win; it showed that it could solve problems as well as a top-tier student who had studied for years, but it did it faster and without getting tired.

The Big Takeaway

This paper proves that if you give a smart AI a strict set of rules (the Protocol) and a safe sandbox to play in, it can become a master hacker. It doesn't need to be a genius on its own; it just needs a good system to keep it focused and prevent it from making up fake facts.

It's the difference between giving a child a box of LEGOs and telling them "Build whatever you want" (they might make a mess), versus giving them a specific instruction manual and a strict supervisor (they build a perfect castle). STRIATUM-CTF is that instruction manual and supervisor for AI hackers.

1. Problem Statement

Offensive cybersecurity operations, such as penetration testing and Capture-the-Flag (CTF) competitions, are traditionally manual, labor-intensive, and prone to human error. While Large Language Models (LLMs) show promise in code generation, they struggle with the multi-step, stateful reasoning required for complex exploitation. Existing autonomous agents face three critical failure modes:

Hallucination: LLMs often generate syntactically invalid commands or non-existent software flags.
Reasoning-Action Gap: Models may correctly identify a vulnerability but fail to generate the precise payload to exploit it due to a lack of grounded environmental feedback.
Context Drift: In long-horizon tasks (like CTFs requiring hundreds of interaction turns), standard agents lose track of early reconnaissance data, leading to error propagation and redundant loops.

Current "copilot" approaches require constant human intervention, creating latency bottlenecks, while "naive" autonomous agents lack the introspection to verify command origins or validate outputs.

2. Methodology: STRIATUM-CTF

The authors propose STRIATUM-CTF, a neuro-symbolic agentic framework that decouples probabilistic neural reasoning from deterministic tool execution using the Model Context Protocol (MCP).

A. Architectural Layers

The system operates as a closed-loop controller with three distinct layers:

Reasoning Layer (Probabilistic): Utilizes Claude Sonnet 4.5 as a strategic planner. It generates high-level strategies and JSON payloads but is physically isolated from the OS to prevent direct execution of hallucinated commands.
Protocol Layer (Symbolic): Acts as a hard constraint layer. It enforces strict schema validation on all function calls (e.g., ensuring port numbers are integers 1–65535). If the LLM generates an invalid request, this layer rejects it before execution, effectively collapsing the model's output manifold onto valid execution paths.
Execution Layer (Deterministic): Consists of containerized security tools wrapped in MCP servers. These include:
- Static/Symbolic Analysis: Ghidra and Angr (for binary analysis and symbolic execution).
- Dynamic Analysis: GDB (for runtime debugging and heap inspection).
- Reconnaissance: Nmap, Nuclei, FFUF, and TLSX.
- All tools return structured JSON observations rather than raw text streams.

B. Formalization

The CTF challenge is modeled as a Partially Observable Markov Decision Process (POMDP). The core innovation is the introduction of a symbolic constraint function $C: A \to \{0, 1\}$ enforced by the Protocol Layer. The agent's policy is modified to project outputs onto a valid schema space $V$ , ensuring that only semantically valid primitives are executed.

C. Control Loop

The agent follows a recursive four-step lifecycle:

Plan: Decomposes high-level objectives into atomic tasks.
Execute: Invokes tools via structured JSON; the Protocol Layer acts as a "circuit breaker" against hallucinations.
Parse: Ingests structured JSON feedback (filtering irrelevant data) to update the agent's state.
Iterate: Refines the plan based on grounded state updates or terminates upon flag capture.

3. Key Contributions

Protocol-Driven Agentic Architecture: A novel framework using MCP to standardize LLM interactions with offensive tools. This reduces hallucinations through strict schema compliance and enables robust error recovery via structured feedback loops.
Integration of Specialized Primitives: The framework integrates complex domain-specific tools (Angr, Ghidra, GDB) directly into the LLM's context window via MCP, enabling deep binary analysis and exploit generation beyond simple scripting.
Live Validation: Unlike prior work relying on static benchmarks, the system was validated in a live, multi-team CTF competition, demonstrating performance against human baselines under time constraints and dynamic uncertainty.

4. Experimental Results

A. Benchmark Evaluation

The system was tested on 15 CTF challenges across four categories (Memory Corruption, Reverse Engineering, Web Exploitation, Cryptography) using an ablation study with four documentation conditions (Baseline, Templates, Lessons, Minimal).

Success Rates: The Baseline (full system) achieved an 86.7% success rate. Remarkably, the Minimal condition (only 55 lines of tool definitions, no documentation) still achieved 77.8%.
Key Insight: The results suggest that performance gains are driven primarily by the Protocol Layer's schematic grounding rather than extensive prompt engineering or documentation. The "Minimal" agent proved that neuro-symbolic grounding is a more token-efficient scaling law than context-heavy prompting.
Efficiency: The Baseline condition had the fastest mean solve time (17.1 mins), while the Minimal condition was slower (20.1 mins), indicating documentation aids efficiency even if it doesn't drastically change success rates for simple tasks.

B. Live CTF Competition

STRIATUM-CTF participated in a university-hosted competition (November 2025) against 22 human teams.

Performance: The system secured 1st Place with 215 points, outperforming the 2nd place human team (205 points) and the 3rd place (200 points).
Comparison: The system scored 3.6x higher than the benchmark for an "A" grade student (60 points).
Advantages: The system outperformed humans due to:
- Machine-Speed Iteration: Eliminating latency in command entry and parsing.
- Systematic Debugging: Methodically varying parameters without cognitive fatigue.
- Resilience: Maintaining systematic hypothesis testing under time pressure where humans tend to guess.

5. Significance and Future Work

Paradigm Shift: The paper demonstrates that standardized context protocols (MCP) are a critical path toward robust autonomous cyber-reasoning, effectively solving the "hallucination" problem in security operations.
Generalization: By competing in a live environment with novel challenges, the system proved its success was driven by generalized reasoning rather than memorized exploit paths.
Future Directions: The authors identify bottlenecks in token consumption (re-injecting full schemas) and long-horizon planning. Future work focuses on:
- Hierarchical Context Management: Using a local LLM to compress tool descriptions.
- Domain-Specific Primitive Layers: Expanding the action space for complex scenarios.
- Triphasic Lifecycle: Implementing Training, Expansion, and Deployment phases to optimize for time-constrained inference.

Conclusion: STRIATUM-CTF establishes a new baseline for agentic security systems, proving that a protocol-driven, neuro-symbolic approach can autonomously solve complex cybersecurity challenges faster and more accurately than human experts.