💻 computer science

unix-ctf: Procedural Environments for Unix-Competence Reinforcement Learning

This paper introduces **unix-ctf**, a procedural environment that generates 656 distinct shell-based capture-the-flag tasks to isolate and train Unix competence, demonstrating that fine-tuning a language model on this surface significantly improves its ability to use operating system primitives independently of general programming skills.

Original authors: Geoffrey Bradway, Roger Creus Castanyer, Lorenz Wolf, Maxwill Lin, Matthew James Sargent, Augustine N. Mavor-Parker

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Geoffrey Bradway, Roger Creus Castanyer, Lorenz Wolf, Maxwill Lin, Matthew James Sargent, Augustine N. Mavor-Parker

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Two Different Skills

Imagine you are teaching a robot to work in a busy office. There are two very different skills involved:

The "Programmer" Skill: The robot needs to write a complex report, solve a math problem, or build a small machine. It can do this using a standard language (like Python) and just use the office terminal as a simple typewriter to type the code.
The "Janitor/Manager" Skill (Unix Competence): The robot needs to know where the hidden keys are, how to open a specific type of locked drawer, or how to read a secret message written in invisible ink on a filing cabinet. This requires knowing the specific rules of the building itself, not just how to write a report.

The Problem: Current tests for AI robots mix these two skills up. A robot that is a genius programmer but knows nothing about the building's secret rules can still pass the test. The authors argue we need a way to test only the "Janitor/Manager" skill—the ability to use the operating system's native tools to find things that aren't obvious.

The Solution: A "Scavenger Hunt" Generator

The authors built a system called unix-ctf. Think of this as an automated factory that creates Scavenger Hunts for AI robots.

The Goal: Hide a secret token (a "flag" like flag{abc123}) inside a digital room (a Linux container).
The Twist: The flag isn't just in a text file. It's hidden using a specific, tricky feature of the computer's operating system.
- Example: Hiding the flag inside the "extended attributes" of a file (like a secret note stuck to the back of a photo that you can't see unless you know exactly how to look).
- Example: Hiding the flag inside the metadata of a music file or a specific part of a program's code.
The Robot's Job: The robot must explore the room, figure out which trick was used, and use the correct command to find the flag.

How They Built It (The Factory)

Creating these puzzles manually is hard. The authors used a smart pipeline to build them automatically:

The Architect (LLM): They asked a powerful AI to come up with ideas for hiding the flag (e.g., "Hide it in a file created before the year 1970").
The Inspector (Mechanical Checks): Before saving the puzzle, a robot inspector checks two things:
- The "No Cheating" Rule: Is the flag hidden so well that a simple search won't find it?
- The "Recovery" Rule: If you start with a fresh room, can the recovery script actually find the flag?
The Result: They started with 750 ideas. Because their "Inspector" was so strict, 656 of them passed. This is a 87.5% success rate.
- Comparison: When they tried to copy a similar project by other researchers, only 17.5% of the puzzles worked. The authors' method is much better at making reliable puzzles.

They ended up with 155 unique types of tricks (like hiding in file names, hidden system logs, or special code sections).

Training the Robot

The authors took a standard AI model (Qwen3-8B) and trained it using these scavenger hunts.

The Training: They didn't just show the robot the answer. They let the robot try, fail, and learn from the feedback (Reinforcement Learning).
The Result:
- Before training, the robot solved about 11.6% of the new puzzles.
- After training, it solved 43.6%.
- This proves that the robot actually learned the specific "Unix" skills, not just memorized answers.

Does It Help With Other Tests?

The authors tested if learning these scavenger hunts helped the robot on other existing tests (like InterCode-CTF, which is a standard test for security skills).

The Surprise: The robot didn't get better at everything. It didn't get better at writing code.
The Win: It got much better at the specific tasks that required "Unix competence" (like Forensics).
- In the "Forensics" category (finding hidden clues), the success rate jumped by 33 percentage points.
- This confirms that the training taught the robot how to be a better digital detective, specifically for finding hidden things in a computer system.

Summary

This paper introduces a new way to train AI to be better at using computer operating systems directly. Instead of just teaching the AI to write code through a terminal, they built a specialized "gym" of hidden-object puzzles. They proved that:

You can automatically generate high-quality, tricky puzzles.
AI can learn these specific "operating system detective" skills.
This training makes the AI significantly better at finding hidden data, a skill that previous training methods often missed.

What they did not claim: They did not claim this makes the AI a better general programmer, nor did they claim it solves real-world hacking attacks or medical problems. They strictly focused on measuring and improving the AI's ability to navigate and find things within a Unix computer system.

Technical Summary: unix-ctf

Problem Statement
Current terminal benchmarks for evaluating language model (LM) agents conflate two distinct capabilities: general programming performed via a terminal and genuine Unix competence. The authors argue that existing benchmarks (e.g., Terminal-Bench 2.0, InterCode, Cybench) are heavily biased toward "coding-in-a-terminal," where a solver fluent in Python but weak in OS internals can succeed by writing scripts that wrap standard tools. Conversely, "Unix competence" is defined as the ability to utilize OS, shell, or file-format primitives that have no clean analogue in general-purpose programming (e.g., accessing POSIX extended attributes, parsing ELF note sections, or interpreting pre-epoch file modification times). The paper posits that this distinction is currently obscured, leading to training pipelines that improve programming proficiency without necessarily enhancing Unix fluency.

Methodology
To isolate and train for Unix competence, the authors developed unix-ctf, a procedural environment generator for capture-the-flag (CTF) tasks. The system operates through a multi-stage pipeline:

Harvest Pipeline: The system generates tasks by alternating between frontier-model exploration (using Claude Opus to hide flags) and mechanical verification.
- Bidirectional Contract: To ensure task validity, the pipeline enforces a strict contract: the "plant" script must leave no plaintext trace of the flag on disk, and the "recovery" script must successfully retrieve the flag in a fresh directory.
- Pipeline Factoring: Unlike prior approaches (e.g., Endless Terminals) that ask LLMs to generate entire Docker containers, unix-ctf separates the infrastructure. A pre-debugged ctf-base image handles the system layer, while the LLM is restricted to generating parameterized plant.sh and recovery.sh scripts. This architectural choice significantly increases the yield of portable, reusable variants.
- Canonicalization: Generated scripts undergo deduplication (using difflib.SequenceMatcher at a 0.85 similarity threshold) to produce a canonical library of techniques.
Environment Generation:
- The generator instantiates fresh Docker containers based on the canonical library.
- Multi-Flag Strategy: To address sparse reward signals in reinforcement learning (RL), each container is seeded with eight distinct flags using different techniques, rather than a single binary outcome.
- Sticky Pool Scheduling: A pool of 50 techniques is maintained during training. After each batch, the top 15 techniques (by solve rate) are rotated out to ensure the agent encounters a broad distribution of skills, preventing overfitting to specific families.
Training Protocol:
- The authors fine-tuned Qwen3-8B using Group Relative Policy Optimization (GRPO) with LoRA adapters.
- Two initialization strategies were tested: direct GRPO from the base model and GRPO from a model that underwent a brief supervised fine-tuning (SFT) pass on successful solver trajectories to align tool-call formats.
- Training involved 40 batches, rotating the technique pool to expose the model to diverse Unix features (file metadata, binary formats, IPC, crypto, etc.).

Key Contributions

High-Yield Procedural Generator: The pipeline achieves an 87.5% portability rate (656 valid variants from 750 raw attempts), a five-fold improvement over a reproduction of the Endless Terminals approach (17.5%). This is attributed to the separation of infrastructure from logic and the bidirectional contract.
Canonical Technique Library: The process yields a library of 155 distinct technique identifiers spanning 16 families (e.g., filesystem metadata, ELF internals, X.509 extensions, process control). These techniques specifically target OS features with no direct Python analogue.
Demonstration of Separability and Trainability: The paper provides empirical evidence that Unix competence is a separable skill that can be trained independently of general programming. Fine-tuning on the unix-ctf surface significantly improved performance on holdout tasks and redistributed the model's capabilities on external benchmarks.

Results

Holdout Performance: On a 15-skill multi-family holdout set ( $n=225$ ), the solve rate increased from 11.6% (Base Qwen3-8B) to 43.6% (GRPO with SFT initialization), representing a 3.8× improvement.
Transfer to InterCode-CTF: Training on unix-ctf redistributed the model's performance on the external InterCode-CTF benchmark (100 tasks). While the aggregate score remained stable, the Forensics category saw a +33 percentage point gain (from 7% to 40% with base-init GRPO), and Reverse Engineering also improved. This suggests the training surface specifically enhances skills relevant to forensic analysis and OS internals.
Efficiency: The SFT-initialized variant utilized the 18-turn budget more efficiently (12.7 mean turns vs. 17.2 for base), indicating better task recognition.
Reward vs. Transfer Divergence: The authors observed a divergence where the SFT-initialized variant achieved higher training rewards but lower transfer to InterCode-CTF compared to the base-initialized variant, cautioning against using training reward as a sole proxy for downstream capability.

Significance and Claims
The paper claims that Unix competence is separable, trainable, and best evaluated directly rather than being folded into general programming-through-a-shell tasks. By isolating the "Unix" component of terminal interaction, the authors demonstrate that procedural generation can effectively target underrepresented OS features (like filesystem metadata and binary formats) that current benchmarks miss. The work suggests that training pipelines should explicitly target these specific OS primitives to improve agent performance in security and system administration contexts, rather than relying on general terminal benchmarks that may primarily measure programming fluency.

The authors remain modest regarding the scope, noting that the library relies on LLMs for generation (potentially missing underrepresented techniques) and is currently limited to CTF-style tasks, which do not cover the full breadth of Unix administration. They release methodological details for reproduction but withhold the full artifact library to prevent lowering barriers for offensive security training.