🤖 AI

Post-Training Local LLM Agents for Linux Privilege Escalation with Verifiable Rewards

This paper introduces PrivEsc-LLM, a 4B local model trained via a two-stage pipeline of supervised fine-tuning and reinforcement learning with verifiable rewards, which achieves 95.8% success in Linux privilege escalation tasks—nearly matching top-tier closed models—while reducing inference costs by over 100 times.

Original authors: Philipp Normann, Andreas Happe, Jürgen Cito, Daniel Arp

Published 2026-03-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Philipp Normann, Andreas Happe, Jürgen Cito, Daniel Arp

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but slightly inexperienced, apprentice detective. This detective (the AI) is great at reading books and knowing facts, but they are terrible at actually solving a mystery in a real, messy room. They might know that a lockpick exists, but they don't know how to use it without breaking the door, or they might get stuck trying to pick the same lock for an hour.

This paper is about teaching that apprentice detective how to become a master hacker (specifically, a "privilege escalation" expert) without needing a super-expensive, cloud-based "Grand Master" detective to hold their hand.

Here is the breakdown of what the researchers did, using simple analogies:

The Problem: The "Cloud-Only" Detective

Right now, the best AI hackers are like Grand Masters who live in a giant, expensive skyscraper (the Cloud). They are incredibly powerful, but:

They are expensive: You have to pay a lot of money to ask them a question.
They are slow: You have to send your secret files to their skyscraper, wait for them to look, and send them back.
They are a privacy risk: You can't send your company's secret code to a stranger's building.

The researchers wanted to build a local detective that lives in your own basement (your local computer). This detective is smaller and cheaper, but usually, they aren't smart enough to solve the hard cases.

The Solution: A Two-Stage Training Camp

The researchers took a small, local AI model (called Qwen3-4B, which is like a smart high school student) and gave it a two-part boot camp to turn it into PrivEsc-LLM.

Stage 1: The "Textbook" Phase (Supervised Fine-Tuning)

Imagine you give the apprentice detective a stack of 1,000 solved mystery novels. These aren't just stories; they are step-by-step transcripts of how a pro solved a specific type of lock-picking problem.

What happened: The AI read these "solutions" and learned the patterns. It learned, "Oh, when I see a specific type of door, I should try this specific key."
The Result: The apprentice got much better. It went from solving 42% of the puzzles to 80%. But it was still a bit slow and sometimes got stuck.

Stage 2: The "Video Game" Phase (Reinforcement Learning with Verifiable Rewards)

This is the magic part. Instead of just reading books, the apprentice is now dropped into a video game where the goal is to get to the "Boss Level" (Root Access).

The Rules: The game has a strict timer. Every time the AI makes a move (asks a question or tries a command), it costs "time."
The Reward System:
- Good Move: If it finds a clue, it gets a small point.
- Winning: If it gets to the Boss Level, it gets a huge jackpot.
- Bad Move: If it tries the same wrong key 10 times, or if it wastes time staring at a wall, it loses points.
The Twist: The game is generated randomly. The doors change color, the keys change names, and the locks move. This forces the AI to learn the logic of picking locks, not just memorize the specific keys.

The AI played this game thousands of times. It learned to stop wasting time, to think faster, and to know exactly when to stop guessing and start attacking.

The Results: The Underdog Wins

The researchers tested their new local detective against the "Grand Masters" (the expensive cloud AIs) on 12 different security puzzles.

Speed: The local AI was incredibly fast. It solved 95.8% of the puzzles within 20 moves.
Comparison: The expensive cloud AI (Claude Opus) solved 97.5%. The local AI was almost as good!
Cost: Here is the kicker. The cloud AI costs about $0.62 to solve one puzzle. The local AI costs about $0.005. That is 100 times cheaper.

Why This Matters

Think of it like this:

Before: To solve a complex security problem, you had to hire a $1,000/hour consultant who worked in a different country.
After: You can train a $10/hour local employee who knows the job just as well, works in your office, and never leaves your building.

The "Secret Sauce"

The paper emphasizes that they didn't just cheat by letting the AI memorize the answers. They used a "procedural generator" (like a game engine that builds a new level every time) to make sure the AI learned the skill of hacking, not just the answers to the specific test questions.

In a Nutshell

The researchers proved that you don't need a giant, expensive, cloud-based supercomputer to be a top-tier security expert. With the right training (reading solved cases + playing a reward-based video game), a small, cheap, local AI can do almost as good a job as the most expensive systems in the world, while saving you a fortune and keeping your data safe at home.

1. Problem Statement

The paper addresses the critical gap in cybersecurity research between frontier API-based Large Language Models (LLMs) and local, open-weight models.

The Challenge: While powerful cloud-based LLMs (e.g., Claude Opus) demonstrate strong capabilities in vulnerability discovery and penetration testing, they are closed-source, expensive, and require transmitting sensitive code or system states to external servers. This violates data sovereignty and makes reproducibility difficult.
The Gap: Small, local models (e.g., 4B parameter models) are resource-efficient and private but currently lack the reliability and reasoning capabilities of frontier models for complex, multi-step security tasks.
The Specific Task: The authors focus on Linux Privilege Escalation. This is an ideal testbed because:
- Success is binary and automatically verifiable (gaining root access).
- It requires multi-step interactive reasoning (reconnaissance $\to$ hypothesis $\to$ exploitation).
- It operates under strict "interaction budgets" (limited number of LLM calls/rounds), where efficiency is paramount.

2. Methodology

The authors propose a two-stage post-training pipeline to transform a 4B-parameter open-weight model (Qwen3-4B) into a specialized agent named PrivEsc-LLM.

A. Leakage-Safe Procedural Training

To ensure the model learns generalizable policies rather than memorizing specific benchmark answers:

Procedural Generation: Instead of using static datasets, they use 10 procedural generator families to create diverse training environments. These randomize variables (usernames, paths, passwords) while maintaining structural vulnerability patterns.
Strict Separation: The training data is procedurally generated, while the evaluation is performed on a static, held-out benchmark of 12 scenarios. Benchmark-specific artifacts (like specific GTFOBins binaries or fixed passwords) are explicitly excluded from the training generators to prevent data leakage.

B. Stage 1: Supervised Fine-Tuning (SFT)

Teacher Model: A larger, open-weight model (Arcee Trinity-Large-Preview) is used to generate "expert traces" (successful multi-step interaction transcripts) in the procedural environments.
Process: The 4B model is fine-tuned using Low-Rank Adaptation (LoRA) on these traces.
Goal: To ground the model in the correct "reconnaissance-to-root" loop, teaching it how to enumerate the attack surface, form hypotheses, and execute exploits.

C. Stage 2: Reinforcement Learning with Verifiable Rewards (RLVR)

Motivation: SFT teaches the model what to do, but not necessarily how to do it efficiently under a strict round budget.
Reward Function: The reward is derived entirely from the environment's verifiable outcomes, avoiding subjective human feedback (RLHF). The reward function $R(e)$ $R (e)$ includes:
- Outcome Reward: Binary reward for reaching root.
- Speed Reward: Higher reward for reaching root within fewer rounds.
- Reconnaissance Bonus: Rewards diverse initial inspection before exploitation.
- Penalties: Deductions for repeated loops, malformed tool calls, stalled rounds, or low-information reasoning.
Algorithm: They use AIPO (Asynchronous Importance-weighted Policy Optimization) via the Prime-RL framework. This allows for asynchronous rollouts, improving hardware utilization while correcting for off-policy lag.

3. Key Contributions

Leakage-Safe Generalization Framework: A rigorous setup that separates procedural training from static evaluation, ensuring performance gains reflect learned policies rather than benchmark memorization.
Budgeted Evaluation with Statistical Rigor: The authors evaluate models under fixed interaction caps (rounds) with repeated trials (10 runs per scenario) and Wilson confidence intervals, moving beyond "best-of-few" or single-run metrics.
Efficient Frontier-Level SLMs: Demonstration that a 4B-parameter model, when post-trained with RLVR, can achieve performance nearly matching frontier API models (Claude Opus 4.6) while reducing inference costs by over 100x.

4. Experimental Results

The evaluation was conducted on a held-out benchmark of 12 Linux privilege escalation scenarios.

Performance (Success Rate at R=20 rounds):
- Base Qwen3-4B: 42.5%
- Qwen3-4B + SFT: 80.8%
- PrivEsc-LLM (SFT + RL): 95.8%
- Claude Opus 4.6 (API): 97.5%
- DeepSeek V3.2 (API): 65.8% (at R=20), rising to 94.2% only at R=60.
Efficiency:
- PrivEsc-LLM achieves high success rates at very low round budgets (e.g., 95% at R=10), whereas other models often require significantly more rounds to converge.
- Cost Analysis: The expected cost per successful root for PrivEsc-LLM is approximately $0.005, compared to $0.62 for Claude Opus 4.6. Even including the one-time training cost (~$270), the local model becomes cost-effective after ~440 successful escalations.
Failure Modes:
- The model occasionally fails on complex multi-step exploits (e.g., Sudo GTFOBins) if it cannot recover from tool errors (e.g., malformed tar commands).
- In "Docker group escape" scenarios, the model sometimes fails to act on the "docker group" signal immediately, reverting to password guessing, suggesting a need for better signal prioritization in zero-shot transfer.

5. Significance and Implications

Democratization of Security AI: The paper proves that high-reliability security agents do not require massive, closed API models. Local, open-weight models can be made viable for sensitive environments (e.g., proprietary codebases) through careful post-training.
Verifiable Rewards as a Paradigm: The success of RLVR in a security context suggests that for any task with an automatically verifiable outcome (e.g., crash reproduction, patch validation), local agents can be trained to match frontier performance without human preference data.
Methodological Rigor: The paper sets a new standard for LLM security research by emphasizing procedural training, strict leakage controls, and budgeted, statistically validated evaluation, addressing common pitfalls in the field.

In conclusion, PrivEsc-LLM demonstrates that with the right training pipeline (SFT + RLVR) and rigorous evaluation, small local models can close the reliability gap with frontier APIs, offering a secure, reproducible, and cost-effective solution for autonomous vulnerability discovery.