💻 computer science

Security in a Workflow: Exploring Role-Based Agentic Architectures for Vulnerability Handling

This paper proposes and evaluates a role-based agentic workflow comprising Planner, Analyzer, Fixer, and Verifier agents to bridge the gap between isolated LLM security tasks and real-world industrial practices, demonstrating a 44% vulnerability detection accuracy and 19% fix accuracy on 25 real-world C/C++ vulnerabilities.

Original authors: Srijita Basu, Miroslaw Staron

Published 2026-06-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Srijita Basu, Miroslaw Staron

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to fix a very old, complex house (a software program written in C or C++) that has hidden cracks and weak spots (security vulnerabilities). In the past, you might have hired a single, super-smart detective (a standard AI) to look at the house, find the cracks, and try to patch them all at once. Sometimes this detective gets it right, but often they get overwhelmed, miss subtle clues, or patch the wrong wall.

This paper proposes a different approach: hiring a specialized team of agents instead of one lone detective. They work together in a strict assembly line, where everyone has a specific job.

Here is how the team works, using the paper's findings:

1. The Team Roles (The "Agentic Workflow")

The researchers set up a digital team with four distinct roles, similar to a construction crew:

The Planner (The Site Manager): Before anyone starts digging, this agent scans the blueprints (the code) to spot obvious trouble spots. It doesn't fix anything; it just points the team toward the areas that look suspicious, like "Check the back door" or "Look at the foundation."
- Key Finding: The paper found that having this "Site Manager" was crucial. When they removed this role, the team's ability to find problems dropped by nearly half.
The Analyzer (The Inspector): This is the main detective. They take the clues from the Planner and the raw code to figure out exactly what is broken, why it's broken, and how a burglar could get in.
- Key Finding: The researchers tried giving this agent a high-tech metal detector (a tool called CodeQL) to help find cracks. Surprisingly, the metal detector didn't always help. Sometimes it gave too many false alarms, confusing the Inspector. The best results came from the AI model itself doing the deep thinking, rather than relying heavily on the extra tool.
The Fixer (The Repairman): Once the Inspector says, "The door frame is rotting," the Fixer tries to build a new door. They write the code to patch the hole.
- Key Finding: This was the hardest job. While the team was decent at finding the problem (about 44% accurate), actually fixing it correctly was much harder (only about 19% accurate). Often, the Fixer would patch the hole but accidentally break something else nearby or add unnecessary parts.
The Verifier (The Safety Inspector): After the repair is done, this agent double-checks the work. They ask: "Did you actually fix the rot? Did you make the house safer, or did you just paint over the crack?"
- Key Finding: This role was quite good at spotting mistakes, catching about 69% of the errors in the repairs.

2. The Experiment

The researchers tested this team on 25 real-world security holes found in popular C/C++ software (like the kind used in safety-critical systems). They used three different "brains" (AI models) to power the team members.

They compared two versions of the team:

Team A: Just the four roles talking to each other.
Team B: The same four roles, but the Inspector was given the CodeQL metal detector to help find the cracks.

3. What They Discovered

The "Manager" Matters Most: The most important part of the process was the Planner. Without a manager to guide the team on where to look, the AI got lost. With the manager, the team performed as well as a top-tier commercial AI (GPT-5.5) at finding the bugs.
Tools Aren't Magic: Giving the Inspector a fancy tool (CodeQL) didn't automatically make them better. In fact, it sometimes made things worse because the AI struggled to interpret the tool's data correctly. The paper suggests that for low-level computer languages (like C), the AI needs to be smart enough to prioritize the clues itself.
Finding vs. Fixing: It is much easier for AI to find a security hole than to fix it. The team found the bugs about 44% of the time, but only fixed them correctly 19% of the time.
Human Touch is Still Needed: Because the "Repairman" (Fixer) often made mistakes or added unnecessary changes, the paper concludes that in real-world security, you can't just let the AI run the show. You need a human to look over the AI's shoulder, check the repairs, and make sure the house is actually safe.

The Bottom Line

This paper doesn't claim that AI can now perfectly secure software on its own. Instead, it shows that organizing AI into a structured team with clear roles is a better way to handle security than letting one AI do everything. However, even with a great team, the "fixing" part is still tricky, and human experts are still essential to verify the work.

Technical Summary: Security in a Workflow – Exploring Role-Based Agentic Architectures for Vulnerability Handling

Problem Statement

Secure software engineering in industrial practice involves a structured, multi-stage workflow comprising vulnerability analysis, remediation, and fix verification. However, current Large Language Model (LLM)-based approaches to software security are often fragmented, focusing on isolated tasks such as detection or patch generation rather than reflecting the end-to-end agentic workflows used in real-world security processes. This creates a gap between existing LLM methods and practical secure software engineering, particularly for low-level systems languages like C and C++. In these languages, vulnerabilities depend on complex memory semantics, pointer manipulation, and buffer management, making diagnosis and remediation challenging for LLMs. Furthermore, there is limited exploration of tool-augmented agentic workflows and Model Context Protocol (MCP) skill integration specifically for these low-level contexts.

Methodology

The authors propose and evaluate a role-based agentic workflow designed to decompose vulnerability handling into explicit stages: investigation planning, vulnerability analysis, patch generation, and fix verification.

Workflow Design

The study implements two workflow variants using the CrewAI framework, instantiated with three open-source LLMs: nemotron-cascade-2:30b, qwen3-coder-next, and gpt-oss:120b. The workflows consist of four distinct roles:

Planner: A rule-based static module (not an LLM) that scans input C code for predefined vulnerability patterns (e.g., buffer overflows, null pointer dereferences) using regular expressions. It outputs prioritized investigation regions and themes to guide the Analyzer.
Analyzer: An LLM agent responsible for technical diagnosis. It determines the primary vulnerability, assigns the Common Weakness Enumeration (CWE) ID, identifies vulnerable lines, explains the root cause, and traces the source-to-sink flow.
- Workflow 1: Analyzer relies solely on Planner guidance.
- Workflow 2: Analyzer is augmented with MCP CodeQL integration, providing static analysis capabilities during diagnosis.
Fixer: An LLM agent that generates a minimal secure patch based on the Analyzer's diagnosis. It is constrained to produce the patch and a concise explanation, avoiding re-diagnosis.
Verifier: An independent LLM agent that assesses whether the generated patch resolves the identified vulnerability, removes the root cause, and avoids introducing regressions.

Experimental Setup

Dataset: A curated set of 25 real-world C/C++ Common Vulnerabilities and Exposures (CVEs) spanning five CWE categories: CWE-787 (Out-of-bounds Write), CWE-476 (Null Pointer Dereference), CWE-190 (Integer Overflow), CWE-125 (Out-of-bounds Read), and CWE-191 (Integer Underflow).
Ground Truth: Derived from National Vulnerability Database (NVD) descriptions and developer-provided patched code.
Evaluation: Manual, rubric-based assessment by two authors (Cohen's kappa $\kappa = 0.83$ ) comparing agent outputs against ground truth. Metrics included detection accuracy (CWE assignment, line identification, root cause) and fix accuracy (semantic similarity to trusted fixes).
Scope: The study is an exploratory, methodology-oriented assessment rather than a large-scale benchmark, focusing on 150 workflow executions (25 CVEs $\times$ 3 models $\times$ 2 variants).

Key Contributions

Role-Based Agentic Architecture: Proposal and evaluation of an end-to-end workflow separating planning, analysis, fixing, and verification, demonstrating how explicit role separation can structure security activities.
Static Planner Design: Implementation of a rule-based planner to guide vulnerability investigation in industry-relevant C projects, showing its critical impact on detection quality.
Curated Reference Set: Provision of 25 real-world C/C++ vulnerability instances with ground-truth metadata, developer fixes, and LLM-generated outputs, alongside open-source CrewAI framework code for reproducibility.
Tool Integration Analysis: Investigation into the efficacy of integrating MCP-based static analysis (CodeQL) into the analysis stage of a multi-agent workflow.

Results

The study reports the following performance metrics (using nemotron-cascade-2:30b as the primary model for comparison):

Detection Accuracy:
- Workflow 1 (No CodeQL): Achieved 44% detection accuracy, comparable to GPT-5.5 baselines.
- Workflow 2 (With CodeQL): Accuracy dropped to 31%, suggesting that tool integration alone does not guarantee improvement if the agent cannot correctly interpret or prioritize tool hints.
- Ablation: Removing the Planner reduced detection accuracy to 25% (without CodeQL) and 19% (with CodeQL), underscoring the Planner's critical role.
Fix Accuracy:
- Overall fix accuracy was low at 19% for Workflow 1 and 13% for Workflow 2.
- Repair proved significantly more difficult than detection. While some patches were "comparable" or even "better" than ground truth (particularly for CWE-476 and CWE-787), many contained unnecessary logic or introduced new bugs.
Verifier Accuracy:
- The Verifier achieved 69% accuracy with nemotron-cascade-2:30b, indicating a stronger capability in post-remediation assessment than in generation.
CWE Variability: CWE-787 and CWE-476 were the easiest to detect, while CWE-191 (Integer Underflow) remained the most difficult across all models and settings.

Significance and Claims

The paper positions this work as an initial exploratory study rather than a definitive benchmark. Its primary significance lies in:

Process Improvement: Demonstrating that decomposing security tasks into structured, role-based stages (Planning, Analysis, Fixing, Verifying) can make security activities more repeatable and transparent.
Human-in-the-Loop Necessity: Reinforcing that despite advances in LLMs, the uneven quality of generated patches and the mixed efficacy of tool integration necessitate continued human oversight in security-critical settings.
Workflow over Model: Shifting the focus from selecting the "best" model to refining the underlying workflow architecture and integration techniques. The authors note that while model screening is essential, the workflow design itself is the primary lever for improvement.
Tool Integration Nuance: Highlighting that integrating tools like CodeQL requires careful agent design to ensure hints are prioritized correctly; simply adding tools does not automatically enhance performance.

The authors conclude that role-based agentic architectures are a promising direction for secure software process improvement, motivating future work on larger datasets, repository-level analysis, and refined workflow alternatives (e.g., multiple analyzers, refinement loops).