🤖 AI

An Insight into Security Code Review with LLMs: Capabilities, Obstacles, and Influential Factors

This empirical study demonstrates that Large Language Models, particularly reasoning-optimized ones like DeepSeek-R1, significantly outperform state-of-the-art static analysis tools in security code review, while identifying that prompt engineering strategies, code complexity, and file size are critical factors influencing their detection accuracy and response quality.

Original authors: Jiaxin Yu, Peng Liang, Yujia Fu, Amjed Tahir, Mojtaba Shahin, Chong Wang, Yangxiao Cai

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiaxin Yu, Peng Liang, Yujia Fu, Amjed Tahir, Mojtaba Shahin, Chong Wang, Yangxiao Cai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the editor of a massive, chaotic library where thousands of people are constantly writing new chapters for a giant, shared story. Your job is to find the dangerous plot holes, the plot twists that could crash the whole story, or the pages that might let a thief sneak in and steal the books. This is Security Code Review.

Traditionally, you've had two ways to do this:

The Human Editor: A tired expert reading every word. They are good at understanding context but slow and expensive.
The Automated Spellchecker: A robot that scans for known bad words. It's fast, but it often screams "ERROR!" when there's nothing wrong (false alarms) and misses subtle, clever tricks.

This paper asks: Can a new kind of "Super-Reader" (Large Language Models or LLMs) do a better job than the old spellcheckers?

Here is what the researchers found, explained through simple analogies:

1. The Contenders: Who showed up to the library?

The researchers brought in seven different "Super-Readers" (LLMs). Some are general-purpose (good at writing poems and emails), and one is a "Reasoning-Optimized" model (trained specifically to think step-by-step like a detective). They also brought in the old "Automated Spellcheckers" (static analysis tools) to see who wins.

The Result: The Super-Readers crushed the old spellcheckers. The "Reasoning-Optimized" detective (called DeepSeek-R1) was the clear champion, followed closely by GPT-4 (the one you might know from ChatGPT). The old tools were left in the dust, mostly because they get confused by complex storylines (like race conditions in code) that require understanding how different parts of the story interact over time.

2. The Magic Prompt: How you ask matters

Just like asking a librarian a question, how you ask the Super-Reader matters. The researchers tried five different ways to ask:

The Basic Ask: "Find the bugs."
The Contextual Ask: "Here is the commit message (the note the author left about what they changed). Think step-by-step."
The Cheat Sheet Ask: "Here is a list of known crime types (CWE list). Look for these."

The Winner:

For DeepSeek-R1, the best method was giving it the author's note (commit message) and telling it to "think step-by-step" (Chain-of-Thought). It's like giving a detective the suspect's diary and asking them to walk through the crime scene logically.
For GPT-4, the best method was handing it the "Cheat Sheet" (the list of known crimes). It works best when it has a specific checklist to follow.

3. The Flaws: Even Super-Readers make mistakes

The researchers didn't just look at who found the bugs; they looked at how the Super-Readers reported them. They found two distinct personality flaws:

GPT-4 (The Vague Poet): It often finds the right problem but describes it in a foggy, vague way. It might say, "There is a security risk somewhere in this file," without pointing to the exact line. It also sometimes ignores instructions, like forgetting to say "No bugs found" when there are none.
DeepSeek-R1 (The Overconfident Fact-Checker): This one is very specific. It points to exact line numbers and code snippets. However, it sometimes hallucinates. It might confidently point to Line 42 and say, "This line is dangerous," when Line 42 is actually safe. It's like a detective who is so eager to solve the case they invent evidence that isn't there.

4. The "Needle in a Haystack" Problem

The researchers discovered that these Super-Readers struggle when the story gets too long.

Short Files: They are great at finding bugs in short, concise code files.
Long Files: As the code gets longer (more "tokens"), the Super-Readers get distracted. They miss the small, dangerous details buried in the middle of a massive file. It's like trying to find a single typo in a 500-page novel; your eyes glaze over, and you miss it.

5. The Complexity Paradox

Here is a surprising twist:

Usually, we think complex code is harder to check.
But for DeepSeek-R1, more complex code actually helped it find certain types of bugs (except for memory-related ones).
Why? The researchers suggest that complex code often has more "clues" and structure within the file itself. The Super-Reader can use these internal clues to reason through the problem. Simple, messy code offers fewer clues, making it harder for the AI to figure out what's going on.

The Bottom Line

The paper concludes that AI Super-Readers are powerful new tools that are currently better than traditional automated scanners at finding security holes in code. However, they aren't perfect replacements for human editors yet.

DeepSeek-R1 is the best at thinking through problems but needs to be watched so it doesn't make up facts.
GPT-4 is good at following checklists but can be too vague.
The Strategy: The best approach isn't to replace humans, but to use these AI tools as a first pass. Let the AI scan the short, complex files first, and then have human editors double-check the AI's work, especially when the AI gets too confident or too vague.

The paper does not claim these tools are ready to run the library alone, nor does it suggest they are perfect. It simply shows they are a very promising new assistant that needs careful handling.

Title: An Insight into Security Code Review with LLMs: Capabilities, Obstacles, and Influential Factors

Problem Statement
Security code review is a critical but labor-intensive process essential for preventing data breaches and financial losses. While organizations increasingly integrate security analysis into code review, existing automated tools (static/dynamic/hybrid analysis) suffer from poor generalization, high false positive rates, and coarse detection granularity. Furthermore, data-driven approaches based on traditional machine learning or deep learning often lack robustness due to the scarcity of realistic training datasets, performing poorly on real-world projects. Although Large Language Models (LLMs) have shown promise in various software engineering tasks, their specific capabilities, limitations, and the factors influencing their performance in the fine-grained context of security code review remain largely unexplored.

Methodology
The authors conducted an empirical study to evaluate the potential of LLMs in detecting security defects during code review. The study utilized a manually curated dataset of 534 code files (258 Python, 276 C/C++) derived from four open-source projects (OpenStack Nova/Neutron and Qt Base/Creator), containing 614 security-related review comments.

Models Evaluated: Seven LLMs were tested, including proprietary models (GPT-4, GPT-4 Turbo, Gemini Pro), open-source models (Llama 2 7B, Llama 2 70B), and a reasoning-optimized model (DeepSeek-R1).
Prompt Engineering: Five distinct prompt templates were designed based on best practices and prior strategies:
1. Basic Prompt ( $P_b$ ): Direct instruction to review code.
2. Project Information ( $P_r$ ): Added project name context.
3. General CWE Instruction ( $P_c$ ): Instructed to use CWE as a reference.
4. Specific CWE Instruction ( $P_{cid}$ ): Provided a list of top-level CWEs.
5. Chain-of-Thought ( $P_{cot}$ ): A two-stage prompt requiring the model to generate code context from a commit message before analyzing for defects. A "guardrail" version ( $P_{cot-guardrail}$ ) was also tested to isolate the impact of generated context.
Evaluation Metrics: Responses were manually labeled into four categories: Instrumental (accurate defect type, location, and description), Helpful (partially accurate), Misleading (failed detection or false negatives), and Uncertain. Metrics included I-Score, IH-Score, and M-Score. Consistency was measured via response entropy across three experimental runs.
Analysis: The study compared LLM performance against state-of-the-art static analysis tools (e.g., CodeQL, Bandit, CppCheck). For the top-performing models, a linguistic analysis identified quality problems (e.g., hallucinations, vagueness). Finally, cumulative link regression models were constructed to analyze the influence of 11 factors (e.g., token count, code complexity, defect type) on performance.

Key Contributions

Fine-Grained Evaluation: Provided a comprehensive evaluation of popular LLMs in security code review, moving beyond binary classification to assess defect localization, description, and remediation suggestions.
Non-Determinism Analysis: Measured the impact of LLM non-determinism on the consistency of security code review results.
Quality Problem Quantification: Identified and quantified specific quality issues, including hallucinations (factual and faithfulness), in the responses of top-performing models.
Factor Influence Analysis: Conducted the first regression analysis to statistically validate how specific code and project attributes influence LLM performance in this domain.

Results

Performance vs. Baselines: LLMs significantly outperformed state-of-the-art static analysis tools in security code review. Among LLMs, the reasoning-optimized model (DeepSeek-R1) surpassed general-purpose models.
Top Performers: DeepSeek-R1 achieved the highest overall performance, followed by GPT-4 (via the ChatGPT platform).
- Optimal Prompts: DeepSeek-R1 performed best with a prompt combining the commit message and Chain-of-Thought (CoT) guidance ( $P_{cot-guardrail}$ ). GPT-4 (ChatGPT) performed best with a prompt containing a specific list of CWEs ( $P_{cid}$ ).
Quality Issues:
- GPT-4 (ChatGPT): Frequently produced vague expressions and struggled with instruction following (e.g., missing line numbers), though it had fewer factual errors regarding code details.
- DeepSeek-R1: More prone to generating inaccurate code details (e.g., incorrect line numbers or function names) and exhibited a higher rate of "Incorrect Fact" hallucinations, despite having better instruction-following capabilities.
Influential Factors:
- Token Count: LLMs were more adept at identifying defects in files with fewer tokens; performance generally declined as token count increased.
- Annotations: The presence of security-related annotations in the code improved detection capabilities.
- Complexity: Higher code complexity correlated with enhanced detection capabilities for DeepSeek-R1 for specific defect types (excluding memory-related ones), contrary to the assumption that complexity always hinders detection.
- Consistency: Models with higher detection capabilities (DeepSeek-R1, GPT-4 ChatGPT) exhibited lower consistency (higher entropy) across runs compared to less capable models.

Significance and Claims
The paper claims that while current LLMs are not yet a replacement for human reviewers, they serve as powerful auxiliary tools that significantly outperform traditional static analysis tools in detecting security defects. The study highlights that the effectiveness of LLMs is highly dependent on prompt engineering (specifically the inclusion of commit messages, CoT, or CWE lists) and the specific characteristics of the code being reviewed (token count, complexity, annotations).

The authors emphasize that the non-determinism of LLMs is a double-edged sword: models capable of deeper reasoning (and thus better detection) tend to produce more inconsistent outputs. They conclude that a multi-layered review strategy is necessary, where LLMs handle initial reviews of low-difficulty code (fewer tokens) while human experts focus on high-difficulty code and validate LLM findings. The study also underscores the need for future work to address hallucinations regarding code details and to improve the stability of LLM outputs without sacrificing detection sensitivity.