💻 computer science

Words Speak Louder Than Code: Investigating Cognitive Heuristics in LLM-Based Code Vulnerability Detection

This paper presents the first systematic investigation demonstrating that LLM-based code vulnerability detection is significantly susceptible to human-like cognitive heuristics (framing, anchoring, and halo effects), which can alter model verdicts and be exploited to suppress up to 97% of detected vulnerabilities.

Original authors: Asif Shahriar, Hongyu Cai, Hadjer Benkraouda, Gang Wang, Z. Berkay Celik

Published 2026-06-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Asif Shahriar, Hongyu Cai, Hadjer Benkraouda, Gang Wang, Z. Berkay Celik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot security guard whose job is to look at computer code and decide: "Is this safe, or is it a ticking time bomb?"

For a long time, we assumed this robot only looked at the code itself—the logic, the math, the structure. But this paper reveals a startling truth: The robot doesn't just read the code; it reads the story around the code. And just like humans, this robot has mental shortcuts (called "cognitive heuristics") that can trick it into making the wrong call.

Here is the breakdown of their findings, using simple analogies.

The Three Ways the Robot Gets Tricked

The researchers tested three specific ways to "hack" the robot's mind without changing a single line of code. They simply changed the context (the metadata) surrounding the code.

1. The Halo Effect (The "Celebrity" Bias)

The Analogy: Imagine a judge in a courtroom. If the defendant is a famous, respected CEO, the judge might subconsciously think, "He's a good person, he wouldn't do anything wrong," and be more lenient. If the defendant is a young, unknown intern, the judge might think, "He's inexperienced, he probably messed up," and be stricter.

The Finding:

When the code was attributed to a "Principal Security Engineer" (a high-status expert), the robot became lazy and trusting. It missed real vulnerabilities because it assumed, "An expert wouldn't write bad code."
When the same code was attributed to a "Junior Developer" (a low-status novice), the robot became suspicious and paranoid. It found the vulnerabilities it missed before, but it also started crying "Wolf!" on safe code, creating false alarms.
The Twist: Some robots (like the one named Claude) did the exact opposite, trusting the junior more than the expert, but the bias was still there.

2. The Framing Effect (The "Warning Label" Bias)

The Analogy: Think of a medicine bottle.

Label A: "This medicine cures 90% of patients." (Positive frame)
Label B: "This medicine fails to cure 10% of patients." (Negative frame)
Even though the math is the same, your reaction changes.

The Finding:

Positive Frame: When the robot was told, "Your job is to make sure the pipeline runs smoothly and avoid false alarms," it became too relaxed. It ignored dangerous bugs to keep things moving fast.
Negative Frame: When the robot was told, "Your job is to find security threats before they destroy the system," it became hyper-vigilant. It found more bugs, but it also started flagging safe code as dangerous.
The Result: This was the strongest trick of all. Every single robot tested was swayed by how the task was described.

3. The Anchoring Effect (The "First Impression" Bias)

The Analogy: If you ask a friend, "Is this house worth $500k?" and they say "Yes," you might think it's a bargain. If you ask, "Is this house worth $1M?" and they say "No," you might think it's cheap. The first number you hear "anchors" your judgment.

The Finding:

If the robot was told, "A previous scan said this code is SAFE," it tended to agree and miss new bugs.
If it was told, "A previous scan said this code is VULNERABLE," it tended to agree and find bugs (even if they weren't there).
The robot was essentially saying, "Well, the other guy said it's safe, so I'll trust that," instead of doing its own independent work.

The Big Problems Discovered

The researchers found three major issues with how these robots work:

1. The "Volume Knob" Problem
The robots didn't get smarter or dumber at finding bugs. They just got louder or quieter.

When the context made them suspicious, they turned the "Volume Knob" up: They flagged everything as dangerous (finding real bugs, but also lots of false alarms).
When the context made them trusting, they turned the "Volume Knob" down: They flagged nothing (missing real bugs, but also having fewer false alarms).
The Takeaway: The robot didn't actually learn to distinguish good code from bad code better; it just changed its mood.

2. The "Hallucination" Trap
When the robot was tricked into thinking a safe code was dangerous (due to a "Junior Developer" label or a "Vulnerable" anchor), it didn't just say "It's dangerous." It often made up a fake reason.

Example: The robot saw safe code but, because it was in a "suspicious" mood, it claimed, "This looks like a memory leak!" when there was no leak. It was confident, but completely wrong.

3. The "Smart" vs. "Dumb" Code

Easy Bugs: Some bugs are obvious, like a typo or a missing semicolon. The robot found these easily, no matter what the context was.
Hard Bugs: Some bugs require deep thinking (like tracking how a variable changes over time). These were the ones the robot got tricked on the most. If the context was "trusting," it missed the hard bugs. If the context was "suspicious," it invented fake hard bugs.

The "Cognitive Attack" (The Real-World Danger)

The researchers didn't just stop at testing; they built a proof-of-concept attack.

Imagine a hacker wants to sneak a dangerous piece of code into a company's software. Instead of trying to hack the robot's brain directly (which is hard), the hacker just fakes the paperwork.

They write the code.
They attach a fake email saying it comes from a famous security expert (Halo).
They write a note saying, "Let's keep the pipeline moving, don't be too picky" (Framing).
They attach a fake report saying, "Previous scans said this is safe" (Anchoring).

The Result: The robot, seeing this "reassuring" package, decided the dangerous code was SAFE. The attack successfully suppressed up to 97% of the vulnerabilities the robot would have normally caught.

The Bottom Line

This paper proves that Large Language Models (LLMs) used for security are not objective machines. They are influenced by the same psychological tricks that influence humans.

They trust the wrong people (experts too much, juniors too little).
They react to how you ask the question (fear vs. comfort).
They get stuck on first impressions (previous reports).

The most dangerous part? You don't need to be a genius hacker to break them. You just need to know how to write a convincing email or a nice-looking commit message. The robot isn't broken; it's just human-like in its flaws.

Technical Summary: Words Speak Louder Than Code

Problem Statement
Large Language Models (LLMs) are increasingly deployed as automated security gatekeepers in real-world workflows, such as CI/CD pipelines and pull request reviews. While prior research has established that LLMs exhibit cognitive heuristics in general reasoning and subjective tasks, no systematic investigation has examined whether these heuristics bias LLMs in the objective task of code vulnerability detection. In practice, vulnerability scanners do not analyze code in isolation; they ingest non-code contextual metadata, including author identity, commit messages, task directives, and prior analysis results. This paper posits that such organic, non-adversarial context can inadvertently trigger cognitive heuristics (halo, framing, and anchoring), leading to inconsistent security verdicts on identical code. The core problem is that these biases can be exploited to suppress vulnerability detection or inflate false positives without altering the code itself.

Methodology
The authors present a controlled framework to isolate the impact of non-code context on LLM verdicts.

Experimental Design: The study holds the code snippet ( $S$ ) fixed while varying only the surrounding context ( $C$ ) to trigger specific cognitive heuristics. The detector function is defined as $f(C, S) \rightarrow \{0, 1\}$ , where the verdict depends on the interaction of code and context.
Heuristics Investigated:
1. Halo Effect: Manipulated via author attribution. Variants include security-relevant prestige (e.g., "Principal Security Engineer from Project Zero" vs. "Junior Developer") and non-security prestige (e.g., "Long-time contributor" vs. "First-time contributor").
2. Framing Effect: Manipulated via task directives and consequence statements. Variants include gain-loss framing (emphasizing the benefits of safe code vs. costs of breaches) and goal framing (verifying compliance vs. identifying violations).
3. Anchoring Effect: Manipulated via prior analysis results. Variants include hard anchors (explicit "SAFE" or "VULNERABLE" verdicts) and soft anchors (implicit outcomes like "passed" or "did not pass" pre-merge checks).
Evaluation Setup: The framework was applied to eight state-of-the-art LLMs (five open-source, three commercial) across three programming languages (C/C++, Java, Python). The datasets used were PRIMEVUL (C/C++ with benign and vulnerable samples) and CLEANVUL (Python and Java, vulnerable only).
Metrics: The study measured Recall ( $R$ ), False Positive Rate (FPR), Precision, and F1-score. Crucially, it introduced the Recall Gap ( $\Delta R$ ) and FPR Gap ( $\Delta FPR$ ) to quantify the magnitude of bias between polarities (e.g., high vs. low halo). A Utility Index (UI) was calculated to determine if a heuristic constructively improved detection (increasing recall more than FPR).

Key Contributions

First Systematic Investigation: This work is the first to systematically explore cognitive heuristics in LLM-driven code vulnerability detection, evaluating three distinct heuristics across eight models and three languages.
Fine-Grained Analysis: The authors provide a code-level analysis revealing that heuristics do not improve a model's analytical capability. Instead, they act as a "volume knob," shifting the model's willingness to flag code as vulnerable without improving its ability to distinguish true vulnerabilities from benign code.
Adversarial Demonstration: The paper demonstrates a proof-of-concept black-box cognitive attack. By forging benign-looking metadata (author identity, PR descriptions, and commit messages), an attacker can suppress up to 97% of previously detected vulnerabilities in a simulated CI/CD workflow.

Results

Universal Susceptibility: All eight evaluated models exhibited susceptibility to cognitive heuristics. The average susceptibility across models was highest for Framing (33.2%), followed by Anchoring (23.5%) and Halo (18.4%).
Directionality:
- Framing: Every model responded in the expected direction (negative framing increased recall, positive framing decreased it).
- Halo & Anchoring: Responses varied. Open-source models generally showed expected responses (low prestige/safe anchor reduced detection), while commercial models like Claude and GPT showed inverse or mixed responses (e.g., GPT became more suspicious under high-prestige author attribution).
No Constructive Utility: Cognitive heuristics did not improve detection capability. The Utility Index was negative or near-zero for most models. When recall increased, FPR increased by a nearly identical magnitude, leaving Precision stagnant (0.48–0.55). Models often flipped verdicts from "Safe" to "Vulnerable" (or vice versa) based on context but failed to identify the actual vulnerability, often hallucinating plausible but incorrect issues.
Reasoning-Dependent Vulnerabilities: Vulnerabilities requiring semantic reasoning (e.g., tracking program state, exception handling) were significantly more susceptible to heuristics (1.5x–2x higher gap) than those identifiable via pattern matching.
Attack Success: The cognitive attack successfully suppressed detection in a realistic threat model. Standard prompt-based defenses (e.g., instructing the model to ignore metadata) failed to mitigate the attack.

Significance
The paper concludes that cognitive susceptibility is a consistent and exploitable property of LLM-based vulnerability detection. The findings challenge the reliability of automated security gatekeepers that rely on LLMs, demonstrating that non-code context can systematically bias security verdicts. The study highlights that current models are not merely "making mistakes" but are susceptible to psychological biases that can be weaponized to bypass security checks. This suggests that the deployment of LLMs in security-critical roles requires rigorous evaluation of their robustness against contextual manipulation, not just code quality.

The Three Ways the Robot Gets Tricked

1. The Halo Effect (The "Celebrity" Bias)

2. The Framing Effect (The "Warning Label" Bias)

3. The Anchoring Effect (The "First Impression" Bias)

The Big Problems Discovered

The "Cognitive Attack" (The Real-World Danger)

The Bottom Line

Technical Summary: Words Speak Louder Than Code

More like this