🤖 AI

LLMs for Qualitative Data Analysis Fail on Security-specificComments in Human Experiments

This paper evaluates the ability of large language models to automate the thematic analysis of security-specific comments in human experiments, finding that while detailed codebooks improve performance, current models still lack the reliability to replace human annotators for this complex task.

Original authors: Maria Camporese, Fabio Massacci, Yuanjun Gong

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Maria Camporese, Fabio Massacci, Yuanjun Gong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Can AI Do the "Grunt Work" of Security Research?

Imagine you are a detective trying to solve a mystery. You have a stack of 263 handwritten notes from witnesses (human students) describing why they thought a piece of code was safe or dangerous.

To make sense of this, you need to read every single note and tag them with specific labels, like:

"Did they mention a specific variable name?"
"Did they mention a specific line of code?"
"Did they say they were confused?"

This process is called Thematic Analysis. It's incredibly useful, but it's also boring, expensive, and slow because you need human experts to do it.

The Question: Can we hire a super-smart AI (a Large Language Model, or LLM) to do this tagging for us? Can the AI replace the human expert?

The Answer: No. Not yet. The paper concludes that while AI is great at many things, it is currently terrible at understanding the nuance of security comments.

The Experiment: The "AI Intern" vs. The "Human Expert"

The researchers set up a test to see if an AI could act as a "Human Intern" to finish the tagging job.

1. The Setup (The Training Ground)

They didn't just ask the AI to guess. They tried to teach it exactly how a human learns, step-by-step:

Step 1 (The Basics): They gave the AI a list of rules (e.g., "Tag 'Var' if a variable name is mentioned").
Step 2 (The Examples): They gave the AI a "Cheat Sheet" with examples of good and bad tags.
Step 3 (The Tricky Cases): They showed the AI examples where humans disagreed, along with a teacher explaining why one human was right and the other was wrong.

It was like giving the AI the best possible training manual, the best examples, and a teacher to explain the hard parts.

2. The Models (The Contestants)

They tested four of the smartest AI models available in 2026 (including big names like GPT-5, Claude-4, and some open-source giants like DeepSeek and Qwen).

3. The Results (The Scorecard)

The researchers compared the AI's tags against the "Gold Standard" (what the human experts agreed on). They used a special score called Cohen's Kappa, which is like a "fairness meter." It asks: "Is the AI actually smart, or is it just getting lucky because most answers are 'No'?"

The Verdict:

The AI got confused. Even with the best training, the AI failed to match human experts.
It missed the forest for the trees. The AI could spot simple things (like "Did they mention the word 'buffer'?") but failed at complex reasoning (like "Did they express uncertainty about a vulnerability?").
More training didn't help. Giving the AI more examples or longer explanations didn't fix the problem. It was like trying to teach a dog to do calculus by giving it a bigger textbook; the dog just got more confused.

Creative Analogies to Explain Why It Failed

1. The "Literal Translator" vs. The "Context Reader"

Imagine a human reading a note that says: "I think the debug variable might be the issue, but I'm not sure."

The Human understands that the student is pointing to a specific code variable named debug but is unsure if it's the cause of the bug.
The AI sees the word "debug" and thinks, "Aha! They mentioned a keyword!" and tags it as a positive match for "Security Keyword Mentioned." It misses the fact that the student was actually expressing doubt. The AI is reading the dictionary definition, not the human intent.

2. The "Weather Forecaster" Problem

The paper mentions that in weather forecasting, if it rains 90% of the time, a forecaster who just says "It will rain" every day is 90% accurate. But that's useless!

In security comments, most people say "I don't see a bug."
If the AI just guesses "No bug" for everything, it gets a high accuracy score.
But when the researchers corrected for this "luck" (using the Chance-Corrected Accuracy), the AI's score dropped to near zero. It was essentially guessing, not thinking.

3. The "Over-Confident Student"

Some AI models acted like a student who is confident but wrong.

Human: "I'm not sure if this line is vulnerable." (Tag: Uncertainty)
AI: "They used the word 'vulnerable'! Tag: Security Keyword Mentioned."
The AI couldn't distinguish between talking about a vulnerability and finding one. It got tripped up by the literal words rather than the meaning.

The Takeaway: AI as a "Co-Pilot," Not the "Pilot"

The paper's main conclusion is a bit of a bummer for those hoping to automate everything: AI cannot replace human experts for this specific job yet.

What AI is good at: Spotting simple, obvious patterns (like counting how many times the word "buffer" appears).
What AI is bad at: Understanding the feeling of the comment, the nuance of uncertainty, and the deep context of security risks.

The Final Advice:
Think of the AI not as a replacement for the human detective, but as a magnifying glass.

Let the AI scan the notes and highlight the obvious stuff (e.g., "Here are all the notes mentioning line numbers").
Let the human expert read the highlighted notes and do the actual thinking and decision-making.

If you try to let the AI drive the car (do the whole analysis) on a bumpy security road, it will crash. But if you let it steer while you keep your hands on the wheel, it might help you get there faster.

1. Problem Statement

In empirical software and security engineering, Thematic Analysis (TA) is the standard method for extracting qualitative insights from free-text justifications provided by human subjects (e.g., explaining why they identified a vulnerability in code). This process is labor-intensive, requiring multiple domain experts to manually code text into specific categories (codes) to ensure reliability.

While Large Language Models (LLMs) have shown success in general sentiment analysis and opinion mining, their ability to replace human annotators in security-specific technical contexts remains unproven. The core problem is whether LLMs possess the necessary contextual understanding to detect nuanced, domain-specific codes (e.g., distinguishing between a general mention of "debug" and a specific code identifier, or identifying a potential exploit vs. a general security concern) with the same reliability as human experts.

2. Methodology

The authors designed an experimental framework to test if LLMs can act as automated annotators for security-relevant comments.

Dataset:
- Source: An unpublished dataset from a controlled experiment involving 263 free-text comments from Computer Science Master's students analyzing vulnerable code snippets.
- Ground Truth: The comments were manually annotated by four human experts following a rigorous TA protocol (emergence of codes, codebook creation, independent annotation, and conflict resolution).
- Codes: 9 specific security-relevant codes were tested (e.g., Var: Variable/Method identifiers mentioned; Lin: Lines of code mentioned; Key: Security keywords; Exp: Potential exploit; NoVul: No vulnerability found).
Models Evaluated:
Four state-of-the-art LLMs were selected based on LiveBench reasoning scores:
- Proprietary: GPT-5 (ChatGPT-5), Claude-4 (Claude Sonnet 4).
- Open Source: DeepSeek-V3.2, Qwen3-Max.
Prompt Engineering Strategy (Simulating Human Workflow):
The study mimicked the iterative human coding process using four progressively complex prompt types:
- P1: Task description and code names only (Emerging codes).
- P2: Task description + Formal codebook definitions with examples.
- P3: P2 + Few-shot examples from other human annotators.
- P3+ (Final): P3 + Conflicting examples (cases where humans disagreed) + Researcher clarifications on those conflicts.
Evaluation Metrics:
- Cohen's Kappa ( $\kappa$ ): The primary metric used to measure agreement between the LLM and human annotators, corrected for chance. This is crucial because security codes are often sparse (rare events), making raw accuracy misleading.
- Heidke Skill Score (HSS): Equivalent to chance-corrected accuracy.
- Statistical Testing: Wilcoxon paired tests were used to determine if improvements from prompt refinements were statistically significant.

3. Key Contributions

Empirical Evidence of Failure: The study provides rigorous evidence that current SOTA LLMs cannot reliably replace human annotators for security-specific qualitative data analysis, even when provided with detailed codebooks and conflict-resolution examples.
Metric Selection: The paper emphasizes the necessity of using chance-corrected metrics (Cohen's Kappa) over raw accuracy in security contexts. Raw accuracy often exceeded 75% (due to the prevalence of negative cases), but chance-corrected accuracy dropped significantly, revealing the models' inability to detect rare, positive security events.
Prompt Engineering Limits: The research demonstrates that iterative prompt refinement (adding definitions, examples, and clarifications) yields diminishing returns. While moving from P1 to P2 showed a slight improvement, further complexity (P3, P3+) did not yield statistically significant gains across models.
Model Performance Disparity: Surprisingly, open-source models (DeepSeek-V3.2, Qwen3-Max) generally outperformed proprietary models (GPT-5, Claude-4) in this specific security annotation task.

4. Results

Overall Performance:
- GPT-5 & Claude-4: Achieved "Limited" replaceability ( $\kappa \approx 0.26 - 0.30$ ). They struggled significantly with nuanced codes.
- DeepSeek-V3.2: Achieved "Moderate" replaceability ( $\kappa \approx 0.53$ ).
- Qwen3-Max: Achieved the highest performance with "Substantial" replaceability ( $\kappa \approx 0.61$ ), but still fell short of the "Almost Perfect" threshold required for full automation.
Code-Specific Variance:
- Models performed reasonably well on explicit, surface-level codes (e.g., Var - identifying identifiers, Lin - line numbers).
- Models failed on interpretive codes (e.g., Unclr - unclear answers, Exp - potential exploits). For instance, models often confused general technical terms with specific code identifiers or misinterpreted uncertainty.
Prompt Impact:
- The transition from P1 (names only) to P2 (definitions) was the only statistically significant improvement.
- Adding conflicting examples (P3+) did not significantly improve performance and, in some cases, confused the models further.

5. Significance and Conclusion

Current Limitation: LLMs are not yet ready to autonomously replace human experts in security qualitative analysis. Their performance is inconsistent and prone to hallucinations or misinterpretation of security context.
Recommended Role: The authors argue that LLMs should currently serve an assistive role (e.g., generating draft annotations, highlighting potential codes for human verification) rather than an autonomous one.
Future Work: The paper suggests that future research should focus on hybrid pipelines, model fine-tuning on security-specific corpora, and structured prompting interfaces rather than relying solely on prompt engineering to bridge the semantic gap.
Broader Impact: This study challenges the assumption that LLMs are a "low-hanging fruit" for all NLP tasks in software engineering, highlighting that domain-specific, high-stakes reasoning requires human oversight.

Artifact Availability: The dataset and code implementation are open-sourced on Zenodo (Record ID: 18742065).