Evaluating Large Language Models for Multilingual Vulnerability Detection at Dual Granularities

This paper presents a comprehensive empirical study evaluating state-of-the-art pre-trained and large language models for multilingual vulnerability detection across seven programming languages at both function and line levels, revealing that instruction-tuned GPT-4o significantly outperforms other models, particularly in identifying high-severity and unique multilingual vulnerabilities.

The Gist

Imagine you are the head of security for a massive, bustling international city. This city is built by thousands of different architects using seven different languages (like C, Python, Java, Go, etc.) to construct their buildings. Your job is to find the hidden traps (vulnerabilities) in these buildings before criminals can use them to break in.

For years, you've relied on a team of specialized inspectors (called Pre-trained Language Models or PLMs). These inspectors are like seasoned detectives who have studied millions of blueprints. They are good at spotting common traps in specific types of buildings, but they struggle when the city gets too complex or when the blueprints are written in a mix of strange dialects.

Recently, a new type of inspector arrived: the Super-Intelligent AI (called Large Language Models or LLMs). These are like geniuses who have read every book in the library, not just blueprints. They can understand context, jokes, and complex logic. But the big question was: Are these geniuses actually better at finding traps in a chaotic, multilingual city than the specialized detectives?

This paper is the report card of a massive experiment to find out. Here is what they discovered, explained simply:

1. The Two Ways to Look for Traps

The researchers tested the inspectors in two ways:

• The "Whole Building" Check (Function-Level): "Is this entire room dangerous?" This is a quick scan. It's easy to say "Yes, this room is risky," but hard to know exactly which floorboard is loose.
• The "Specific Floorboard" Check (Line-Level): "Exactly which line of code is the trap?" This is much harder. It's like finding a single needle in a haystack of hay.

2. The Contenders

• The Specialists (PLMs): Models like CodeT5P. They are like detectives who have memorized the rules of specific building codes. They are reliable but can get confused by new or mixed languages.
• The Geniuses (LLMs): Models like GPT-4o, Llama 3, and Code Llama. They are smart, but they need to be told how to do the job.
  • Zero-Shot: "Just go find the traps." (The AI guesses based on general knowledge).
  • Few-Shot: "Here are three examples of traps. Now find more." (The AI learns by example).
  • Instruction Tuning: "Here is a manual on how to spot traps. Study it, then go find them." (The AI is specifically trained for this task).

3. The Big Results

🏆 The Winner: GPT-4o with a "Study Guide"

The experiment found that the Specialized Detectives (PLMs) were okay, but the Genius AI (GPT-4o) became unstoppable when you gave it two things:

1. Instruction Tuning: A specific "study guide" (training data) on how to spot vulnerabilities.
2. Few-Shot Prompting: A few examples of what a trap looks like.

The Analogy: Imagine asking a genius to find a lost key in a dark room.

• Without help, they might just guess.
• If you show them a picture of the key (Few-Shot), they do better.
• If you give them a flashlight and a map of the room (Instruction Tuning), they find the key instantly.

GPT-4o was the clear winner. It found more traps than any other model, both in the "Whole Building" check and the "Specific Floorboard" check. It was especially good at finding the most dangerous traps (like SQL Injection or Buffer Overflows) that could crash the whole city.

📉 The "Size Doesn't Matter" Surprise

The researchers wondered: "If we make the AI even bigger (more brain power), will it get better?"

• The Result: Not necessarily. A massive 70-billion-parameter model didn't always beat a smaller 7-billion one.
• The Lesson: It's not about how big the brain is; it's about how well you teach it. A smaller genius with a good study guide beats a giant genius who is confused.

🧠 The "Reasoning" Trap

New "Reasoning" AIs (models that think step-by-step like a human) were tested.

• The Result: They were slightly better at the "Whole Building" check but didn't help much with the "Specific Floorboard" check.
• The Lesson: Sometimes, thinking too hard slows you down. For finding specific code errors, a fast, well-trained instinct is often better than a slow, step-by-step deduction.

4. The Cost of Doing Business

The paper also looked at the price tag.

• The Specialists (PLMs): You can run them on a standard computer in your office. They are cheap to run once you buy the hardware.
• The Geniuses (LLMs): You have to rent them from a giant cloud server (like OpenAI). It costs money every time you ask a question.
• The Verdict: If you are a small shop, the Specialists are cheaper. If you are a massive corporation that needs the absolute best accuracy and can afford the bill, the Geniuses (GPT-4o) are worth the cost.

5. The "Data Leak" Fear

A common worry is: "Did the AI just memorize the test questions?"

• The researchers tested the AI on brand new vulnerabilities that happened after the AI was trained.
• The Result: The AI still performed well! It didn't just memorize; it actually learned the logic of how traps are built.

🌟 The Bottom Line

This paper tells us that Artificial Intelligence is ready to help secure our software, but we can't just throw a raw AI at the problem.

• Don't just ask: "Is this code safe?"
• Do this: "Here is a guide on how to find bugs, and here are three examples. Now, look at this code and tell me exactly where the danger is."

When you give the AI the right tools and training, it becomes a super-powered security guard that can understand seven different languages and find the most dangerous traps better than any human or old-school software ever could. It's a huge step forward for keeping our digital world safe.

Technical Summary

Here is a detailed technical summary of the paper "Evaluating Large Language Models for Multilingual Vulnerability Detection at Dual Granularities."

1. Problem Statement

Automated Vulnerability Detection (AVD) is critical for software security. While Pre-trained Language Models (PLMs) like CodeBERT and CodeT5 have shown promise, existing research suffers from three major limitations:

1. Single-Language Focus: Most studies focus exclusively on C/C++, failing to address the reality of modern polyglot software ecosystems (e.g., Python, Go, Java).
2. Insufficient Generalizability: It is unclear if models pre-trained on multilingual corpora can effectively generalize vulnerability semantics across different language paradigms (e.g., memory corruption in C vs. logic errors in Python).
3. Lack of Dual-Granularity Assessment: Existing evaluations primarily focus on function-level detection (identifying if a function is vulnerable). They largely ignore line-level detection (pinpointing the exact vulnerable line), which is crucial for developer remediation. Furthermore, the performance of Large Language Models (LLMs) in these multilingual and multi-granularity scenarios remains under-explored.

2. Methodology

The authors conducted a comprehensive empirical study to evaluate state-of-the-art PLMs and LLMs across seven programming languages (C, C#, C++, Go, Java, JavaScript, Python).

• Dataset: The study utilized the REEF dataset, containing 4,466 CVEs and 30,987 real-world vulnerability-fixing patches.
  • Function-Level: 20,165 functions (balanced 1:1 vulnerable/non-vulnerable).
  • Line-Level: 8,134 vulnerable functions with specific line labels derived from code diffs.
  • Preprocessing: Code comments were removed to prevent bias/leakage. Tree-sitter was used for parsing, and edit distance algorithms matched pre/post-change functions.
• Models Evaluated:
  • PLMs (6 models): CodeBERT, CodeT5, CodeT5P, UniXCoder, LineVul, and OpenAI Text-Embedding models.
  • LLMs (5 models): DeepSeek-Coder, Code Llama, Llama 3, GPT-3.5-Turbo, and GPT-4o.
  • Baselines: Dummy classifiers (predicting all vulnerable or all clean) and GRACE (a retrieval-augmented baseline).
• Strategies for LLMs:
  • Zero-Shot Prompting: Direct instruction without examples.
  • Few-Shot Prompting: Providing 3 demonstration examples (selected via BM25) in the prompt.
  • Instruction Tuning: Supervised fine-tuning using Low-Rank Adaptation (LoRA) on instruction-filled datasets, followed by zero-shot or few-shot inference.
• Metrics: Accuracy, Precision, Recall, F1-score, FPR, FNR, MCC, and AUC. For line-level detection (highly imbalanced), F1-score was the primary metric.

3. Key Contributions

1. First Comprehensive Multilingual & Dual-Granularity Study: Systematically evaluated PLMs and LLMs across 7 languages at both function and line levels.
2. Strategy Comparison: Compared zero-shot, few-shot, and instruction-tuning strategies, demonstrating that Instruction Tuning + Few-Shot Prompting yields the best results.
3. Fine-Grained Analysis: Analyzed model performance on the Top-25 most dangerous CWEs, vulnerability severity (CVSS), and unique correct/incorrect detections (orthogonality).
4. Cost & Scalability Analysis: Evaluated the impact of model size (7B vs. 70B), reasoning capabilities (Reasoning vs. Non-Reasoning LLMs), and deployment costs (API vs. Local GPU).
5. Open Science: Released a replication package including data, code, and analysis scripts.

4. Key Results

RQ1: Function-Level Detection

• Best Performer: GPT-4o with Instruction Tuning + Few-Shot Prompting achieved the highest accuracy (0.7196), significantly outperforming the best PLM (CodeT5P at 0.6037).
• LLM Limitations: Without instruction tuning, LLMs performed near random chance (Accuracy ~0.50) or worse, even with few-shot prompting (except GPT-4o, which saw a slight improvement).
• PLM Performance: CodeT5P was the strongest PLM but still lagged behind the tuned GPT-4o.

RQ2: Line-Level Detection

• Best Performer: GPT-4o with Instruction Tuning + Few-Shot Prompting achieved the highest F1-score (0.6641), surpassing CodeT5P (0.4841).
• Prompting Impact: Zero-shot prompting failed significantly for line-level detection (F1 < 0.35). Few-shot prompting improved performance, but Instruction Tuning was critical for maximizing precision and recall.
• Language Variance: GPT-4o performed best on JavaScript (F1: 0.7815) and worst on C# (F1: 0.4348).

RQ3: Strengths, Weaknesses, and Analysis

• Orthogonality: GPT-4o (IT+FSP) identified unique correct detections that PLMs missed and had the fewest unique incorrect detections (0 for line-level).
• Dangerous CWEs: GPT-4o detected 75.87% of function-level and 50.79% of line-level instances for the Top-25 most dangerous CWEs, outperforming all other models.
• Severity: GPT-4o excelled at detecting High and Critical severity vulnerabilities.
• Code Complexity: GPT-4o handled complex logical structures (high cyclomatic complexity) and long contexts better than PLMs. PLMs struggled with long code but were slightly better at detecting patterns in deeply nested, short code.
• Model Size & Reasoning:
  • Size: Increasing model size (e.g., 7B to 70B) did not guarantee better performance; in some cases, larger models performed worse without specific tuning.
  • Reasoning: Reasoning LLMs (e.g., DeepSeek-R1) showed marginal improvements over non-reasoning models but did not significantly outperform them, suggesting reasoning capabilities are not yet decisive for this specific task.
• Imbalanced Data: In realistic imbalanced scenarios (few vulnerable, many benign), PLMs (specifically UniXcoder) maintained better balance (F1: 0.46) than LLMs (GPT-4o F1: 0.40), though LLMs improved significantly with few-shot prompting.
• Cost: Local PLM deployment (e.g., CodeT5P on A6000 GPU) is ~15x cheaper per training run than API-based GPT-4o, making PLMs more viable for large-scale continuous scanning, while LLMs offer better scalability for on-demand analysis.

5. Significance

This study provides a critical benchmark for the software security community, demonstrating that LLMs, when properly adapted via instruction tuning and few-shot prompting, are superior to traditional PLMs for multilingual vulnerability detection.

• Practical Impact: It validates the use of GPT-4o for identifying high-severity, cross-language vulnerabilities that traditional tools miss.
• Guidance for Practitioners: It highlights that simply using a large model is insufficient; the strategy (Instruction Tuning + Few-Shot) is the key driver of performance.
• Future Directions: The paper suggests that while LLMs are powerful, cost-effective PLMs remain viable for high-volume, imbalanced scenarios. Future work should focus on unifying code representations and refining learning strategies to bridge the gap between high-resource and low-resource languages.

In conclusion, the paper establishes that GPT-4o with Instruction Tuning and Few-Shot Prompting is the current state-of-the-art for multilingual, dual-granularity vulnerability detection, offering a robust solution for modern, polyglot software security challenges.