💻 computer science

Vulnerability Detection with Interprocedural Context in Multiple Languages: Assessing Effectiveness and Cost of Modern LLMs

This study empirically evaluates four modern LLMs on 509 interprocedural vulnerabilities across C, C++, and Python, demonstrating that incorporating caller and callee context significantly enhances detection effectiveness and that Gemini 3 Flash offers the best cost-effectiveness while Claude Haiku 4.5 excels in accurate identification and explanation.

Original authors: Kevin Lira, Baldoino Fonseca, Davy Baía, Márcio Ribeiro, Wesley K. G. Assunção

Published 2026-04-10

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Kevin Lira, Baldoino Fonseca, Davy Baía, Márcio Ribeiro, Wesley K. G. Assunção

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a crime in a massive, bustling city (the software code). Your job is to find the "vulnerabilities"—the hidden traps or broken locks that bad guys could use to break in.

For a long time, detectives (and computer programs) have been looking at crime scenes one room at a time. They would walk into a single room (a single function of code), look around, and say, "This looks safe," or "This looks dangerous."

But here's the problem: The crime often happens between rooms.

Maybe the thief didn't break the lock in the kitchen; they tricked the person in the living room into handing them the keys. If you only look at the kitchen, you miss the whole story. This is called an interprocedural vulnerability—a flaw that only exists when you see how different parts of the code talk to each other.

This paper is a big experiment to see if using AI detectives (Large Language Models or LLMs) works better when we give them the whole story (the caller and the callee) instead of just the single room.

The Cast of Characters (The AI Models)

The researchers tested four different AI detectives, each with a different personality and price tag:

Claude Haiku 4.5: The meticulous, expensive expert who writes very detailed reports.
Gemini 3 Flash: The fast, cost-effective worker who gets the job done well without breaking the bank.
GPT-4.1 Mini: The budget option that sometimes gets confused when given too much information.
GPT-5 Mini: A newer, slightly smarter budget option, but still prone to getting overwhelmed.

The Experiment: Three Ways to Look at the Crime Scene

The researchers took 509 real-world "crimes" (vulnerabilities) from the ReposVul dataset (a library of known software bugs) and asked the AI to find them using three different strategies:

Strategy A (Code Only): "Here is the room where the crime happened. Is it safe?"
Strategy B (Code + Callers): "Here is the room, AND here is the person who walked in and handed the keys to the thief."
Strategy C (Code + Callees): "Here is the room, AND here is the person the thief called for help."

The Big Surprises

1. "More Info" Doesn't Always Mean "Better"

You might think, "If I give the detective more clues, they'll solve it faster!"
The Reality: For some detectives, giving them more clues actually made them worse.

The Analogy: Imagine trying to find a needle in a haystack. If you hand the detective a second, bigger haystack (extra code context), they might get distracted by the extra straw and miss the needle.
The Result: The GPT models (especially the Mini versions) got confused. When the researchers added extra context, the GPT-4.1 Mini's accuracy dropped by a huge 25%. It was like a student who, when given a 10-page textbook instead of a 1-page cheat sheet, forgot the answer entirely.
The Winners: Claude and Gemini were the "smart" detectives. They could look at the extra info, ignore the noise, and still find the bug. Their performance stayed steady no matter how much context they got.

2. The Cost of "Thinking Hard"

Every time you ask an AI to read more code, it costs money (based on how many "tokens" or words it processes).

The Analogy: Reading a single room costs $0.50. Reading the whole house (with all the extra context) costs $1.00.
The Result: Adding context doubled the cost for almost everyone. But did it double the results? No. In fact, for the GPT models, it made the results worse while costing more.
The Best Value: Gemini 3 Flash was the clear winner for the wallet. It found the bugs almost perfectly (97.8% accuracy) for about $0.50 per check. It was the "Goldilocks" model: not too expensive, not too confused.

3. Speaking the Language Matters

The experiment tested three languages: C, C++, and Python.

C/C++: These are like old, complex machinery where one broken gear can crash the whole engine. The "noise" from extra context hurt the GPT models the most here.
Python: This is like a modern, self-contained app. The code is usually cleaner and more isolated. The results were a bit more mixed, but generally, the AI did well regardless of context.

4. The "Why" Matters (Explanations)

Finding the bug is good, but explaining why it's a bug is better.

The Analogy: A doctor telling you "You have a broken leg" is helpful. A doctor telling you "You have a broken leg because you tripped on a loose rug, and here is how to fix the rug" is great.
The Result: Claude Haiku 4.5 was the best teacher. It explained the bugs correctly and clearly 93.6% of the time. The GPT models, however, sometimes got the diagnosis wrong and gave a confusing explanation.

The Bottom Line

If you are building a security tool to scan code for bugs:

Don't just throw everything at the AI. Giving an AI the entire history of a function (all the callers and callees) might confuse it, especially if you use a smaller, cheaper model. Sometimes, looking at the specific function in isolation is actually better.
Pick your detective based on your budget.
- If you want the best balance of price and performance, use Gemini 3 Flash.
- If you need a detailed, high-quality explanation for a human to read, use Claude Haiku 4.5.
- Avoid using the GPT Mini models if you plan to give them a lot of extra context; they tend to get overwhelmed.
Context is a double-edged sword. In the world of AI security, "more data" isn't always "better data." Sometimes, less is more.

In short: The study teaches us that AI is powerful, but it's not magic. You have to know which AI to use, how much information to feed it, and how much you're willing to pay to get the job done right.

1. Problem Statement

While Large Language Models (LLMs) show promise for automated software vulnerability detection, existing research suffers from two critical limitations:

Single-Function Focus: Most studies evaluate LLMs on isolated functions, ignoring interprocedural dependencies. Many vulnerabilities (e.g., use-after-free, buffer overflows triggered by caller arguments) only manifest through data or control flows spanning multiple functions (callers and callees).
Lack of Multi-Language and Cost Analysis: Prior work often targets a single programming language and neglects the economic costs and explainability of LLM outputs, which are crucial for real-world deployment.

The paper addresses the gap in understanding whether providing interprocedural context (caller/callee code) actually improves detection accuracy across different languages (C, C++, Python) and models, while balancing inference costs.

2. Methodology

The authors conducted a comprehensive empirical study involving the following components:

Dataset: They extracted 509 interprocedural vulnerabilities from the ReposVul dataset, covering 236 CWE types across C, C++, and Python.
Models Evaluated: Four modern, cost-effective LLMs were tested:
- Anthropic: Claude Haiku 4.5
- Google: Gemini 3 Flash
- OpenAI: GPT-4.1 Mini and GPT-5 Mini
Experimental Design (Context Variations): For each function, three input configurations were tested:
1. Code Only (CO): The target function code alone.
2. Code + Callees (CC): Target function + functions it invokes.
3. Code + Callers (CK): Target function + functions that invoke it.
Metrics:
- Effectiveness: Accuracy, Precision, Recall, and F1-score.
- Efficiency: Inference cost (USD) based on token consumption and provider pricing.
- Explainability: Manual evaluation of 986 model explanations using a 0–2 rubric for Correctness (identifying type/location) and Comprehensiveness (explaining root cause/exploitation vector).
Statistical Analysis: McNemar's test with Holm–Bonferroni correction was used to determine statistical significance between context configurations.

3. Key Contributions

Empirical Evidence on Context: The study challenges the assumption that "more context is better," demonstrating that interprocedural context does not consistently improve detection and can significantly degrade performance depending on the model.
Cross-Language Analysis: It provides a comparative analysis of LLM performance across C, C++, and Python, highlighting language-specific sensitivities to context.
Cost-Performance Trade-off: The paper quantifies the economic impact of adding context, showing that token consumption (and cost) nearly doubles without proportional accuracy gains.
Explanation Quality Assessment: It introduces a rigorous manual evaluation of LLM-generated security explanations, linking detection accuracy to the quality of the rationale provided.

4. Key Results

A. Effectiveness of Interprocedural Context (RQ1 & RQ2)

Counter-Intuitive Finding: The Code Only (CO) configuration consistently yielded the best or equal-best performance across all models. Adding context (CC or CK) often degraded performance.
Model Sensitivity:
- GPT Models: Highly sensitive to context. In C, adding callee context reduced GPT-4.1 Mini's accuracy from 75.6% to 54.0% and GPT-5 Mini's from 90.5% to 83.5%. The authors hypothesize that GPT models introduce "noise" by attending to irrelevant details in the expanded context.
- Claude & Gemini: Demonstrated robustness. Claude Haiku 4.5 and Gemini 3 Flash maintained stable performance across all configurations, showing no statistically significant degradation when context was added.
Language Differences: The degradation in C was statistically significant for GPT models. In Python, trends were similar but less pronounced (likely due to Python's lower interprocedural coupling). C++ results were inconclusive due to small sample sizes.

B. Cost Efficiency (RQ3)

Token Consumption: Adding interprocedural context nearly doubled the token count (e.g., C prompts went from ~2,367 tokens to ~4,785 tokens).
Best Trade-off: Gemini 3 Flash offered the best cost-performance balance. It achieved an F1-score ≥0.978 for C vulnerabilities at an estimated cost of $0.50–$ 0.58 per configuration.
Cost vs. Gain: The increased cost of adding context did not yield proportional accuracy gains; in many cases, it resulted in lower accuracy for higher cost.

C. Explanation Quality (RQ4)

Top Performer: Claude Haiku 4.5 led in explanation quality, achieving a perfect (2,2) score in 93.6% of evaluations. It provided accurate vulnerability types, locations, and root causes.
GPT Weakness: GPT-4.1 Mini and GPT-5 Mini had significantly higher rates of "zero-score" explanations (6.1%–6.8%), often providing fundamentally incorrect justifications.
Correlation: There is a strong coupling between detection accuracy and explanation quality; models that misclassified more frequently also provided less coherent explanations.

5. Significance and Implications

Tool Design: Security tools should not blindly include caller/callee code for all LLMs. The optimal context strategy is model-dependent. For GPT models, single-function analysis may be superior; for Claude/Gemini, context can be included without penalty.
Economic Viability: Organizations should prioritize models like Gemini 3 Flash for high-volume scanning due to their superior cost-effectiveness, while reserving Claude Haiku 4.5 for scenarios requiring high-quality, actionable explanations.
Research Direction: Future work must move beyond single-function benchmarks. The study highlights that "more context" is not a universal heuristic and that prompt engineering must be tailored to specific model architectures to avoid performance regression.

In conclusion, the paper establishes that while modern LLMs are effective at vulnerability detection, the strategy for providing context is critical. Blindly expanding context can be detrimental, and the choice of model should be driven by a balance of detection accuracy, explanation quality, and inference cost.