Technical Summary: Reducing False Positives in Static Bug Detection with LLMs: An Empirical Study in Industry

Problem Statement

Static Analysis Tools (SATs) are critical for ensuring software security and reliability but suffer from high false positive (FP) rates, particularly in large-scale enterprise systems. This issue stems from the necessity of over-approximating program behaviors to guarantee soundness, which generates spurious warnings along infeasible paths. In industrial settings like Tencent, these false alarms create a severe bottleneck: developers must manually inspect every alarm, a process that consumes 10–20 minutes per alarm. With hundreds of alarms generated per scan, the manual effort required to filter out false positives is substantial, leading to inefficiencies in code review workflows. While Large Language Models (LLMs) have shown promise in reducing FPs on open-source benchmarks, their effectiveness in proprietary, large-scale enterprise environments remains unverified.

Methodology

The authors conducted the first comprehensive empirical study of LLM-based false alarm reduction techniques within Tencent, utilizing data from its enterprise-customized SAT, BkCheck, deployed on the Advertising and Marketing Services (AMS) business line.

Data Collection

• Dataset: A real-world dataset of 433 bug alarms was constructed, comprising 328 false positives and 105 true positives.
• Bug Types: The data covers the three most frequent bug types reported by BkCheck: Null Pointer Dereference (NPD), Out-of-Bounds (OOB), and Divide-by-Zero (DBZ).
• Validation: Labels were verified through a rigorous two-round manual review process involving developers and professional code reviewers, adhering to a "security-first" principle to ensure no true positives were discarded.

Experimental Setup

The study evaluated a diverse range of techniques to answer four Research Questions (RQs):

1. RQ1 (Prevalence): Investigated the application of SATs and the manual effort required via developer interviews and data analysis.
2. RQ2 (Effectiveness): Evaluated various LLM-based approaches:
   • Basic LLMs: Direct application of four backbone models (GPT-4o, Claude-Opus-4, Qwen-3-Coder, DeepSeek-R1).
   • Prompting Strategies: Chain-of-Thought (CoT), Few-shot learning, and Bug-type augmentation.
   • Hybrid Techniques: LLM4SA (LLM analyzing code context around sinks) and LLM4PFA (integrating static analysis path constraints with agent-driven reasoning).
   • Baseline Comparison: Compared against traditional deep learning methods (Devign, ReGVD, LLMAO, LineVul).
3. RQ3 (Costs): Measured time (seconds per alarm) and economic costs (API fees).
4. RQ4 (Breakdown & Case Analysis): Analyzed performance across bug categories and conducted a "bad case" analysis to identify limitations.

Metrics: Accuracy, Precision, Recall, F1, False Positive Reduction Precision (FPR_P), and False Positive Reduction Recall (FPR_R).

Key Results

1. Prevalence and Impact of False Positives (RQ1)

• High FP Rate: The dataset showed a 76% false positive rate. When accounting for discarded records with incomplete context, the rate likely exceeds 90%.
• Manual Effort: Each alarm requires 10–20 minutes of manual inspection (often involving two review rounds). This represents a significant waste of engineering resources.

2. Effectiveness of LLMs (RQ2)

• Hybrid Superiority: Hybrid techniques combining LLMs with static analysis outperformed all other methods. Specifically, LLM4PFA achieved the highest accuracy (0.93–0.94) and eliminated 94–98% of false positives while maintaining high recall (0.86–0.88) across different backbone models.
• Prompting Strategies: Among standalone LLMs, Few-shot learning yielded the best results. Surprisingly, Chain-of-Thought (CoT) often underperformed basic prompts, suggesting that advanced models may have internalized reasoning capabilities that CoT prompts inadvertently disrupt.
• Comparison with Traditional ML: Traditional deep learning methods (GNNs and PLMs) performed poorly (accuracy 0.36–0.44) due to a lack of large-scale, labeled enterprise training data and an inability to generalize beyond superficial statistical patterns. One method (LineVul) showed high accuracy (0.76) only by biasing toward classifying all warnings as false positives.
• Interpretability: Unlike traditional models that output binary labels, LLM-based methods provide natural language explanations, aiding human validation.

3. Cost Analysis (RQ3)

• Time: LLM processing times ranged from 2.1 to 109.5 seconds per alarm. Even the slowest model (DeepSeek-R1 at ~1-2 minutes) is significantly faster than the 10–20 minutes required for manual review.
• Economic: API costs ranged from $0.0011 to $0.12 per alarm. Open-source models (e.g., Qwen-3-Coder) can be deployed internally to eliminate API costs entirely.
• Conclusion: The cost-benefit ratio is highly favorable, offering orders-of-magnitude savings compared to manual inspection.

4. Limitations and Breakdown (RQ4)

• Bug Type Variance: LLMs performed best on Divide-by-Zero (DBZ) bugs (simple patterns) and worst on Null Pointer Dereference (NPD) bugs (complex, diverse patterns with intricate constraints).
• Identified Limitations:
  1. Long-Context Reasoning: Performance degrades significantly for warnings involving long code contexts (average 95.6 lines longer than successful cases).
  2. Complex Cascaded Constraints: LLMs struggle with deep nesting, loops, and intricate logical expressions, often missing boundary conditions.
  3. Semantic Understanding: Difficulties arise with uncommon syntax, pointer manipulations, and deeply nested user-defined structures.

Significance and Contributions

This paper makes three primary contributions to the field of software engineering and AI:

1. Industrial Investigation: It provides the first empirical evidence of the severe prevalence of false positives in a major enterprise (Tencent), quantifying the substantial manual effort wasted (10–20 mins/alarm) and validating the urgent need for automated reduction.
2. Extensive Comparison: It offers the most comprehensive evaluation of LLM-based false alarm reduction in an industrial setting to date. The study demonstrates that hybrid techniques (LLM4PFA) are highly effective, eliminating up to 98% of false positives with high recall, significantly outperforming both standalone LLMs and traditional deep learning approaches.
3. Practical Insights and Implications:
   • For Practitioners: The study confirms that LLM-based techniques are cost-effective and can feasibly replace at least one round of manual review in multi-stage workflows, drastically reducing developer burden without compromising safety.
   • For Researchers: It highlights that while LLMs are powerful, they face fundamental limitations in long-context reasoning and complex constraint analysis. The paper advocates for Knowledge-Enhanced LLMs and deeper integration with Static Analysis (combining LLM semantic understanding with SAT soundness) as the most promising path forward.

The authors conclude that while LLMs are not a silver bullet, they represent a transformative step toward making static analysis viable for large-scale industrial codebases by effectively filtering the noise of false positives.