Flaky Tests in a Large Industrial Database Management System: An Empirical Study of Fixed Issue Reports for SAP HANA

Technical Summary: Flaky Tests in a Large Industrial Database Management System

Problem Statement
Flaky tests, which yield inconsistent results when executed multiple times against the same source code version, undermine automated code assessment, hinder code merging, and erode developer trust. While various root causes for flakiness exist (e.g., concurrency, timeouts, configuration), existing mitigation strategies are often tailored to specific causes. A comprehensive understanding of prevalent root causes is necessary before evaluating general fix strategies. However, prior research suggests that root causes vary significantly across programming languages, application domains, and project sizes. Consequently, findings from smaller or different-domain studies may not generalize to very large industrial systems like SAP HANA, a database management system with millions of lines of code and unique architectural constraints. Furthermore, manually labeling flaky test reports to identify root causes is time-consuming and tedious at scale.

Methodology
The authors conducted an empirical study on 559 issue reports tagged as "resolved fixed" for flaky tests within the SAP HANA repository, sampled over a 34-month period (2023–2025). The dataset encompasses two primary test types: native C++ unit tests and Python-based system tests that interact with a running database instance via SQL.

To analyze the root causes at scale, the authors employed an LLMs-as-annotators approach:

1. Categorization Schema: They utilized a taxonomy of 15 flakiness categories derived from related work (e.g., Concurrency, Timeout, Oracle-Brittleness, Isolation, Platform).
2. Automated Labeling: Three distinct Large Language Models (gemini-2.5-pro, gpt-5, claude-4-sonnet) were prompted to label each issue report based on its description, root cause, symptom, and resolution fields.
3. Consistency Mechanism: To ensure reliability, each model was queried five times per report. A response was considered valid only if the model assigned the same category in 4 out of 5 runs. A majority vote among the three models determined the final label.
4. Codebook for Ambiguity: A codebook was introduced to resolve overlapping categories (e.g., prioritizing "Platform" over "Timeout" if the error is setup-specific, or "Concurrency" over "Memory Management" if race conditions are the driver).
5. Validation: The approach was validated against a manually annotated ground truth of 232 tests (96% accuracy in test type classification) and a subset of 50 issue reports. Inter-rater agreement was measured using Fleiss' Kappa (model-model: 0.78) and Cohen's Kappa (model-human: 0.63).

Key Results

• Prevalent Root Causes: Concurrency was identified as the most common root cause across the entire dataset, appearing in 130 of 559 reports (23%).
• Test Type Differences:
  • Native Unit Tests (C++): Showed a higher susceptibility to Concurrency (39% of unit test reports) and Platform issues (12%), likely due to low-level functionality testing and hardware-specific features like NUMA.
  • System Tests (Python): While also affected by Concurrency (19%), they faced significantly higher rates of Timeout issues (18%) and Oracle-Brittleness (11%). The latter was often attributed to missing ORDER BY statements or brittle string comparisons.
• Systemic Flakiness: In reports linking multiple tests (127 reports), Concurrency remained the dominant cause (35%). However, the proportion of Platform-related flakiness increased from 7% (single-test reports) to 13% (multi-test reports).
• Temporal Trends: Analysis of quarterly data revealed that Concurrency and Oracle-Brittleness rates remained relatively constant. Conversely, Timeout issues dropped notably following an architectural change in early 2024 that harmonized global timeout values. Async Wait issues showed a sharp spike (2.2x increase) in Q2 2024.

Contributions

1. Evaluation of LLMs for Flakiness Analysis: The study demonstrates that LLMs, when guided by consistency checks and a specific codebook, can achieve substantial agreement with human annotators (Kappa = 0.63), enabling large-scale flakiness analysis in industrial contexts.
2. Comparative Analysis of Test Types: The work provides a detailed comparison of flakiness root causes between native unit tests and system tests within a single large-scale industrial project, highlighting distinct challenges for each type.
3. Longitudinal Overview: The study tracks the evolution of flakiness issue reports over time, identifying how architectural interventions (e.g., timeout harmonization) impact specific failure categories.

Significance and Claims
The authors claim that their findings highlight the context-dependency of flakiness root causes. Specifically, they conclude that Concurrency is the primary challenge in SAP HANA, but the nature of the problem differs by test type. The paper argues that future research on flakiness mitigation should not assume a "one-size-fits-all" solution; instead, approaches must be evaluated for generalizability across different test types (unit vs. system).

Furthermore, the authors suggest that identifying root causes is inherently a multi-label task. The observation that different categories often overlap within a single test (e.g., a timeout occurring only on a specific platform due to a race condition) implies that future tooling should aim to understand interdependencies between contributing factors rather than isolating a single root cause. The study positions LLMs as a viable tool for scaling this complex analysis in large industrial environments.