Automating Detection and Root-Cause Analysis of Flaky Tests in Quantum Software

This paper presents an automated pipeline leveraging Large Language Models to detect and diagnose flaky tests in quantum software, successfully expanding an existing dataset by 54% and demonstrating that models like Google Gemini can achieve high accuracy (F1-scores up to 0.9643) in classifying flakiness and identifying root causes.

The Gist

Imagine you are baking a cake. In a perfect world, if you follow the recipe exactly, the cake should taste the same every single time. But sometimes, for no obvious reason, one batch comes out perfect, and the next batch (made with the exact same ingredients and steps) turns out soggy or burnt.

In the world of software, this is called a "flaky test." It's a test that says "Pass" one minute and "Fail" the next, even though the code hasn't changed.

Now, imagine doing this baking in a quantum kitchen. Quantum computers are weird; they don't just bake cakes; they bake "probability clouds" of cakes. Because of this, the "flakiness" is even harder to understand. Sometimes the cake fails because of a random gust of wind (randomness), sometimes because the oven is noisy (hardware noise), and sometimes because the baker got distracted (multi-threading).

This paper is about building a super-smart, automated detective to find these flaky quantum tests and figure out why they are failing, so developers don't waste hours trying to bake the same cake over and over again.

Here is the breakdown of their work:

1. The Problem: The "Ghost" Failures

In classical software (like the apps on your phone), flaky tests are annoying. But in Quantum Software, they are a nightmare.

• The Cost: Running a test on a real quantum computer is like renting a private jet. It's incredibly expensive and you have to wait in a long line to use it.
• The Confusion: If a test fails, is the code broken? Or was it just a "glitch" in the quantum universe? Developers often ignore these failures, thinking they are just bad luck, until a real bug slips through and breaks the software in the real world.

2. The Solution: The "AI Detective" Pipeline

The researchers built an automated system to hunt down these ghosts. Think of it as a digital bloodhound.

• Step 1: Expanding the Evidence Board.
  Previously, they only knew about 46 "flaky" cases. They used a technique called Cosine Similarity (imagine it as a "vibe check" for text) to scan thousands of other bug reports. They found 25 new flaky tests that nobody had noticed before, growing their database by 54%.
• Step 2: The Root Cause Analysis.
  They asked: Why are these tests failing? They found that unlike regular software (where the problem is usually two people trying to edit a file at the same time), quantum software fails mostly because of Randomness.
  • Analogy: It's like rolling dice. If your test relies on rolling a 6, and you don't lock the dice in a box (set a "seed"), you might get a 3 next time. The fix is often just "locking the dice" so the result is the same every time.

3. The Star of the Show: Large Language Models (LLMs)

The researchers didn't just write a simple script; they hired AI detectives (Large Language Models like Google Gemini, GPT-4, and Claude) to read the bug reports and the code.

• The Task: They asked the AI: "Is this bug report about a flaky test? If so, what is the cause?"
• The Results: The AI was surprisingly good at it!
  • Google Gemini 2.5 Flash was the champion, getting a score of 94% in detecting flaky tests and 96% in guessing the cause.
  • It's like giving a human expert a stack of 100 messy notes and a code snippet, and they can instantly say, "Ah, this failed because the random number generator wasn't locked down."

4. Why This Matters

Before this paper, finding these issues was like looking for a needle in a haystack while wearing blindfolded gloves.

• Before: Developers manually read thousands of reports, guessed what was wrong, and wasted money re-running expensive quantum tests.
• After: This automated pipeline acts as a filter. It sorts the "real bugs" from the "ghost failures" instantly. It tells developers, "Don't worry, this is just a flaky test caused by randomness. Here is the fix: lock the random seed."

The Bottom Line

This paper is a major step forward in making Quantum Software Engineering practical. By using AI to automate the detection and diagnosis of these confusing, intermittent failures, the researchers are helping developers stop fighting ghosts and start building reliable quantum applications.

In short: They taught AI to spot the "ghosts" in the quantum machine, saving developers time, money, and sanity.

Technical Summary

Here is a detailed technical summary of the paper "Automating Detection and Root-Cause Analysis of Flaky Tests in Quantum Software."

1. Problem Statement

Quantum software systems, like classical software, rely on automated testing. However, they face a unique challenge known as quantum flakiness: tests that pass or fail inconsistently without any changes to the source code.

• Distinct Challenges: Unlike classical flakiness (often caused by concurrency or async waits), quantum flakiness is primarily driven by the probabilistic nature of quantum computation and hardware noise (decoherence, gate errors).
• Impact: Flaky tests mask real defects, reduce developer productivity, and create technical debt.
• Cost: Reproducing quantum flakiness is expensive, especially on real hardware (e.g., IBM Quantum), where re-execution incurs significant financial costs and queue constraints.
• Gap: Existing detection methods rely heavily on manual keyword searches or static code analysis, which lack scalability and fail to leverage natural language signals from issue reports (IRs) and pull requests (PRs). There is a lack of systematic tooling for automated detection and root-cause diagnosis in quantum software.

2. Methodology

The authors propose an automated pipeline that combines embedding-based similarity search for dataset expansion and Large Language Models (LLMs) for classification and root-cause analysis.

A. Dataset Expansion (RQ1 & RQ2)

• Base: Built upon a prior manual dataset of 46 flaky tests from 12 open-source quantum repositories (e.g., Qiskit, NetKet).
• Expansion Technique:
  • Used the mixedbread-ai/mxbai-embed-large-v1 transformer to generate contextual embeddings for GitHub IRs and PRs.
  • Calculated cosine similarity between new issues and known flaky cases.
  • Iteratively identified top-ranked candidates, manually verified them, and added them to the dataset.
• Result: Identified 25 new flaky tests, increasing the dataset size by 54% (totaling 71 cases).
• Root Cause Taxonomy: Categorized causes into 9 types (e.g., Randomness, Floating Point Ops, Multi-threading, Network) and mapped them to fix patterns (e.g., fixing PRNG seeds, adjusting tolerances).

B. Automated Detection Pipeline (RQ3, RQ4, RQ5)

The pipeline uses LLMs to analyze three types of input contexts:

1. Textual Context:
   • Rp (Partial): Initial IR/PR description.
   • Rf (Full): Description + all associated comments.
2. Code Context:
   • Cp (Partial): Method-level code changes.
   • Cf (Full): Complete code listing.
3. Enrichment (Few-Shot): Added high-similarity examples (Ep/Ef) to the prompt to guide the model.

Research Questions Addressed:

• RQ3: Can an LLM classify if an IR/PR is flaky?
• RQ4: Does adding code context improve classification?
• RQ5: Can an LLM identify the specific root cause (e.g., Randomness vs. Network)?

Models Evaluated:

• Open Source: Meta Llama-3.1 (70B, 405B).
• Closed Source: OpenAI (GPT-4o, GPT-4o-mini, GPT-4.1-mini), Google (Gemini-2.5-Flash), Anthropic (Claude-4-Haiku).

3. Key Contributions

1. Expanded Dataset: Created the largest public dataset of quantum flaky tests (71 cases), including buggy code, fixes, and root-cause labels.
2. Automated Discovery Pipeline: Developed a semi-automated method using embedding transformers and cosine similarity to discover new flaky tests, reducing reliance on manual keyword searches.
3. LLM-Based Diagnosis: Demonstrated that LLMs can effectively classify flaky reports and diagnose root causes using a combination of natural language and code context, addressing the scarcity of labeled data in quantum engineering.
4. Empirical Analysis: Provided a detailed taxonomy of quantum flakiness causes, confirming that Randomness (PRNG issues) is the dominant cause (19.2%), distinct from classical concurrency issues.

4. Key Results

• Dataset Growth: Successfully expanded the known flaky test dataset by 54% (from 46 to 71 cases).
• Root Cause Distribution:
  • Randomness: 19.2% (Most common; fixed by setting PRNG seeds).
  • Software Environment: 11.0%.
  • Multi-Threading: 13.7%.
  • Floating Point Ops: 9.6%.
• Model Performance:
  • Best Performer: Google Gemini 2.5 Flash achieved the highest scores.
    • Flakiness Detection (RQ3): F1-score of 0.9420 (with full text and method-level code).
    • Root Cause Identification (RQ5): F1-score of 0.9643.
  • Context Impact:
    • Text: Full context (Rf) generally outperformed partial context (Rp).
    • Code: Method-level code (Cp) often yielded better results than full code listings (Cf), likely because full code introduces noise that distracts the model.
  • Model Comparison: While GPT-4o is powerful, Gemini 2.5 Flash showed superior performance in this specific domain. Larger models (e.g., Llama-405B) did not always outperform smaller ones, suggesting task-specific tuning or prompt engineering is critical.

5. Significance and Future Work

• Significance: This work bridges the gap between classical software maintenance and quantum software engineering. It proves that LLMs can serve as practical, automated tools for triaging flaky reports and diagnosing causes, significantly reducing the cost and time associated with debugging quantum software.
• Future Directions:
  • Dynamic Testing: Incorporating test re-execution on simulators to validate flakiness dynamically.
  • Automated Fixes: Moving from diagnosis to automated program repair (recommending specific code fixes).
  • Dataset Expansion: Broadening the dataset to include more quantum platforms and testing frameworks.

Conclusion

The paper establishes that quantum flakiness is a distinct and costly problem requiring specialized detection methods. By leveraging embedding-based similarity for discovery and LLMs for classification and diagnosis, the authors provide a scalable solution that significantly improves the maintainability of quantum software systems. The results indicate that with the right context (text + method-level code), modern LLMs can achieve near-perfect accuracy in identifying and explaining quantum flaky tests.