LLM-Based Automated Diagnosis Of Integration Test Failures At Google

1. Problem Statement

Integration testing is critical for complex, distributed software systems but presents significant challenges for developers when failures occur. Unlike unit tests, integration tests involve multiple interacting components, complex environment setups, and massive, heterogeneous log outputs.

• Key Pain Points:
  • Volume and Heterogeneity: A single failing integration test can generate a median of 16 log files and 2,801 log lines (mean of 26 files and 11,058 lines), often split across different servers and log levels.
  • Low Signal-to-Noise Ratio: Logs contain routine operational data, warnings, and recoverable errors that obscure the actual root cause.
  • High Cognitive Load: Developers must manually sift through unstructured text to correlate events across distributed components.
  • Time Cost: Developers report spending significantly more time diagnosing integration failures (often >1 hour, sometimes >1 day) compared to unit test failures.
• Current Limitations: Existing automated diagnosis techniques (statistical debugging, spectrum-based fault localization) are largely designed for unit tests or require specific instrumentation. They struggle with the unstructured, massive scale of integration test logs.

2. Methodology: Auto-Diagnose

The authors introduce Auto-Diagnose, a novel tool that leverages Large Language Models (LLMs) to automate the diagnosis of integration test failures. The system is integrated into Critique, Google's internal code review system.

• System Architecture:

  1. Log Aggregation: Upon a test failure, the system automatically collects logs from the Test Driver and all System Under Test (SUT) components. These logs (INFO level and higher) are joined and sorted by timestamp into a single stream.
  2. Prompt Engineering: The aggregated logs and component metadata are fed into an LLM (specifically Gemini 2.5 Flash) using a carefully crafted prompt template.
     • Constraints: The prompt includes strict negative constraints to prevent speculation (e.g., "Do not draw conclusions without log evidence").
     • Reasoning Steps: The prompt forces the LLM to follow a step-by-step reasoning process: scanning sections, identifying failure points, correlating errors with command-line arguments, and verifying if sufficient information exists before concluding.
     • Output Format: The LLM is instructed to output a structured Markdown report containing a Conclusion, Investigation Steps, and Most Relevant Log Lines with specific formatting (file name, timestamp, callsite, content).
  3. Post-Processing: The LLM response is converted into a "finding" within Critique, where log lines are hyperlinked to their source files for easy navigation.

• Model Configuration:

  • Model: Gemini 2.5 Flash (chosen for speed and cost-effectiveness).
  • Parameters: Temperature set to 0.1 (deterministic) and Top-p to 0.8.
  • Training: The model was not fine-tuned on Google's specific logs; it relies on zero-shot capabilities.

3. Key Contributions

1. First Large-Scale LLM Application for Integration Tests: While previous LLM debugging research focused on unit tests or small code snippets, this paper addresses the complex, distributed nature of integration testing.
2. Seamless Workflow Integration: The tool is embedded directly into the code review process (Critique), providing "in-time" assistance without requiring developers to switch contexts or run separate tools.
3. Robust Prompt Engineering: The paper details a prompt strategy that enforces strict adherence to evidence, preventing hallucinations and ensuring the LLM only concludes based on provided log data.
4. Empirical Validation at Scale: The study provides quantitative and qualitative data from a massive production deployment, moving beyond small-scale academic benchmarks.

4. Results and Evaluation

A. Manual Evaluation (Accuracy)

• Dataset: 71 real-world integration test failures from 39 distinct teams.
• Evaluation: 3 expert developers assessed the LLM's diagnosis.
• Accuracy: Auto-Diagnose correctly identified the root cause in 64 out of 71 cases (90.14%).
• Failure Analysis: The 7 failures were primarily due to infrastructure bugs where log files were not saved correctly during crashes, not LLM reasoning errors.

B. Production Deployment (Scale & Feedback)

• Scale: Deployed across Google's codebase, analyzing 52,635 distinct failing tests across 91,130 code changes.
• Performance: The tool posts findings to Critique in a median of 56 seconds (p90: 346 seconds), significantly faster than manual debugging.
• User Feedback:
  • "Not Helpful" Rate: Only 5.8% of feedback marked the tool as unhelpful (well below the 10% threshold for tool retention).
  • Helpfulness Rank: Ranked #14 out of 370 tools that post findings in Critique (top 3.78%).
  • Helpfulness Ratio: 62.96% of explicit feedback (Helpful vs. Not Helpful) was positive.
  • Note: The discrepancy between the 90.14% manual accuracy and the 62.96% user-reported helpfulness is attributed to negativity bias (users are more motivated to report negative experiences than positive ones).

C. User Interviews

• Positive Reception: Developers appreciated the reduction in cognitive load and the elimination of the need to manually scan logs.
• Expectation Shift: As developers become accustomed to LLM capabilities, some now expect not just diagnoses but automated fixes.
• Sensitivity: Developers are highly sensitive to "unhelpful" findings, especially when they appear automatically in their workflow.

5. Significance and Conclusion

• Validation of LLMs for Complex Logs: The study demonstrates that LLMs are highly effective at processing and summarizing complex, unstructured textual data (logs) from distributed systems, a task where traditional static analysis tools often fail.
• Workflow Transformation: Integrating AI-driven diagnostics directly into the developer workflow (Code Review) is perceived positively and significantly improves productivity.
• Future Directions: The authors plan to:
  1. Filter out "noise" logs (errors that are later recovered) by comparing against passing test runs to improve accuracy.
  2. Extend the tool to provide automated code fixes rather than just diagnoses, aligning with evolving developer expectations.

In summary, Auto-Diagnose proves that LLMs can successfully bridge the gap between massive, unstructured integration test logs and actionable root-cause analysis, significantly reducing the time and cognitive effort required for developers to maintain complex software systems.