Technical Summary: Holmes – Multimodal Agentic Diagnosis for Mixed-Language Mobile Crashes at Industrial Scale

1. Problem Statement

Diagnosing mobile crashes in ultra-large-scale industrial applications (e.g., WeChat with 70 million lines of code and millions of daily crash reports) presents three critical challenges:

• Scalability of Static Analysis: Traditional static analysis and global inter-procedural analysis are computationally prohibitive for repositories of this magnitude (requiring >12 hours and >500GB memory for call graph generation).
• Limitations of Existing LLM Agents: State-of-the-art software engineering agents typically rely on reproducible environments and failure-inducing tests. These are unavailable in post-mortem mobile crash scenarios due to privacy constraints and the transient nature of device states.
• Semantic Gaps in Mixed-Language Environments: Mobile applications often combine open-source business logic with closed-source system frameworks (C/C++/ObjC/Swift). Existing methods struggle to trace execution flows across these boundaries, often hitting dead ends when source code is unavailable for system binaries.
• Single-Modal Analysis: Current approaches typically analyze logs, stack traces, or code in isolation, failing to synthesize the spatial, temporal, and concurrency clues necessary to diagnose complex non-local defects.

2. Methodology: The Holmes Framework

Holmes is a multi-agent system designed to automate root cause analysis (RCA) by synthesizing multimodal runtime signals without requiring a reproducible environment. It employs a hierarchical Retrieve-Explore-Reason architecture.

Layer 1: Parallel Context Retrieval

The Dispatcher parses raw crash reports to extract three primary artifacts: the crashing thread's stack trace, the full log stream, and relevant source/assembly code. This layer utilizes three specialized agents to collect evidence in parallel:

• Stack Code Retriever: Retrieves source code for internal business logic and assembly code for system frameworks based on the crashing thread's stack trace, ensuring visibility across system boundaries.
• Log Miner: Processes a 1-minute temporal window of logs using a semantic Map-Reduce strategy. Instead of naive truncation, it distills log lines that share causal dependencies with the crash summary, mitigating the "lost-in-the-middle" phenomenon.
• Thread Inspector: Analyzes the full thread dump to detect cross-thread interference (e.g., race conditions, deadlocks). It uses a relevance ranking strategy to identify the top-k threads most relevant to the crash, filtering out noise.

Layer 2: Agentic Code Exploration

To navigate the 70-million-line codebase without a pre-built global call graph (which is too expensive to maintain), Holmes adopts an agentic search paradigm:

• Lightweight Indexing: Uses a function-to-file-path index (incremental updates take ~10 minutes) rather than a dense static graph.
• Iterative Expansion: The system enters a loop (max 3 iterations) where an LLM agent identifies missing logical links in the accumulated context, generates target function signatures, and fetches implementations in parallel. This dynamically compresses the search space to locate non-local root causes.

Layer 3: Synthesis & Reasoning

Acting as a lead analyst, this layer synthesizes all collected evidence (stacks, logs, threads, expanded code context) into a final diagnostic report. Key mechanisms include:

• Attention Guidance: Predicts a "blame-frame" (file:line anchor) to constrain the LLM's reasoning and prevent attention diffusion.
• Hybrid-Source Reasoning: Integrates low-level artifacts (registers, memory snapshots, assembly) to bridge the semantic gap between high-level business logic and closed-source system frameworks.
• Heuristic Rule Injection: Applies domain-specific rules (e.g., distinguishing null pointers from wild pointers based on memory addresses) to resolve ambiguities.

3. Key Contributions

1. Multimodal Agentic Diagnosis Framework: A collaborative multi-agent system that synthesizes heterogeneous signals (stacks, logs, threads, code) to reconstruct complex failure contexts, overcoming the limitations of single-modal analysis.
2. Hybrid-Source Reasoning with Low-Level Artifacts: A method to bridge the semantic gap in mixed-source environments by integrating registers and assembly code, enabling hypothesis verification across open-source and closed-source boundaries.
3. Scalable Navigation via Dynamic Compression: A retrieval-augmented strategy that leverages dynamic runtime signals to aggressively compress the search space, allowing precise navigation to non-local root causes in 70M+ LOC repositories without expensive global static analysis.
4. Industrial-Scale Evaluation: A comprehensive evaluation on real-world WeChat crashes demonstrating high accuracy and significant efficiency gains.

4. Experimental Results

The system was evaluated on a dataset of 73 real-world crash reports from the WeChat iOS production environment, annotated by senior developers with ground-truth fault locations and root causes.

• Accuracy:
  • Function-Level Fault Localization: 87.6% pass@1 accuracy.
  • File-Level Accuracy: 90.4%.
  • Root Cause Identification: 65.7% accuracy (lower due to semantic ambiguities, e.g., null vs. garbage pointers).
  • Category Accuracy: 79.4%, with near-perfect classification for Memory Errors (97.4%) and Concurrency issues (100%).
• Efficiency:
  • Time-to-Insight: Reduced average investigation time by >98%, from 2–3 hours (manual) to approximately 77 seconds (production average) or 168.5 seconds (on the complex evaluation dataset).
  • Cost: The estimated cost per session is ~$0.13, compared to >$70 for manual engineering time, representing a ~99% cost reduction.
• Ablation Studies: Removing low-level artifacts (registers/assembly) dropped root cause accuracy to 57.5%, confirming their necessity for mixed-source reasoning. Removing the Code Explorer dropped function accuracy to 68.5%, highlighting the need for dynamic exploration beyond static stack traces.

5. Significance and Claims

The paper claims that Holmes successfully transforms labor-intensive debugging into an efficient verification workflow. Its significance lies in:

• Post-Mortem Reasoning: It demonstrates that high-precision automated diagnosis is possible without reproducible environments, relying solely on sparse, read-only artifacts.
• Scalability: It proves that agentic exploration with dynamic compression can effectively navigate massive codebases where traditional static analysis fails.
• Industrial Viability: The system has been piloted with 45+ developers, achieving a 92% adoption rate for complex crash tickets and a 78% acceptance rate for AI-generated root causes without further manual inspection.

The authors note that while the paradigm is platform-agnostic, the current validation is specific to the iOS environment. They also acknowledge limitations, such as challenges with "Heisenbugs" (race conditions invisible in static snapshots) and information entropy reduction due to strict privacy redaction. Future work focuses on deploying lightweight models locally and integrating automated fix generation.