EROICA: Online Performance Troubleshooting for Large-scale Model Training

This paper presents EROICA, the first online troubleshooting system deployed on production-scale GPU clusters (~100,000 GPUs) that effectively diagnoses complex hardware and software performance issues in large-scale model training through fine-grained profiling and differential observability with minimal impact.

The Gist

Imagine you are the conductor of a massive orchestra with 100,000 musicians (GPUs) playing a complex symphony (training a giant AI model). The goal is to play the song perfectly and quickly.

But sometimes, the music slows down, gets stuck, or sounds terrible. In the past, figuring out why was like trying to find a single out-of-tune violin in a stadium full of 100,000 people while blindfolded.

EROICA is a new "super-conductor" system designed to solve this. Here is how it works, explained simply:

1. The Problem: The "Blind Spot"

Before EROICA, engineers had two bad options to find the problem:

• Option A (The Wide-Angle Lens): They could watch the whole orchestra from a distance. They could see that the music was slow, but they couldn't tell which musician was playing wrong. It was too blurry to be useful.
• Option B (The Microscope): They could zoom in on one musician with a super-microscope to see exactly what they were doing. But this took so much time and energy that they could only do it on one person at a time. By the time they found the problem, the orchestra had already played for hours, wasting money.

2. The Solution: EROICA's "Smart Snapshot"

EROICA is the first system that can do both at the same time: it watches the whole orchestra and sees every tiny detail, without slowing anyone down.

It works in three clever steps:

Step 1: The "Traffic Jam" Detector

EROICA constantly checks the speed of the music. If the orchestra suddenly slows down (a "performance degradation"), it instantly triggers a 20-second "super-snapshot" of every single musician simultaneously.

• Analogy: Imagine a traffic camera that only snaps a photo of every car on a highway the exact second a traffic jam starts, rather than filming the whole day.

Step 2: The "Summarizer" (The Magic Trick)

This is the most important part. Usually, a snapshot of 100,000 musicians would create a mountain of data (terabytes of video) that no computer could analyze quickly.
EROICA doesn't save the video. Instead, it asks every musician to write a tiny 3-line report:

1. How much of the time were you actually playing? (Did you sit idle?)
2. How hard were you working? (Was your heart rate high?)
3. Was your rhythm steady or shaky? (Did you stutter?)

• Analogy: Instead of watching a 2-hour movie of the concert, EROICA asks every musician to send a single text message saying: "I played 90% of the time, my heart rate was normal, but my rhythm was shaky."
• Result: Instead of 3 Terabytes of data, EROICA only receives 30 Kilobytes of text. It's like shrinking a library down to a single postcard.

Step 3: The "Detective"

The central computer takes all those tiny text reports and compares them.

• If 99,999 musicians say "My rhythm was steady," but one says "My rhythm was shaky," the computer instantly knows: "That one guy is the problem!"
• It can also spot if everyone is playing slowly because the conductor (the code) is giving bad instructions.

3. Real-World Superpowers

Because EROICA is so smart and fast, it has solved problems that were previously impossible:

• The Broken Wire: It found a single broken network cable connecting two computers in a cluster of thousands, which was slowing down the whole group.
• The Lazy Worker: It spotted a specific computer that was doing extra, unnecessary work (like a musician practicing scales while everyone else was playing the song), causing the rest to wait.
• The AI Auto-Fix: In one case, EROICA found a specific line of code causing a "deadlock" (a traffic jam in the software). It fed this information to an AI assistant, which wrote the fix code instantly, restarting the training without human help.

Why It Matters

Before EROICA, fixing these issues took days or weeks of guessing and testing. Now, EROICA can diagnose a problem in a 3,000-computer cluster in 3 minutes and a 1,000,000-computer cluster in 7 minutes.

It turns the impossible task of finding a needle in a haystack into simply asking the haystack, "Who is holding the needle?" and getting an immediate answer.

Technical Summary

Here is a detailed technical summary of the paper "EROICA: Online Performance Troubleshooting for Large-Scale Model Training."

1. Problem Statement

Large Model Training (LMT) on massive GPU clusters (e.g., ~100,000 GPUs) faces significant challenges in performance troubleshooting.

• Scale and Complexity: Modern LMT involves complex interactions between hardware (GPUs, NVLink, NICs, CPUs) and software (PyTorch, NCCL, custom data loaders). Performance issues can stem from hardware faults, software bugs, misconfigurations, or a mixture of both.
• Limitations of Existing Tools:
  • Online Monitoring: Traditional tools (e.g., DCGM, eBPF) provide coarse-grained data (second-level sampling) to cover entire clusters. They can detect general symptoms (e.g., throughput drops) but fail to pinpoint fine-grained root causes (e.g., specific code lines or network links).
  • Offline Profiling: Tools like Nsight Systems or Torch Profiler offer fine-grained data (microsecond-level) but generate massive overhead (hundreds of MBs per second per worker). They cannot be run on all workers in a production cluster simultaneously due to data volume and performance impact. Engineers often resort to small testbeds, which lack timeliness and fail to reproduce large-scale specific issues.
• The Gap: There is a lack of a system that offers fine-grained observability (like offline profiling) across all machines (like online monitoring) with minimal overhead in a production environment.

2. Methodology: EROICA

EROICA is the first online troubleshooting system designed to bridge this gap. Its core insight is leveraging the homogeneous architecture of LMT: in a well-optimized training job, function execution patterns across workers should be highly consistent. EROICA uses differential observability to identify anomalies without analyzing raw data.

Key Components:

1. Triggered Online Profiling:

   • EROICA monitors training iteration times (from dataloader.next() to optimizer.step()) without accessing user code.
   • When a performance degradation is detected (e.g., iteration time exceeds the recent average by >5%), it triggers a synchronized profiling session across all workers simultaneously.
   • It uses a 20-second profiling window, collecting execution events (Python, CPU, CUDA) and high-frequency hardware metrics (10 kHz) via Torch Profiler and nsys.

2. Runtime Behavior Pattern Summarization:

   • Instead of aggregating raw profiling data (which would be terabytes), EROICA summarizes the behavior of functions on the critical path into compact statistical patterns.
   • Critical Path Definition: Functions prioritized by relevance to GPU computation (GPU kernels > Memory Ops > Collective Comm > Python functions).
   • The Pattern Vector ($P_{f,w}$): For each function $f$ on worker $w$, EROICA computes a 3-dimensional vector:
     • $\beta$ (Critical Path Duration): Percentage of time the function blocks the critical path.
     • $\mu$ (Resource Utilization): Average hardware resource usage (e.g., GPU SM frequency, NIC bandwidth) during the critical execution duration (excluding idle wait times).
     • $\sigma$ (Resource Fluctuation): Standard deviation of resource usage.
   • Efficiency: This reduces data size from ~3 GB per worker (raw) to ~30 KB (patterns), a $10^5$ reduction. Crucially, these metrics are independent of absolute timestamps, eliminating the need for precise clock synchronization across hosts.

3. Root Cause Localization:

   • EROICA compares the behavior patterns of all workers to identify outliers using two distance metrics:
     • Distance from Expectation ($D_{f,w}$): Measures deviation from a pre-defined "healthy" range based on production experience (e.g., Python functions should rarely block the critical path).
     • Differential Distance ($\Delta_{f,w}$): Measures how unique a worker's behavior is compared to the median of the cluster.
   • Algorithm: A function is flagged as abnormal if it significantly contributes to the critical path ($\beta > 0.01$) AND either deviates from the expected range OR is a statistical outlier compared to peers.
   • Output: The system identifies specific functions on specific workers that are causing the slowdown (e.g., "Worker 7 has a slow NIC link," or "Worker 0-3 have inefficient GEMM kernels").

3. Key Contributions

• First Online Profiling System for LMT: EROICA is the first system to provide fine-grained, cluster-wide observability for production LMT without prohibitive overhead.
• Differential Observability Technique: Introduces a novel method of summarizing runtime behavior patterns (statistical metrics) rather than analyzing raw traces, enabling scalable analysis of massive clusters.
• Production Deployment: Successfully deployed on Alibaba Cloud's GPU clusters (~100,000 GPUs) for 1.5 years.
• AI Integration: The system's output is designed to be fed directly into AI coding assistants (like Cursor) to automatically generate fixes for code-level bugs.

4. Results and Evaluation

• Success Rate: EROICA diagnosed 97.5% (78 out of 80) of serious performance issues that existing state-of-the-art tools failed to resolve.
• Performance Impact:
  • Overhead: The profiling window introduces a one-time data generation delay (~20 seconds). Pattern summarization and localization occur asynchronously, adding zero overhead to the training loop after the initial window.
  • Scalability: The system can analyze a 1,000,000-GPU cluster in under 7 minutes (3 minutes for localization alone).
• Case Studies:
  • Code Issues: Identified inefficient data loading (socket throughput) and asynchronous garbage collection causing GPU idle time in a 3,072-GPU job.
  • Mixed Issues: Detected a downed NIC, unoptimized network flow scheduling, and load imbalance in a 3,400-GPU video generation job, improving throughput by ~34%.
  • Auto-Fix: Successfully identified a distributed deadlock in a 128-GPU robotics model and used an AI assistant to automatically patch the code.

5. Significance

EROICA represents a paradigm shift in LMT operations:

• From Reactive to Proactive: It moves troubleshooting from post-mortem offline analysis to real-time online diagnosis.
• Scalability: It solves the "data deluge" problem of fine-grained profiling by compressing data into actionable statistical patterns, making it feasible for the next generation of million-GPU clusters.
• AIOps Foundation: By structuring diagnostic data into clear patterns, EROICA serves as a critical foundation for AI-driven IT operations (AIOps), enabling automated root cause analysis and code repair.
• Practical Impact: It directly addresses the "needle in the haystack" problem in large-scale AI infrastructure, saving significant compute resources and reducing production incidents.