Technical Summary: DEADPOOL

Problem Statement

State-of-the-art Large Language Model (LLM) training requires tens of thousands of GPUs operating for months, making node failures an expected operating condition rather than a rare event. Existing fault-tolerance mechanisms face a fundamental trade-off between error-free execution overhead and recovery cost:

• Checkpoint-Restart: Traditional approaches periodically write persistent checkpoints (e.g., every 1,000 steps). While they amortize overhead during normal execution, they incur significant recovery costs upon failure. This includes the latency to reload checkpoints and the "replay cost" of re-executing all training steps performed since the last checkpoint. In large-scale deployments, this replay cost can exceed the restart latency itself.
• Elastic/Exclude-and-Run: Methods like Bamboo tolerate failures by excluding faulty nodes and continuing with reduced capacity. However, these often degrade training throughput due to imbalanced workloads or require complex pipeline bubble management.
• In-Memory Checkpointing: Prior attempts to use in-memory checkpoints reduce write latency but often fail under permanent node failures because the memory state itself is lost, necessitating a full restart.

The core challenge is to design a system that achieves zero overhead during error-free execution while minimizing recovery latency and recomputation to a bounded, small fraction of a training step, without requiring a full job restart.

Methodology

The authors present DEADPOOL, a fault-tolerant system designed for LLM training via hot-swapping. Instead of restarting the entire job, DEADPOOL treats node failure recovery as an online topology-repair problem, replacing failed nodes with spares while maintaining training continuity. The system is built on two primary insights:

1. Asynchronous Offload Pipeline (Zero-Overhead Checkpointing)

To achieve zero overhead during error-free execution, DEADPOOL employs an asynchronous, two-phase offload pipeline that overlaps state protection with the training critical path (forward pass, backward pass, and gradient synchronization) within a single iteration:

• Device-to-Host (D2H): The full rank state (optimizer shards, local model parameters, and metadata) is copied to pinned host memory. This allows for fast local recovery in case of transient errors.
• Host-to-Host (H2H): Only the optimizer state shards and metadata are transmitted to a peer node in a Data-Parallel (DP) ring topology. Model parameters are not replicated over the network; they are reconstructed from healthy peers sharing the same tensor/pipeline parallelism coordinates upon failure.
• Overlap Strategy: By transmitting only the optimizer shards (which are smaller than full states) and utilizing separate execution streams (CUDA streams for D2H, MPI threads for H2H), the system ensures that communication completes within the compute window of a single training step. This eliminates exposed stalls on the critical path.

2. Distributed Communicator Reconstruction (Hot-Swapping)

Upon detecting a permanent node failure (via heartbeats or NCCL timeouts), DEADPOOL executes a recovery policy that avoids full job termination:

• Topology Repair: The system identifies a spare node, updates the set of involved nodes, and reconstructs the distributed communication groups (e.g., NCCL communicators) to include the new node.
• State Reconstruction:
  • Model Parameters: Reconstructed from healthy ranks sharing the same logical partition via targeted collective operations.
  • Optimizer States: Restored directly from the in-memory replicas maintained by the H2H pipeline.
• Resumption: Training resumes from the last completed step with at most one step of re-execution (the interrupted step), preserving step atomicity.

Error Classification

DEADPOOL includes a layer to distinguish between transient failures (handled via local reset/re-initialization) and permanent failures (handled via node replacement). This ensures that topology changes only occur when necessary, maintaining system invariants regarding shard completeness and topology consistency.

Key Contributions

1. DEADPOOL System: An online hot-swapping recovery mechanism for large-scale LLM training that replaces failed nodes without terminating the job.
2. Asynchronous Neighbor-Memory Replication: A novel strategy for replicating optimizer state shards to DP-ring peers, enabling per-iteration checkpointing with zero overhead on the training critical path.
3. Scalable Evaluation: Comprehensive evaluation across scales (up to 512 NVIDIA A100 GPUs and 64 GH200 GPUs) and model sizes (up to 65B parameters), demonstrating zero overhead during error-free execution and consistent recovery times.

Experimental Results

The authors evaluated DEADPOOL on production HPC systems (NERSC Perlmutter and TACC Vista):

• Zero Overhead: In error-free execution, DEADPOOL introduces negligible runtime overhead. Across weak-scaling experiments (32 to 256 GPUs) and various 3D parallel topologies, the per-iteration training time of DEADPOOL closely tracks the baseline (original training without DEADPOOL), with differences attributed to system noise rather than checkpointing costs.
• Recovery Latency: DEADPOOL consistently completes node replacement and restores training in under 40 seconds (specifically 32.6s to 29.8s in tested scenarios).
  • This is a >3.7x reduction compared to traditional checkpoint-restart (which took ~150s including initialization and loading).
  • Recovery latency is largely independent of model size and memory pressure; it scales primarily with the time required for communicator reconstruction (topology repair).
• End-to-End Efficiency: In a simulated 100K-step training job with realistic failure rates, DEADPOOL reduced the total failure-induced recovery time to 5.3% of the time required by checkpoint-restart, representing an 18.8x reduction in time to resume training progress.

Significance and Claims

The paper claims that DEADPOOL simultaneously achieves two previously conflicting objectives: zero-overhead error-free execution and extremely low recovery cost. By decoupling recovery performance from model size and avoiding the replay and restart costs inherent in checkpoint-based approaches, DEADPOOL offers a practical and scalable solution for fault-tolerant LLM training.

The authors note that while the zero-overhead property depends on specific hardware configurations (GPU cluster, model size, deployment strategy), the design relies on generic capabilities (sharded states, spare node management) and is expected to remain effective as GPU compute capabilities and interconnect bandwidth increase in future supercomputers. The system transforms failure recovery from a disruptive, global event into a bounded, local topology repair process.