Technical Summary: HetCCL

Problem Statement

The rapid scaling of Large Language Models (LLMs) necessitates the use of massive GPU clusters. However, practical constraints such as cost, supply chain limitations, and incremental upgrades are driving enterprises to adopt mixed-vendor heterogeneous clusters (e.g., combining NVIDIA, AMD, Huawei, and other accelerators).

Existing collective communication libraries (CCLs) fail to support these environments effectively:

1. Device-centric libraries (e.g., NCCL, RCCL) are highly optimized for single-vendor hardware but lack cross-vendor interoperability due to vendor-specific programming models and kernel implementations.
2. Host-centric libraries (e.g., Gloo, OpenMPI) offer hardware compatibility but rely on copying data from device memory to host memory (CPU bounce buffers) for transfer. This introduces significant host-device memory copy overhead and PCIe bottlenecks, severely degrading performance in large-scale LLM training where communication costs are critical.

The core challenge is designing a collective communication framework that enables efficient, low-overhead data transfer and reduction across heterogeneous devices without compromising the performance of homogeneous clusters.

Methodology: HetCCL Architecture

The authors propose HetCCL, a framework that enables heterogeneous collective communication by decoupling control logic from the data path and introducing a hierarchical topology abstraction. The design rests on three key pillars:

1. Host-Driven Device-Buffer RDMA (Decoupled Control and Data Paths)

To eliminate host-device memory copy overhead while maintaining cross-vendor compatibility, HetCCL adopts a novel host-driven device-buffer RDMA mechanism:

• Control Path (Host-side): All control logic (connection management, memory region registration, work request posting, and event handling) is offloaded to a host proxy. This ensures compatibility across different vendors' programming models (e.g., CUDA, ROCm) without requiring vendor-specific device kernels.
• Data Path (Device-side): The actual data movement remains entirely on the device. The sender copies data from the application buffer to an RDMA send buffer on the device; the RNIC transfers this buffer directly to the remote peer; the receiver copies the data from the RDMA receive buffer to the target device buffer.
• Coordination: To maintain execution dependencies (e.g., stream ordering) without device-side control logic, HetCCL uses host functions launched on the device stream as synchronization placeholders.

2. Vendor-Independent Reduction via Border Communicators

For combining collectives (e.g., AllReduce, ReduceScatter), HetCCL avoids writing custom reduction kernels for every hardware type. Instead, it utilizes a border-communicator mechanism:

• Topology Abstraction: The heterogeneous cluster is decomposed into homogeneous sub-clusters (groups of devices from the same vendor). Within each sub-cluster, native vendor CCLs are used for efficient homogeneous operations.
• Border Ranks: Specific ranks within each sub-cluster are designated as "border ranks" to handle inter-cluster communication.
• Reduction Strategy: For global reduction, data is transferred to the border ranks of the destination cluster. The reduction computation is then performed using the native reduction interface of the destination cluster's vendor CCL. This achieves vendor independence by leveraging existing homogeneous libraries rather than implementing new cross-vendor kernels.

3. Hierarchical Algorithm and Pipelined Execution

HetCCL introduces a hierarchical algorithm design to balance flexibility and efficiency:

• Cluster-Level Primitives: Collective operations are broken down into three steps: (1) intra-cluster homogeneous operations, (2) inter-cluster (C2C) data transfer or reduction, and (3) final intra-cluster operations.
• Pipelined Execution: To prevent bandwidth underutilization caused by sequential execution, HetCCL implements a pipelined workflow. It overlaps independent inter-cluster data transfers with intra-cluster collective operations, maximizing the utilization of cross-cluster links.

Key Contributions

1. First Full Device-Buffer Heterogeneous CCL: HetCCL is the first library to support full device-buffer data transfer and MPI-style collective operations across mixed-vendor clusters, eliminating the CPU forwarding bottleneck.
2. Novel Host-Device Coordination: It successfully decouples RDMA control (host) from data movement (device), achieving high performance without relying on vendor-specific device kernels for control logic.
3. Portable Reduction Mechanism: By utilizing border communicators and native vendor libraries for reduction, it solves the portability challenge for combining collectives without offloading computation to the CPU.
4. Hierarchical Topology Abstraction: It provides a systematic way to model heterogeneous clusters as interconnected homogeneous sub-clusters, optimizing data transfer volumes and bandwidth utilization.

Evaluation Results

The authors implemented HetCCL in C and Python (>30k LOC) and integrated it with PyTorch. Evaluation was conducted on a cluster with hardware from four vendors (NVIDIA A800, and three anonymized vendors V1, V2, V3).

• P2P Performance: In heterogeneous SendRecv operations, HetCCL achieves 17–19× higher bandwidth than Gloo (the standard host-forwarding baseline). It also achieves 13–45% higher bandwidth than the lower-bandwidth vendor's native P2P in asymmetric settings.
• Collective Performance:
  • AllGather: Achieves 85.7%–97.8% of the bandwidth of the slower homogeneous vendor implementation.
  • AllReduce: Achieves up to 70.8% of the bandwidth of the slower vendor implementation.
  • C2C Efficiency: In a 4-node, 32-GPU cluster, the C2C algorithm achieves 97.4% of the bandwidth of native NCCL for AllGather.
• End-to-End Training:
  • For Llama3-3B and Llama3-8B models, HetCCL reduces per-step time by 9.1% and 16.9%, respectively, compared to host-forwarding approaches.
  • The paper notes that speedups are more pronounced in larger, more communication-intensive scenarios.
• Serving Performance: In serving Qwen2-7B with prefill-decode disaggregation, HetCCL reduces Time to First Token (TTFT) by >65% compared to host-forwarding baselines.

Significance and Claims

The paper claims that HetCCL addresses a critical gap in the infrastructure for modern LLM training, where reliance on single-vendor homogeneous clusters is becoming impractical. By enabling efficient mixed-vendor training without sacrificing performance, HetCCL allows enterprises to:

• Utilize fragmented and diverse compute resources (e.g., legacy hardware, different cloud instances) effectively.
• Avoid vendor lock-in and procurement constraints.
• Achieve near-native performance in heterogeneous environments, making mixed-vendor clusters a viable alternative to homogeneous ones for large-scale AI workloads.

The authors emphasize that their solution maintains the "on-device" data path principle, which is essential for high-performance computing, while solving the portability issues that previously forced a trade-off between efficiency and compatibility.