Profiling-Driven Adaptive Distributed Transformer Inference on Embedded Edge Deployment

Technical Summary: Profiling-Driven Adaptive Distributed Transformer Inference on Embedded Edge Deployment

Problem Statement
Deploying Transformer models on embedded edge devices, such as the NVIDIA Jetson Orin Nano, is constrained by limited memory and compute resources. While distributing inference across multiple devices offers a potential solution, prior research relying on simulations has failed to account for specific hardware bottlenecks inherent to embedded System-on-Chip (SoC) architectures. Specifically, the lack of PCIe or NVLink pathways in integrated GPUs prevents the use of efficient GPU-to-GPU communication backends like NCCL. Consequently, distributed inference must rely on the GLOO backend, which forces all inter-device data to be staged through host CPU memory. This paper identifies that this CPU–GPU staging overhead, rather than network bandwidth, is the dominant bottleneck. For medium-sized models (e.g., ViT) operating at small batch sizes, the fixed cost of staging and synchronizing full intermediate tensors renders standard distributed approaches (like "Voltage") slower and less energy-efficient than single-device inference.

Methodology
The authors present a hardware prototype study using two NVIDIA Jetson Orin Nano devices connected via WiFi 6E. The study evaluates a system called Prism, which employs Segment Means (SM) compression to reduce communication volume.

• Segment Means Communication: Instead of exchanging full intermediate feature vectors (as in position-wise partitioning), devices exchange column-wise averages of non-overlapping token segments. This reduces communication volume by a factor controlled by the Compression Rate (CR).
• Profiling-Driven Adaptation: Recognizing that the optimal execution mode (local vs. distributed) depends on dynamic factors like batch size, network bandwidth, and compression rate, the authors propose an adaptive policy. This involves an offline profiling phase where the system sweeps through various configurations (batch sizes 1–32, CR values, and network bandwidths 200–900 Mbps) to build a performance map.
• Runtime Decision: At runtime, the terminal device queries this map to select the execution mode (single-device or distributed with the optimal CR) that minimizes latency or energy for the current workload conditions.

Key Contributions

1. Hardware Prototype and Characterization: The authors developed a working prototype instrumented for latency breakdown and per-sample energy measurement. They quantified the CPU–GPU staging overhead imposed by the GLOO backend, demonstrating that full-tensor exchange methods are counter-productive on Jetson hardware across all tested batch sizes.
2. Adaptive Inference Policy: They introduced a lightweight, profiling-driven policy that dynamically switches between local and distributed execution. This policy identifies a "crossover point" (batch size) below which local execution is preferred due to fixed communication/staging costs.
3. Bandwidth Sensitivity Analysis: Experiments revealed that while full-tensor exchange (Voltage) degrades severely at lower bandwidths, Prism's compact Segment Means communication maintains low latency across the tested range (200–900 Mbps), effectively mitigating the staging bottleneck.

Results
Experiments on ViT models with CIFAR-10 inputs yielded the following results:

• Latency and Energy Reduction: Compared to a static distributed setup using full-tensor exchange (Voltage), the adaptive Prism strategy reduced latency by 65%–77% and energy consumption by 34%–52% across all batch sizes.
• Staging Dominance: For the Voltage baseline, CPU–GPU I/O transfer dominated the latency budget, often exceeding the total latency of single-device inference even at batch size 1. Prism reduced the staged data volume by approximately 90%, shifting the bottleneck back to computation.
• Accuracy Trade-offs: At a high compression rate (CR = 9.9), accuracy dropped from 98.01% to 95.64% but recovered to 97.43% after fine-tuning. Lower compression rates (CR = 3.3) retained 97.06% accuracy without fine-tuning but offered less communication speed-up.
• Crossover Point: The profiling policy correctly identified that for small batch sizes (1–4), single-device execution is superior. Distributed execution only becomes beneficial at larger batch sizes (e.g., batch 8 and above) where the communication cost is amortized over more samples.

Significance and Claims
The paper claims that profiling-driven adaptation is essential for practical distributed Transformer inference on embedded hardware. The authors assert that theoretical models or simulations are insufficient because they overlook the specific CPU–GPU staging costs of integrated GPUs. By demonstrating that a simple offline profiling sweep can guide runtime decisions, the work provides a practical foundation for deploying foundation models at the edge. The study concludes that while compression (Prism) significantly mitigates the GLOO staging bottleneck, it does not eliminate it; therefore, adaptive policies that fall back to local execution when overheads are too high are necessary for real-world edge deployments. The authors position their testbed and methodology as a rigorous characterization of the benefits and remaining limitations of GLOO-based distributed inference.