Technical Summary: Energy-Efficient GPU DVFS for Fine-Tuning of SLMs on Resource-constrained Embedded Devices

Problem Statement

The increasing demand for on-device adaptation of Small Language Models (SLMs) in privacy-sensitive and latency-critical domains (e.g., healthcare, finance, autonomous systems) necessitates local fine-tuning rather than cloud offloading. However, fine-tuning is significantly more energy- and time-intensive than single-pass inference due to repeated forward-backward optimization over many mini-batches.

Existing Dynamic Voltage Frequency Scaling (DVFS) solutions for embedded platforms, such as the NVIDIA Jetson AGX Orin, rely on coarse-grained, static power modes (e.g., 15W, 30W, 50W, and MAXN). These modes do not guarantee energy efficiency because total energy consumption is a function of both power draw and execution time. Static modes often operate away from the energy-minimizing point, leading to avoidable energy consumption. Furthermore, while DVFS for Large Language Model (LLM) inference and datacenter training has been studied, there is a scarcity of research characterizing fine-tuning workloads on resource-constrained embedded platforms to determine optimal frequency settings.

Methodology

The authors propose a five-phase methodology to characterize fine-tuning behavior and develop an energy-efficient GPU frequency selection policy:

1. Experimental Setup & Data Collection:

   • Platform: NVIDIA Jetson AGX Orin (64GB) with an Ampere-architecture GPU and 12-core ARM Cortex-A78AE CPU.
   • Workloads: Encoder-only BERT variants (Small to xLarge) and decoder-only Pythia variants (31M to 1B parameters).
   • Benchmarks: GLUE tasks (SST-2, QNLI, MRPC for characterization; QQP, RTE, CoLA, STS-B, MNLI, WNLI for evaluation).
   • Procedure: GPU frequencies were swept across 11 discrete levels (306 MHz to 1300 MHz) while the CPU governor remained at default. Data on training time, power consumption (CPU + GPU), and utilization were collected.

2. Workload Characterization:

   • Latency: Execution time decreases almost exponentially as GPU frequency increases.
   • Power: Total power consumption is proportional to frequency ($P \propto f$), with higher frequencies often exceeding the power draw of the default MAXN mode.
   • Energy: The relationship between frequency and total energy exhibits a distinct U-shaped curve. Low frequencies increase energy due to prolonged execution time, while high frequencies increase energy due to elevated power draw. The optimal "sweet spot" for energy efficiency was found to be in the 510–816 MHz range, significantly lower than the maximum 1300 MHz.

3. ML Model Selection:

   • The problem of selecting the optimal GPU frequency is formulated as a multi-class classification task.
   • Features: Model type, GPU utilization, number of layers, hidden size, and average sequence length.
   • Target: Optimal GPU frequency level (encoded as labels 1–6, corresponding to 510–1122 MHz).
   • Algorithm: A Decision Tree (DT) classifier was selected for its interpretability. Hyperparameters were optimized using Leave-One-Out Cross-Validation (LOOCV) due to the limited dataset size (27 model-task configurations).

4. Governor Policy Design:

   • An interpretable DT-based policy (MLG) was derived to map structural features (Transformer depth and sequence length) to specific frequency settings.
   • Logic:
     • For shallow models (Layers $\le$ 7): Short sequences ($\le$ 32.5) map to 510 MHz; longer sequences map to 612 MHz.
     • For deep models (Layers > 7): Short sequences ($\le$ 53.0) map to 714 MHz; longer sequences map to 816 MHz.
     • A secondary condition adjusts frequency based on GPU utilization ($\le$ 22.0 $\rightarrow$ 1020 MHz).

5. Evaluation:

   • The proposed ML-based policy (MLG) and a fixed-frequency policy (FFG) were tested against the unconstrained MAXN Mode (Mode 0) on unseen workloads.

Key Contributions

• Comprehensive Characterization: The study provides the first detailed characterization of how GPU frequency scaling affects latency, power, and total energy for both encoder-only and decoder-only SLM fine-tuning on embedded platforms.
• Identification of Energy Sweet Spots: The authors demonstrate that the minimum-energy point is workload-dependent and follows a U-shaped pattern, often occurring at frequencies significantly lower than the platform's maximum capability.
• Lightweight ML-Based Policy: The paper proposes an interpretable Decision Tree-based governor policy that selects energy-optimal GPU DVFS settings without requiring complex runtime profiling.
• Empirical Validation: The approach is validated on the NVIDIA Jetson AGX Orin across multiple SLM architectures and GLUE benchmarks.

Experimental Results

The evaluation compared the proposed policies against the standard MAXN Mode (Mode 0) on six unseen test workloads:

• Overall Energy Savings: The ML-based governor (MLG) achieved an average energy saving of 13.11% across all models and tasks, with a maximum saving of 26.73% (observed on BERT-base for the MNLI task).
• Model Size Impact:
  • Large Models (>70M parameters): Showed stable and significant improvements, with average savings exceeding 22%. These models exhibited clearer U-shaped energy curves, making optimal frequency selection more predictable.
  • Small Models ($\le$70M parameters): Showed more variable results. While the average improvement was positive, some specific task-model combinations resulted in negative savings (increased energy) due to fluctuating U-shaped patterns or spikes in the energy-frequency curve.
• Policy Comparison: The MLG policy outperformed the Fixed Frequency Governor (FFG) by an additional 0.9% on average. The MLG was particularly effective at handling longer-sequence workloads (e.g., RTE task), whereas the FFG struggled with the variability of smaller models.

Significance and Claims

The paper claims that static power modes on embedded devices are suboptimal for SLM fine-tuning, often resulting in up to 25% avoidable energy consumption. By introducing a workload-aware, ML-based frequency selection policy, the authors demonstrate that significant energy savings can be achieved without degrading model accuracy (as validation accuracy remained identical across configurations).

The authors acknowledge limitations, noting that the current ML model has moderate accuracy (0.62 validation accuracy) and that the results for very small models can be unstable. They identify future work as expanding the dataset with more diverse SLMs and benchmarks, and employing more sophisticated ML models to further improve the stability and energy savings of the governor policy.