Technical Summary: Towards Resource-Efficient LLMs: End-to-End Energy Accounting of Distillation Pipelines

Problem Statement

The rapid deployment of Large Language Models (LLMs) has intensified concerns regarding electricity consumption, datacenter scaling, and environmental impact. While model distillation is frequently promoted as a pathway to more efficient, cheaper models, current sustainability claims often rely on incomplete accounting. Specifically, prior work typically reports only student-side training costs (e.g., FLOPs, runtime) while omitting the substantial "teacher-side" workloads required for distillation, such as synthetic data generation, logit caching, filtering, and evaluation. This omission makes it difficult to substantiate claims that distillation is inherently more energy-efficient or "greener" than standard Supervised Fine-Tuning (SFT).

Methodology

The authors propose a comprehensive, end-to-end energy accounting framework designed to measure the complete computational cost of distillation pipelines.

Experimental Setup:

• Hardware/Environment: Experiments were conducted on a single NVIDIA H100 SXM 80 GB GPU and 16 Intel Xeon Gold CPU cores to ensure consistent conditions. The software stack (Linux, drivers, CUDA) was fixed across all runs.
• Models: The study utilizes the OLMo-2 family, featuring a 32B parameter teacher model and student models of 1B, 7B, and 13B parameters.
• Tasks: Three supervised workloads were used: instruction following (TULU-3), math reasoning (OpenR1-Math-220k), and code generation (Open-R1 Codeforces).
• Distillation Regimes: Three pipelines were compared under fixed training budgets and hyperparameters:
  1. Baseline SFT: Direct training on original dataset labels.
  2. Logit-based Knowledge Distillation (KD): The teacher generates logits (cached offline), and the student is trained to match the teacher's softened distribution.
  3. Synthetic Supervised Fine-Tuning (Synthetic SFT): The teacher generates synthetic outputs, which serve as hard labels for student training.

Measurement Protocol:

• Stage-wise Tracking: The pipeline is decomposed into distinct stages: environment stabilization ($E_{prerun}$), teacher-side computation ($E_{teacher}$: logit caching or data generation), student training ($E_{student}$), and evaluation ($E_{eval}$).
• Metrics: GPU energy is measured directly via NVML telemetry (treated as ground truth). CPU energy and CO2e are estimated using standard tools (e.g., CodeCarbon-style estimators). Results are normalized to Joules per token (J/token) and total kWh.
• Quality Metrics: A scalar quality score ($Q$) is derived by averaging normalized scores across five benchmarks (AlpacaEval 2, IFEval, MT-Bench-101, GSM8K, MMLU) relative to the teacher's performance.

Key Contributions

1. Distillation Energy Protocol and Harness: The authors formalize a stage-wise logging protocol and release an open-source measurement harness. This allows for comparable energy charts that explicitly separate teacher-side costs from student training costs.
2. Controlled Benchmark with Pareto Frontiers: The study benchmarks 1B/7B/13B students against a 32B teacher across the three pipelines. It constructs energy–quality Pareto frontiers that reveal regimes where distillation is or is not efficient.
3. Design Rules and Break-even Conditions: The paper quantifies how factors like teacher reuse, sequence length, and hyperparameters (e.g., KD temperature, loss weights) shift the energy–quality frontier. It derives specific conditions under which distillation becomes energy-competitive.

Key Results

• Teacher Overhead Dominates: When accounting for the full pipeline, both KD and Synthetic SFT consume significantly more end-to-end energy than Baseline SFT for comparable quality gains. The teacher-side work (logit caching or data generation) acts as a near-fixed overhead that shifts distillation curves to the right on the energy–quality frontier.
• Scale and Efficiency: Larger student models (e.g., 13B) achieve higher quality but with diminishing returns on energy. Smaller models (e.g., 1B) minimize energy but often fail to match the quality of larger baseline SFT models.
• The Role of Reuse: Distillation is not inherently cheaper in a "one-off" scenario. However, the fixed teacher cost can be amortized. The authors derive a break-even threshold ($N^*$) for the number of student runs required to reuse teacher artifacts (cached logits or synthetic data) before distillation becomes more energy-efficient than baseline SFT. For example, a 7B student requires roughly 5–6 reuse instances to break even with KD, whereas a 13B student requires only 2–3.
• Hyperparameter Sensitivity:
  • In KD, increasing the temperature ($T$) yields modest quality gains for slight energy increases; the mixing weight ($\alpha$) is a more critical control for the trade-off.
  • In Synthetic SFT, the generation budget (max new tokens) is the dominant energy driver. Increasing token length improves quality but exhibits non-linear energy growth with diminishing returns.

Significance and Claims

The paper argues that distillation is not inherently sustainable when viewed through an end-to-end lens. The common perception of distillation as a "cheaper" method often stems from ignoring upstream teacher costs.

The authors claim that:

1. Workflow Dependency: Whether distillation is cost-efficient is primarily a workflow question (specifically, the ability to reuse teacher artifacts) rather than an intrinsic property of the algorithm.
2. Transparency: Current reporting practices obscure the true environmental cost of AI development. By adopting stage-wise accounting, practitioners can make informed decisions about when to use distillation versus baseline SFT.
3. Practical Guidelines: The paper provides concrete design rules:
   • Report both student-only and full-pipeline energy with explicit reuse assumptions.
   • Treat teacher artifacts as shared infrastructure (cached and versioned) to amortize costs.
   • Use Baseline SFT when reuse is low; favor distillation only when teacher artifacts can be reused across multiple students or hyperparameter sweeps.

The authors explicitly note that their quantitative thresholds are hardware-specific and that their analysis focuses on operational energy, excluding embodied emissions from hardware manufacturing. They position their work as a foundation for more honest reporting of compute and energy in both academic and industrial settings.