🤖 machine learning

Instant GPU Efficiency Visibility at Fleet Scale

This paper introduces Overall FLOP Utilization (OFU), a hardware-level, instrumentation-free GPU efficiency metric derived from on-chip performance counters that achieves high accuracy in predicting application-level MFU across diverse workloads and GPU generations, enabling effective fleet-scale monitoring and the detection of efficiency regressions.

Original authors: Connor Pedersen, Dong H. Ahn, Michel Migdal, Collin Neale, Nik Konyuchenko

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Connor Pedersen, Dong H. Ahn, Michel Migdal, Collin Neale, Nik Konyuchenko

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you own a massive fleet of delivery trucks (GPUs) that are supposed to be hauling heavy cargo (AI calculations) across the country. You want to know: Are these trucks actually moving, or are they just idling in traffic?

For a long time, checking this was a nightmare. You had to ask every single driver (the software) to keep a manual log of how much cargo they moved. But drivers are busy, they might forget, or they might lie about how much they carried. Plus, if you bought a new type of truck next year, you'd have to rewrite all the rules for how to count the cargo.

This paper introduces a new, smarter way to check: Overall FLOP Utilization (OFU).

Here is the simple breakdown of what the authors did and why it matters.

The Problem: The "Driver's Log" is Broken

Currently, big tech companies try to measure efficiency by asking the AI software to count its own work.

The Flaw: The software often gets the math wrong. It might think it's moving 100 boxes when it's actually only moving 50.
The Consequence: In one real-world example the authors found, a training job looked like it was running at 54% efficiency (great!), but when they checked the actual hardware, it was only running at 25% efficiency (terrible!). The software was lying because it didn't understand a new, complex type of cargo it was carrying.

The Solution: The "Speedometer and Engine Light"

Instead of asking the driver what they did, the authors built a system that looks directly at the truck's dashboard. They created a metric called OFU using two simple signals that every NVIDIA GPU already reports:

Tensor Pipe Activity: This is like a light that turns on whenever the engine is actually crunching numbers.
SM Clock Frequency: This is the speedometer, telling you how fast the engine is spinning.

By multiplying the "light on" time by the "engine speed," they get a perfect, real-time estimate of how hard the GPU is working. It doesn't matter what kind of cargo (AI model) is being hauled or what language the driver speaks; the hardware just knows if it's working.

The "Hidden Costs" (Why it's not 100% perfect)

The authors realized that even looking at the dashboard has a few quirks, so they ran thousands of tests to map them out:

The "Tile" Problem: Imagine you are packing a box. If the box is slightly too big for the items, you have to fill the empty space with bubble wrap (padding). The GPU does this too. It calculates a little bit of "bubble wrap" math that doesn't actually help the AI. The authors figured out exactly how much extra math this adds so they can subtract it from the total.
The "Speedometer Glitch": Sometimes the speedometer flickers because the engine speed changes rapidly. The authors found that if you check the speed often enough (every few seconds), the flickers average out and the number is very accurate.
The "Small Parts" Problem: The GPU has a main engine (Tensor Cores) and a tiny side engine (CUDA cores) for small tasks. The authors found that the main engine does 99.8% of the heavy lifting, so ignoring the tiny side engine doesn't really change the result.

The Results: Catching the Bugs

When they tested this new "dashboard" method against 608 real-world AI training jobs, it worked incredibly well:

It matched the truth: It correlated strongly with the actual work being done.
It caught lies: It spotted two major cases where the software was counting the work wrong. One was a complex "Mixture of Experts" model where the software forgot to count a step, making the job look twice as efficient as it really was.
It saved money: In one case with robot-training AI, the dashboard showed the trucks were idling. The team investigated and found a "debug mode" was accidentally left on, forcing the trucks to stop and check their paperwork constantly. They turned it off, and efficiency jumped 2.5 times.

The Bottom Line

The authors didn't just invent a new math formula; they built a universal, instant, and honest way to see if your AI computers are actually working.

No installation needed: You don't have to change the AI code.
Works everywhere: It works on old and new GPUs, and with any type of AI model.
Instant visibility: It tells you immediately if a job is wasting money, allowing teams to fix it before they lose millions.

In short, OFU is like installing a tamper-proof fuel gauge on every truck in your fleet, so you never have to guess if your drivers are actually delivering the goods.

Technical Summary: Instant GPU Efficiency Visibility at Fleet Scale

Problem Statement

The rapid escalation in capital expenditure for AI High Performance Computing (HPC) infrastructure—projected to exceed $2.8 trillion by 2029—has made GPU compute efficiency a critical economic factor. However, achieving fleet-wide visibility into GPU utilization is hindered by the limitations of existing measurement techniques:

User-level Profiling: Tools like NVIDIA Nsight Compute and PyTorch Profiler offer high fidelity but require per-workload instrumentation, impose runtime overhead, and are unsuitable for continuous, fleet-scale monitoring.
Framework-level Estimates: Metrics like Model FLOPs Utilization (MFU) rely on manually derived formulas based on model architecture. These are fragmented across frameworks, brittle as architectures evolve (e.g., Mixture-of-Experts, latent-space routing), and prone to silent miscalculations.
Hardware Counters: While non-intrusive, existing uses of hardware counters (e.g., via DCGM) lack rigorous accuracy characterization, validation against controlled benchmarks, and demonstrated scalability across GPU generations and numeric precisions.

There is a critical need for a utilization metric that is non-intrusive, requires no software stack modifications, covers all workloads and GPU generations, and provides bounded, well-characterized accuracy.

Methodology: Overall FLOP Utilization (OFU)

The authors propose Overall FLOP Utilization (OFU), a hardware-level, precision-agnostic metric derived from two on-chip performance counters available through NVIDIA DCGM:

Tensor Pipe Activity (TPA): The fraction of cycles during which the GPU executes Tensor Core floating-point instructions.
SM Clock Frequency ( $f_{SM}$ ): The instantaneous clock frequency of the Streaming Multiprocessors.

Derivation

The metric is defined as the product of TPA and the ratio of the current SM clock to the maximum boost clock ( $f_{max}$ ):
$\text{OFU} = \text{TPA} \times \frac{f_{SM}}{f_{max}}$

This approach relies on the observation that in modern AI workloads dominated by matrix multiplications, the Tensor Core pipeline accounts for the vast majority (approx. 99.8%) of floating-point operations. By monitoring Tensor Core activity and normalizing for clock frequency fluctuations, OFU approximates the ratio of actual FLOPs/s to peak FLOPs/s without needing model-specific information.

Error Characterization

The paper rigorously characterizes five sources of approximation error through controlled GEMM experiments on H100 and GB200 GPUs across FP16, TF32, FP8, and NVFP4:

Tile Quantization: Software constraints in GEMM kernels (e.g., cuBLAS) often pad matrices to tile boundaries, causing the hardware to execute more FLOPs than the theoretical $2MNK$ cost. The authors derive a correction factor based on profiled overhead.
Precision Scaling: Validation confirms OFU correctly tracks throughput scaling across different precisions (e.g., FP8 vs. TF32) and architectures.
Clock Sampling Noise: Analysis shows that sampling intervals of 5–30 seconds introduce negligible noise (<0.22 percentage points) relative to the metric's magnitude.
Clock Domains: The paper accounts for cases where Tensor Core pipelines operate at different maximum frequencies than the general SM boost clock (e.g., H100 FP16 vs. FP32).
Non-Tensor Undercounting: The exclusion of CUDA-core work (activations, normalization) is deemed negligible for transformer-based workloads.

Key Contributions

First-Principles Metric: A derivation of OFU as a simple product of hardware counters, providing a precision- and architecture-agnostic proxy for MFU.
Accuracy Validation: Controlled experiments on H100 and GB200 demonstrating that after tile-quantization correction, OFU predicts application-level MFU within ≤2 percentage points for 99–100% of cases.
Production Correlation: Evaluation against 608 production training jobs on H100 GPUs, achieving a correlation of r = 0.78 with application-level MFU.
Operational Deployment: Integration of OFU into large-scale fleet infrastructure (per-job dashboards, resilience services, and goodput monitoring), demonstrating its ability to detect inefficiencies invisible to application-level metrics.

Results

Controlled Benchmarks: In GEMM experiments, raw OFU consistently overestimates MFU by 1–3 percentage points due to tile quantization. Applying the derived correction tightens the error to ≤2 percentage points.
Production Jobs:
- Across 608 jobs, the mean absolute error between OFU and reported MFU was 6.2%.
- The correlation improved to r = 0.78 after excluding jobs with known framework-level FLOPs miscalculations.
- Case Study 1 (MoE): A 288-GPU DeepSeek-style MoE job reported 54.27% MFU but showed 25.58% OFU. Investigation revealed the framework incorrectly calculated FLOPs by ignoring latent-space projections (inflating counts by ~3x).
- Case Study 2 (Hybrid MoE): A hybrid architecture job showed a 57.5% relative error due to the framework failing to account for non-attention layers in its FLOPs formula.
Operational Impact:
- Embodied Agent Training: OFU detected a 2.5× efficiency regression caused by a debug flag (TORCH_DISTRIBUTED_DEBUG=DETAIL) injecting CPU-based validation calls, which serialized NCCL operations and left Tensor Cores idle.
- Mixed-Precision Pretraining: In a 6,144-GPU run, OFU successfully tracked utilization changes when switching between mixed-precision (NVFP4, FP8, BF16) and BF16-only modes, maintaining high correlation (r = 0.977 at the job level) despite the lack of precision-specific configuration in the metric.

Significance and Claims

The paper claims that OFU provides a practical, deployment-ready complement to application-level MFU for continuous fleet-wide efficiency monitoring. Its significance lies in:

Zero Instrumentation: It requires no changes to training code, model architecture, or software stacks, making it immediately applicable to heterogeneous fleets with diverse frameworks and users.
Robustness to Architectural Evolution: Unlike framework-level MFU, which breaks when new model types (e.g., Mamba, hybrid MoE) emerge, OFU remains valid as it measures actual hardware execution.
Actionable Visibility: By surfacing discrepancies between hardware reality and software reporting, OFU enables the detection of silent inefficiencies (such as library-level overheads or debug flags) that would otherwise double the effective cost of compute.

The authors conclude that while OFU is a coarse signal requiring profiling for root-cause diagnosis, it is a superior tool for identifying where inefficiencies exist across a fleet, significantly lowering the barrier to optimizing AI workloads on HPC systems.