🤖 AI

Multi-Level Modeling of Large Language Model Inference Latency and Energy via Hybrid Analytical--Machine-Learning Predictors

This paper introduces HYMELL, a hybrid three-level framework that combines analytical modeling and machine learning to accurately estimate the inference latency and energy consumption of diverse Large Language Model architectures on hardware like the NVIDIA H100, achieving less than 5% error and enabling efficient, hardware-free design space exploration.

Original authors: Saeid Shokoufa, Mohammad Erfan Sadeghi, Mehdi Kamal, Massoud Pedram

Published 2026-08-10

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Saeid Shokoufa, Mohammad Erfan Sadeghi, Mehdi Kamal, Massoud Pedram

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a giant, talking robot that can write stories, solve math problems, and chat like a human. This robot is called a Large Language Model (LLM). To make it work, you need a super-fast computer brain, usually a Graphics Processing Unit (GPU), which is the same kind of chip found in high-end video game computers. But here's the catch: these robots are getting so big and smart that they are becoming incredibly expensive to run. They gulp down massive amounts of electricity and take a long time to think, which makes them hard to use in real-world situations like hospitals or schools.

The big question scientists are asking is: "How can we know exactly how much energy and time a robot will need before we even build it?" Right now, to find out, you usually have to build the robot, run it, and measure it with special tools. This is slow, expensive, and requires you to have the actual super-computer handy. If you want to try out a thousand different robot designs to find the most efficient one, you'd have to build and test them all one by one, which would take forever and cost a fortune.

This is where a team of researchers from the University of Southern California steps in with a new idea called HYMELL. Think of HYMELL as a "crystal ball" for computer scientists. Instead of building the robot and testing it, this new tool lets you predict exactly how fast and how energy-hungry a robot will be just by looking at its blueprints. It uses a clever mix of old-school math formulas and modern machine learning to guess the cost of running these giant models. The researchers tested their crystal ball on a powerful computer chip called the NVIDIA H100 and found that it can predict the robot's speed and energy use with amazing accuracy—often being wrong by less than 5%. This means engineers can now quickly try out thousands of different robot designs on a laptop to find the most efficient one, without needing to buy a super-computer or wait days for results.

The Three-Level Detective

To understand how HYMELL works, imagine you are trying to estimate how long it takes to bake a massive, multi-layered cake and how much electricity your oven will use. You could try to guess the total time by just looking at the final cake, but that's often inaccurate because you miss the small details. Instead, HYMELL acts like a three-level detective, breaking the problem down into smaller, manageable pieces.

Level 1: The Tiny Ingredients (Analytical Modeling)
First, the system looks at the smallest "ingredients" of the robot's brain. These are basic math operations like multiplying huge grids of numbers (called GEMM), normalizing data (RMSNorm), and calculating attention scores (Softmax). The researchers realized that these tiny operations behave differently depending on how big the job is.

The Analogy: Think of a small kitchen task like chopping one onion. If you only have one onion, the time it takes is mostly about how long it takes to walk to the fridge, grab the knife, and start chopping (this is "launch-bound"). But if you have to chop a mountain of onions, the time is mostly about the actual chopping speed and how fast you can move the onions around (this is "memory-bound").
HYMELL uses math formulas to calculate the cost of these tiny ingredients based on whether the job is small or huge. It doesn't guess; it uses physics-based rules to figure out the baseline cost of these operations.

Level 2: The Recipe Blocks (Machine Learning)
Next, the system groups these tiny ingredients into larger "blocks," like the Attention Block (which helps the robot focus on important words) and the Feed-Forward Network (which helps it process information).

The Analogy: Imagine you know exactly how long it takes to chop an onion and how long it takes to whisk eggs. But when you combine them to make an omelet, there are extra steps: cracking the shell, cleaning the bowl, and maybe a little bit of waiting. These extra steps are hard to calculate with simple math.
So, HYMELL uses a small, smart computer program (a Machine Learning model) to learn these "extra steps." It looks at the math-based estimates from Level 1 and adds a correction factor for the messy, real-world overheads like reshaping data or switching between tasks. This allows it to predict the cost of a whole "recipe block" very accurately.

Level 3: The Whole Cake (End-to-End Prediction)
Finally, the system puts all the blocks together to predict the cost of the entire robot running a full conversation.

The Analogy: Now you have the time for the omelet, the cake, and the salad. But baking a whole banquet involves more than just adding the times together. You have to worry about the oven heating up, the kitchen getting crowded, and the time it takes to move dishes from the counter to the table.
HYMELL uses another smart computer program to take the costs of all the blocks and add in these "system-level" effects. It considers things like how many people are asking questions at once (batch size) and whether the robot is reading a long prompt or just generating one word at a time. This gives a final, accurate prediction of the total time and energy needed.

What They Found

The researchers tested this three-level detective on a powerful NVIDIA H100 GPU using famous robot models like LLaMA 3, Mistral, and Qwen. The results were impressive.

High Accuracy: For the tiny ingredients (Level 1), the predictions were incredibly close to reality, with errors often below 4%. For the whole robot (Level 3), the system predicted the time and energy use with an error of less than 5% for many models. For example, when testing the LLaMA 3 8B model, the error was just 4.19% for time and 2.92% for energy during the "prefill" phase (reading the prompt), and even lower (around 1.5%) during the "decode" phase (generating words).
Speed and Flexibility: Because HYMELL predicts based on the blueprint (architectural parameters) rather than running the robot, it is incredibly fast. It allows engineers to explore thousands of design changes instantly. For instance, they could ask, "What happens if we double the number of attention heads?" and get an answer in seconds, showing that 64 heads might be the most energy-efficient choice for a specific setup.
Working on Different Hardware: The team also showed that this method isn't tied to just one type of computer. They tested it on a different GPU (the NVIDIA RTX A6000) and the system still worked well, with errors around 8%. This suggests that if you want to use HYMELL on a new computer chip, you don't need to reinvent the whole wheel; you just need to measure the new chip's basic behaviors once, and the rest of the system adapts automatically.

What It's Not

It is important to note what HYMELL does not do. It is not a magic wand that tells you how to build a robot from scratch, nor does it replace the need for real testing entirely. It is a prediction tool. The researchers explicitly state that while their model is very accurate, it still has small errors, especially when dealing with very complex interactions between different parts of the robot or when switching between different types of memory optimizations. Also, while they tested it on single computers, they noted that predicting how these robots work across many computers working together (multi-GPU) would require adding a few more steps to the system, which they haven't fully solved yet.

Why This Matters

The beauty of HYMELL is that it turns a slow, expensive guessing game into a fast, precise science. By combining the reliability of math formulas with the flexibility of machine learning, it gives designers a powerful tool to build greener, faster, and more efficient AI. Instead of burning through electricity and time to test every idea, they can use this "crystal ball" to find the best designs before they even build them, paving the way for a more sustainable future for artificial intelligence.

Technical Summary: Hybrid Modeling for Energy and Latency of LLMs (HYMELL)

Problem Statement

The rapid scaling of Large Language Models (LLMs) has led to substantial increases in computational costs, energy consumption, and inference latency. Accurate estimation of these metrics is critical for sustainable AI deployment and hardware-aware design. Existing approaches face significant limitations:

Purely Analytical Models (e.g., AMALI, LIFE, SweetSpot) often require extensive manual modeling, detailed per-kernel analysis, and explicit hardware specifications. They struggle with scalability when adapting to new architectures or kernels and rely on assumptions that may not hold under varying system overheads.
Proxy-Based Methods (e.g., using inference time as a proxy for energy) fail to capture fine-grained architectural effects or system-level overheads, providing only coarse-grained estimates.
Purely Data-Driven Models often lack interpretability and may not generalize well across diverse sequence lengths and batching regimes without extensive retraining.

There is a need for a framework that balances accuracy, interpretability, and adaptability to model both dense and Mixture-of-Experts (MoE) LLMs across prefill and decode phases without requiring hardware execution for every design iteration.

Methodology: The HYMELL Framework

The authors propose HYMELL (Hybrid Modeling for Energy and Latency of LLMs), a three-level hierarchical framework that combines analytical modeling for primitive operations with machine learning (ML) for higher-level components. The framework operates on an NVIDIA H100 GPU (with validation on RTX A6000) and supports both Dense and MoE architectures, as well as Multi-Head Attention (MHA) and Grouped-Query Attention (GQA).

Level 1: Analytical Estimation of Primitive Operators

This level models low-level GPU operations using regime-aware analytical equations. The authors distinguish between launch-bound (small workloads dominated by kernel launch overhead) and memory-bound (larger workloads dominated by memory traffic) execution regimes.

RMSNorm: Modeled as a memory-bound operator with fixed overhead. The estimator uses regression coefficients based on the number of tokens ( $N$ ) and hidden dimension ( $d_{model}$ ), distinguishing regimes based on total elements processed.
Softmax: Modeled for causal attention matrices ( $N \times N$ ). The estimator captures quadratic memory traffic ( $N^2$ ) and reduction depth ( $\log_2 N$ ), switching regimes based on sequence length thresholds.
GEMM (General Matrix Multiplication): The dominant computational block. The model partitions GEMMs into five regimes based on total floating-point operations ( $F$ ) and Arithmetic Intensity (AI): kernel-launch-bound, memory-bound, balanced (memory-dominant), balanced (compute-dominant), and compute-bound. Features include parallelism factors ($PF = MN/K$), memory traffic ( $B_{in}, B_{out}$ ), and interaction terms to capture GPU utilization nuances.

Level 2: ML Prediction of Block-Level Components

Analytical estimates from Level 1 serve as inputs to lightweight Multi-Layer Perceptrons (MLPs) to predict the costs of complex blocks, capturing residual overheads not explicitly modeled analytically (e.g., tensor reshaping, masking, activation functions, and routing).

Attention Block: Takes GEMM and Softmax predictions, along with architectural parameters (heads, dimensions), to predict total attention time and energy. It supports both MHA and GQA, explicitly modeling KV-cache length ( $L_{KV}$ ) during autoregressive decoding.
FFN Block: Uses GEMM predictions for up/down projections. For MoE architectures, it incorporates the number of experts and top- $k$ routing as inputs to predict the aggregate cost.

Level 3: End-to-End System-Level Estimation

A final MLP aggregates block-level predictions with global system parameters (model dimension, layer count, sequence length, batch size, inference mode, MoE configuration). This level captures system-level effects such as kernel scheduling, cross-layer synchronization, and framework overheads that are difficult to model at lower levels.

Key Contributions

Hybrid Three-Level Architecture: A novel framework that leverages the interpretability of analytical models for primitives and the flexibility of ML for complex interactions and overheads.
Regime-Aware Analytical Modeling: Explicit separation of launch-bound and memory-bound (or compute-bound) regimes for RMSNorm, Softmax, and GEMM, significantly improving accuracy for small tensors and varying workloads.
Hardware-Agnostic Portability: While trained on specific hardware, the framework is designed to be portable. Adapting to new GPUs requires only re-profiling to update analytical coefficients and fine-tuning lightweight MLPs, avoiding the need to re-derive structural equations.
Support for Diverse Architectures: The model handles dense and MoE FFNs, MHA and GQA mechanisms, and various sequence lengths and batch sizes.

Experimental Results

The framework was evaluated on an NVIDIA H100 GPU using approximately 1,200 model configurations (including LLaMA 3 8B, Mistral 7B, and Qwen 1.5 2.7B MoE).

Operator-Level Accuracy:
- RMSNorm: Achieved MAPE of 1.69% (Time) and 3.38% (Energy).
- Softmax: Achieved MAPE of 3.27% (Time) and 1.92% (Energy).
- GEMM: Achieved MAPE of 6.38% (Time) and 5.79% (Energy) across diverse regimes.
Block-Level Accuracy:
- Attention: MAPE of 1.98% (Time) and 2.42% (Energy).
- FFN: MAPE of 4.62% (Time) and 5.47% (Energy) for dense; slightly higher for MoE due to routing overhead.
End-to-End Accuracy:
- Achieved an average MAPE of 10.98% for latency and 11.78% for energy across dense and MoE models in both prefill and decode phases.
- Comparison with AMALI: On LLaMA 3 8B (batch size 1), HYMELL significantly outperformed the analytical model AMALI. HYMELL achieved a time MAPE of 4.19% (prefill) and 1.58% (decode) compared to AMALI's 15.56% and 34.90%, respectively. HYMELL also provided energy estimates, which AMALI did not.
Generalization:
- Cross-Device: When ported to an NVIDIA RTX A6000, the framework maintained strong accuracy (Average Time MAPE: 8.14%, Energy MAPE: 8.62%) after re-profiling.
- Architectural Extensibility: The framework successfully estimated linear attention variants with low error (Time MAPE: 1.91%), demonstrating adaptability to non-standard attention mechanisms.

Significance and Claims

The paper claims that HYMELL provides a fast, hardware-free surrogate for LLM inference evaluation. By predicting execution costs directly from architectural parameters, it enables:

Rapid Design Space Exploration: Researchers can sweep parameters (e.g., attention heads, model dimensions, MoE settings) to identify configurations that minimize energy or latency without running physical experiments.
Energy-Efficient Optimization: The framework supports joint optimization of architectural parameters for hardware-aware design.
Scalability: The hybrid approach avoids the complexity of fully analytical models while maintaining better generalization than purely data-driven black-box models.

The authors emphasize that while the evaluation focuses on single-GPU inference, the modular nature of the framework allows for extension to multi-GPU scaling by incorporating collective communication primitives and network bandwidth as global features. The work serves as a foundation for sustainable AI deployment by enabling accurate, early-stage energy and latency estimation.