💬 NLP

Hardware Co-Design Scaling Laws via Roofline Modelling for On-Device LLMs

This paper proposes a principled hardware-software co-design framework that uses scaling laws and roofline modeling to jointly optimize model accuracy and inference latency, enabling the rapid identification of Pareto-optimal architectures for resource-constrained on-device LLM deployment.

Original authors: Luoyang Sun, Jiwen Jiang, Yifeng Ding, Fengfa Li, Yan Song, Haifeng Zhang, Jian Ying, Lei Ren, Kun Zhan, Wei Chen, Yan Xie, Cheng Deng

Published 2026-02-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Luoyang Sun, Jiwen Jiang, Yifeng Ding, Fengfa Li, Yan Song, Haifeng Zhang, Jian Ying, Lei Ren, Kun Zhan, Wei Chen, Yan Xie, Cheng Deng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to pack a suitcase for a trip. You have two main goals: you want to bring as many cool things as possible (this is Accuracy), but you also have a very strict weight limit and you need to be able to move quickly through the airport without being slowed down (this is Latency/Speed).

If you pack heavy, high-quality items, you’ll be accurate in your "travel style," but you’ll be too slow and heavy. If you pack only light, cheap items, you’ll be fast, but you’ll have a boring trip.

This paper, "Hardware Co-Design Scaling Laws," is essentially a mathematical "Master Packing Guide" for engineers building Artificial Intelligence (LLMs) that need to live inside small devices like robots, self-driving cars, or smartphones.

Here is the breakdown of how they solved this problem:

1. The Problem: The "Brain" vs. The "Body"

Usually, AI researchers focus only on making the "brain" smarter (more parameters). But when you put that brain into a small device (the "body"), the body has limits. A self-driving car can't wait 5 seconds for the AI to "think" before it hits the brakes. The hardware (the body) has a limited amount of "muscle" (computing power) and "blood flow" (memory bandwidth).

In the past, engineers would build a huge AI and then try to "shrink" it to fit the device. This paper says: "Don't build first and shrink later. Design the brain and the body together from day one."

2. The Solution: The "Roofline" Map

The researchers use something called a Roofline Model. Imagine a house with a slanted roof.

If you are walking under the low part of the roof, you are limited by how much space you have (this is Memory-Bound—the device is struggling to move data around).
If you are under the high part, you are limited by how fast you can run (this is Compute-Bound—the device is struggling to do the math).

By using this "map," the researchers can predict exactly where an AI model will get "stuck" on a specific piece of hardware (like an NVIDIA Jetson chip) before they even build it.

3. The Secret Sauce: "Wide and Shallow" (The MoE Strategy)

The most surprising discovery in the paper is how the "brain" should be shaped.

Most big AIs are like a deep, narrow tower: many layers of logic stacked on top of each other. But for small devices, the researchers found that it is much better to build a "Wide and Shallow" brain using something called Mixture of Experts (MoE).

The Analogy:

Standard AI (Deep/Narrow): Imagine a single, incredibly smart professor who has to read every single page of every single book in the library to answer one question. It’s very smart, but it takes forever.
Co-Designed AI (Wide/Shallow/MoE): Imagine a massive library filled with 16 different specialists (the "Experts"). When you ask a question about biology, you don't wake up the math expert or the history expert; you only wake up the biologist.

Because you only "activate" a few experts at a time, the AI stays incredibly smart (high capacity) but stays lightning-fast (low latency) because it isn't doing unnecessary work.

4. Why does this matter?

Before this paper, choosing the right AI model for a robot could take months of trial and error. The researchers created a mathematical formula that does the math for you. Now, it takes days.

The Result: They created a custom-designed AI that is just as fast as a standard model (like Qwen), but it is 19% more accurate. In the world of self-driving cars or medical robots, a 19% increase in "intelligence" at the same speed is a massive leap forward.

Summary in one sentence:

Instead of building a giant brain and trying to squeeze it into a small robot, this paper provides a mathematical blueprint to build a "specialist" brain that is perfectly sized for the robot's muscles from the very start.

Technical Summary: Hardware Co-Design Scaling Laws via Roofline Modelling for On-Device LLMs

1. Problem Statement

The deployment of Large Language Models (LLMs) on resource-constrained edge devices (e.g., autonomous vehicles, robots, and mobile devices) presents a fundamental tension between model accuracy and inference efficiency. Traditional LLM architectures are optimized for high-throughput cloud GPUs, but on-device platforms face strict constraints on memory bandwidth, peak compute, and total memory capacity.

Existing Neural Architecture Search (NAS) methods often optimize for a single objective (like validation loss) without considering the specific hardware-software interplay. Consequently, a model that is "optimal" in terms of parameters may be unusable due to high latency or memory footprint on a specific SoC (System-on-Chip). There is a lack of a principled framework that jointly models how architectural hyperparameters (depth, width, sparsity, etc.) interact with hardware characteristics (Roofline limits) to determine the accuracy-latency Pareto frontier.

2. Methodology

The authors propose a Hardware Co-Design Scaling Law framework that integrates two distinct modeling domains:

Precision Modeling (Loss Scaling Laws): Instead of training every candidate architecture, the authors model the validation loss $\hat{L}(\theta)$ as an explicit analytical function of architectural hyperparameters (depth $l$ , width $d$ , FFN expansion ratio $r$ , MoE activation rate $\rho$ , and GQA ratio). They empirically fit this using a polynomial approximation derived from training 170 diverse architectures (dense and MoE) for 10B tokens each.
Performance Modeling (Roofline-based Latency): The authors use the Roofline Model to characterize inference latency. They classify operators as either compute-bound or memory-bound based on their arithmetic intensity. They derive mathematical expressions for end-to-end latency, accounting for both the Prefill phase (compute-bound, $O(S^2)$ complexity) and the Decode phase (memory-bandwidth-bound, $O(S)$ complexity), including the impact of the KV-cache.
Pareto-Optimal Architecture Search (PLAS): By coupling the loss model with the latency model, the framework identifies the Pareto frontier—the set of architectures that offer the lowest possible loss for any given latency budget.
Theoretical Optimization: The authors extend the empirical findings into a formal mathematical framework, casting architecture search as a constrained optimization problem. They derive closed-form solutions for optimal parameters across different hardware regimes: Latency-constrained, Memory-constrained, and Dual-constrained.

3. Key Contributions

First Principled Co-Design Law: The paper introduces the first operational framework that treats hardware constraints and model scaling as a unified optimization problem for on-device LLMs.
Empirical Scaling Law for Architecture: A high-fidelity loss prediction model that captures how sparsity (MoE) and width/depth trade-offs affect model quality.
Hardware-Aware Design Principles: The derivation of specific "rules of thumb" for edge deployment, such as the Width-Sparsity Scaling Law ( $\rho^* \propto d^{(\beta_1-\beta_2)/\alpha_\rho}$ ), which dictates how sparsity should scale with model width under memory constraints.
Significant Efficiency Gains: The framework reduces the architecture selection process from months of manual tuning/training to just a few days of theoretical prediction and validation.

4. Key Results and Findings

MoE Dominance: Under the batch-size-1 constraint typical of edge devices, Mixture-of-Experts (MoE) architectures are 100% of the Pareto-optimal configurations. They provide higher capacity with the same per-token compute cost as dense models.
"Wide-and-Shallow" Preference: Unlike cloud models that favor depth, optimal on-device models are relatively shallow (typically <20 layers) and wide. Width provides more efficient loss reduction per unit of latency than depth under strict constraints.
Phase-Specific Optimization:
- Prefill-optimized models favor fewer experts and larger GQA ratios.
- Decode-optimized models favor more experts to maximize capacity since increasing the expert pool has a negligible impact on per-token decode latency.
Empirical Validation: On an NVIDIA Jetson Orin, a co-designed architecture achieved 19.42% lower perplexity on WikiText-2 compared to Qwen2.5-0.5B while maintaining the same inference latency.
Quantization Impact: INT8 quantization shifts the Pareto frontier toward lower latency, though gains are sub-linear due to the overhead of non-linear operations (softmax, layer norm) that remain in higher precision.

5. Significance

This work is highly significant for the field of Physical AI and Embodied AI. As LLMs move from data centers to autonomous systems, the ability to rapidly design "hardware-native" models is critical. By providing a mathematical bridge between transformer architecture and the Roofline model, the authors enable engineers to design highly efficient, high-performance models tailored to specific hardware budgets, accelerating the deployment of intelligent agents in real-world, resource-constrained environments.