🤖 AI

Transferable Latency Prediction for Fast LLM Screening on Heterogeneous Edge Devices

This paper proposes a runtime-aware latency prediction framework that adaptively fuses static model descriptors with dynamic hardware telemetry to enable accurate, transferable LLM deployment screening across heterogeneous edge devices with minimal calibration.

Original authors: Xiaolong Tu, Vinod K. Mishra, Venkat R. Dasari, Anu G. Bourgeois, Haoxin Wang

Published 2026-07-27

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiaolong Tu, Vinod K. Mishra, Venkat R. Dasari, Anu G. Bourgeois, Haoxin Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to pick the perfect video game to play on a handheld console. You have a massive library of games, from tiny puzzle adventures to massive, sprawling role-playing epics. But here's the catch: your console is a bit finicky. It gets hot if you play too long, its battery drains fast, and it handles different games in weird ways depending on how you hold it or what time of day it is. If you try to download and test every single game to see which one runs smoothly, you'd spend all day waiting for downloads, only to find out half of them crash or lag. You need a way to guess, just by looking at the game's description and your console's current mood, which one will actually work. This is the exact problem facing the world of Artificial Intelligence on our phones and small gadgets.

Scientists are trying to run "Large Language Models" (LLMs)—super-smart AI brains that can write stories, answer questions, and solve problems—directly on our devices instead of sending data to giant cloud servers. This is great for privacy and speed, but it's a nightmare to predict. An AI model might run perfectly on one phone but freeze on another, even if the phones look similar. Why? Because the AI's speed depends on a chaotic mix of things: how long the question is, how hot the phone is getting, how much battery is left, and even the specific software used to run the AI. If we can't predict how fast an AI will run before we actually try it, we waste huge amounts of time and energy testing models that might fail. This is where the story of "predicting the future" comes in.

The Crystal Ball for AI Speed

In this paper, a team of researchers from Georgia State University and the Army Research Laboratory built a "crystal ball" for AI speed. They call it a runtime-aware latency prediction framework. Let's break that down. "Latency" is just a fancy word for how long something takes to happen. "Runtime-aware" means the system pays attention to what's happening while the AI is working, not just what the AI looks like on paper. And "transferable" means this crystal ball works even if you move from one type of device to another.

The researchers realized that trying to test every AI model on every single device is like trying to taste every flavor of ice cream in the world to find your favorite. It's too slow and expensive. Instead, they wanted a system that could look at a new device and a new AI model and say, "Hey, this combo will probably take about 10 seconds to answer."

How the Crystal Ball Works

The secret sauce of their system is that it doesn't just look at the static facts. Imagine you are trying to guess how fast a car will drive. A "static" guess would just look at the car's engine size and weight. But a "runtime-aware" guess would also check the weather, the traffic, and whether the driver is sleepy.

The researchers' system does two things at once:

The Static Path: It looks at the "ID card" of the setup. This includes the model's size, the type of phone or gadget, and the settings.
The Dynamic Path: It watches the "heartbeat" of the device while it runs. It tracks things like how hot the chip gets, how fast the processor is spinning, and how much memory is being used.

The system splits the AI's work into two phases, like a two-part race:

The Prefill Phase: This is when the AI reads your question and gets its thoughts together. It's usually a quick, bursty sprint.
The Decode Phase: This is when the AI writes out the answer, word by word. This is a longer, steady jog.

By treating these two phases separately and mixing the "ID card" info with the "heartbeat" data, the system builds a much smarter prediction. They used a special "gated fusion" mechanism, which is like a smart traffic light that decides whether to listen more to the static facts or the live traffic conditions depending on the situation.

The Magic of Calibration

Here is the most important trick: the system isn't perfect right out of the box. If you train it on a Pixel 8 phone and try to use it on a Pixel 8 Pro, it might get the numbers wrong because the two phones are slightly different.

To fix this, the researchers use a calibration set. Think of this as a quick "test drive." You run the AI on the new device just a few times (maybe 6 to 20 times) to see how it actually behaves. The system then uses these few real results to tweak its crystal ball, aligning its predictions with the new device's reality.

The results of this calibration are dramatic. Without it, the system's predictions for how long the AI takes to "decode" (write the answer) on a Pixel 8 Pro were wildly off, with a score (called $R^2$ ) of -1.085. That's a terrible score, meaning the prediction was worse than just guessing randomly. But after just a tiny bit of calibration, that score jumped to 0.927, which is nearly perfect. It's like taking a blurry photo and suddenly snapping it into crystal clear focus with just a few adjustments.

Testing the System

The team tested their idea on a variety of gadgets to make sure it wasn't just a fluke for one specific phone. They used:

Pixel 8 and Pixel 8 Pro: The main stars of the show, representing modern mobile phones.
Jetson Nano: A tiny computer often used for robots and drones.
Orange Pi 5 Pro: A small, single-board computer.
RTX 3090: A powerful graphics card usually found in high-end gaming PCs.

They found that the same AI model could be incredibly fast on the big gaming card (64.38 tokens per second) but painfully slow on the small Orange Pi (8.42 tokens per second). This proves that you can't just guess speed based on the model alone; you have to know the device.

Why This Matters: The Screening Process

The ultimate goal isn't just to predict speed; it's to save time. Imagine you have 100 different AI models and you need to pick the best one for your specific phone. Without this system, you'd have to download and test all 100. With this system, you can predict the speed of all 100 instantly.

The researchers then use a strategy called Pareto optimization. This is a fancy way of saying they look for the "sweet spot." They want a model that is fast and smart. If Model A is slower than Model B but not smarter, Model A is useless. The system filters out the bad options and leaves you with a short list of the best candidates to test for real.

In their tests, this screening process successfully kept the best possible models in the running. For example, when moving from a Pixel 8 Pro to a Pixel 8, the system filtered out half the candidates but made sure the absolute best one was still there.

What They Found (and What They Didn't)

The paper suggests that this method is a powerful tool for making AI deployment faster and cheaper. They found that:

Static guesses aren't enough: Just knowing the model size doesn't tell you how fast it will run on a specific phone.
Calibration is key: You can't just copy-paste a prediction from one device to another; you need a tiny bit of real data to fix the prediction.
Phase matters: Treating the "reading" part and the "writing" part of the AI differently makes the prediction much more accurate.

However, the authors are careful to note that this is a starting point. Their best results came from the Pixel phones, and while they showed the system could work on other devices like the Jetson and Orange Pi, they didn't do a full, deep-dive training on those yet. They also noted that sometimes, on powerful computers, smaller models don't always run faster than big ones, which is a weird quirk that their system helps explain.

In short, this paper offers a clever, flexible way to guess how fast an AI will run on your gadget without having to wait around for hours to find out. It turns a chaotic guessing game into a calculated, data-driven decision, helping us get the best AI experiences on our devices without burning through our time or battery.

Technical Summary: Transferable Latency Prediction for Fast LLM Screening on Heterogeneous Edge Devices

Problem Statement

Deploying Large Language Models (LLMs) on heterogeneous edge devices (e.g., mobile phones, single-board computers, embedded GPUs) is increasingly critical for low-latency, privacy-sensitive, and connectivity-constrained applications. However, determining which model is suitable for a specific device under specific deployment requirements remains a significant systems bottleneck.

The core challenge is that LLM inference latency is not determined solely by static model architecture or hardware capabilities. Instead, it is a complex function of:

Static Factors: Model architecture, parameter scale, quantization format, and hardware specifications.
Dynamic Factors: Prompt length, generated token behavior, Dynamic Voltage and Frequency Scaling (DVFS), hardware utilization, memory pressure, and thermal throttling.
Execution Phases: LLM inference consists of distinct prefill (processing the prompt and building the KV cache) and decode (autoregressive token generation) phases, which exhibit different latency characteristics and hardware behaviors.

A straightforward solution—exhaustively benchmarking every candidate model on every target device—is prohibitively expensive due to the costs of runtime setup, model conversion, execution, and telemetry collection. Existing latency predictors often rely only on static features, failing to capture platform-specific runtime behaviors, scheduling effects, and backend-dependent execution differences. Consequently, there is a need for a runtime-aware and transferable prediction framework that can screen candidate models before expensive target-device execution.

Methodology

The paper proposes a runtime-aware latency prediction framework designed for deployment-oriented LLM selection across heterogeneous environments. The system operates through four main stages:

1. Configuration Representation

Each inference request is modeled as a configuration tuple $c = (m, d, r, p)$ , where:

$m$ : Pretrained model (family, size, quantization).
$d$ : Device (hardware profile, telemetry capabilities).
$r$ : Runtime backend and platform settings (e.g., MLC-LLM, PyTorch, CPU/GPU/NPU).
$p$ : Prompt/task configuration (input length, max generation length).

2. Dual-Path Predictor Architecture

The framework employs a transferable predictor with two complementary branches fused via a gated mechanism:

Static Path: Encodes stable descriptors (device profile, model descriptor, deployment config) into a static embedding ( $S$ ). This captures expected compatibility before execution.
Dynamic Path: Processes runtime telemetry traces collected during profiling. Crucially, it applies phase-aware modeling:
- Prefill traces are aligned under normalized phase time to reconstruct sparse, bursty hardware states.
- Decode traces preserve temporal order to capture autoregressive behavior.
- A temporal encoder aggregates these into a dynamic embedding ( $D$ ).
Gated Fusion: A lightweight gating network computes adaptive weights ( $w_s, w_d$ ) to fuse $S$ and $D$ . This allows the model to rely more on static information when configurations are stable and more on dynamic data when runtime variations dominate.
Regression Head: Outputs predicted prefill latency, decode latency, and total latency.

3. Transfer and Calibration

To handle cross-device deployment, the system uses a calibration set. A small number of target-device executions are measured to correct device-specific biases. This aligns the source-trained predictor with the target device's latency scale and runtime regime without requiring exhaustive profiling.

4. Pareto-Based Screening

The predicted latency is combined with model accuracy (or task quality) to perform Pareto optimization. The system filters out dominated candidates (those with worse latency and lower quality) to identify a compact set of Pareto-optimal models, balancing responsiveness and output quality.

Key Contributions

Formulation: Defined LLM latency prediction as a hardware-runtime-model-prompt configuration problem, enabling prediction across diverse devices, models, prompts, and backends.
Architecture: Proposed a runtime-aware transferable predictor that integrates static descriptors with dynamic telemetry, separates prefill/decode behaviors via phase-aware modeling, and uses gated fusion for adaptive embedding combination.
Workflow: Designed a deployment-oriented screening workflow that uses calibrated predictions to filter candidates before target-device execution, supporting latency-quality trade-off analysis.
Validation: Built and evaluated a multi-platform profiling pipeline across Pixel mobile devices, Jetson Nano, Orange Pi 5 Pro, and an RTX 3090-class GPU, validating both mobile prediction accuracy and heterogeneous profiling coverage.

Experimental Results

The framework was evaluated on Pixel 8 and Pixel 8 Pro devices, with profiling validation on Jetson Nano, Orange Pi 5 Pro, and RTX 3090-class hardware.

Within-Device Prediction: On Pixel 8, the full predictor improved total-latency $R^2$ from 0.953 to 0.960 and decode-latency $R^2$ from 0.957 to 0.973 compared to a static-only baseline. On Pixel 8 Pro, it significantly improved prefill-latency $R^2$ from -1.383 to 0.966.
Cross-Device Transfer: Calibration proved essential. Without it, transfer $R^2$ values were often negative (e.g., -0.974 for Pixel 8 Pro $\to$ Pixel 8 total latency). With calibration, $R^2$ improved to 0.940.
Heterogeneous Profiling: Results confirmed that latency is highly device- and runtime-dependent. For instance, the SmolLM2-1.7B model achieved 64.38 tokens/s on an RTX 3090-class GPU but only 8.42 tokens/s on an Orange Pi 5 Pro. Furthermore, on the RTX 3090, smaller models were not always faster, indicating that kernel overhead and memory behavior can dominate parameter-count scaling.
Ablation Studies: Confirmed that static descriptors alone are insufficient for cross-device transfer and that calibration is the most critical factor for stabilizing predictions. Phase-aware modeling provided clear benefits for prefill latency prediction.
Screening Efficiency: The predictor successfully reduced the candidate model set (filtering ratios of 0.333 and 0.500 in tested scenarios) while preserving the best feasible models.

Significance and Claims

The paper claims that runtime-aware prediction with lightweight calibration can significantly reduce the cost of profiling and support latency-aware LLM deployment across heterogeneous edge platforms.

Practical Impact: The framework addresses the "systems bottleneck" of model selection by replacing exhaustive benchmarking with a predictive screening process.
Necessity of Dynamics: The results demonstrate that static models alone cannot capture the complex interplay of DVFS, thermal throttling, and runtime scheduling that dictates edge device performance.
Transferability: The system demonstrates that a predictor trained on source devices can generalize to target devices effectively, provided a small calibration set is used to align the latency scales.
Deployment Loop: The work frames latency prediction not as an isolated regression task but as a component of a closed-loop deployment system that iteratively improves decisions with limited measurements.

The authors note limitations, including that the most complete prediction results are based on the Pixel mobile dataset, while other platforms served primarily for profiling validation. Future work is identified as extending the framework to jointly consider throughput, memory, and energy alongside latency.