🤖 AI

Think Before You Grid-Search: Floor-First Triage for LLM Serving

This paper proposes "Floor-First Triage," a compositional, estimation-driven workflow that models LLM decoding as a five-dimensional resource vector to analytically determine performance bounds and identify binding constraints before resorting to heavy profiling or grid search, thereby enabling computable layout decisions for diverse operating points.

Original authors: Yihua Liu

Published 2026-07-08✓ Author reviewed ⓘ

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yihua Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Guessing vs. Knowing

Imagine you are running a massive, high-speed restaurant (a Large Language Model, or LLM) that serves millions of customers. You want to serve food as fast as possible without the kitchen exploding.

Currently, when the kitchen gets slow, most teams panic and try everything. They change the number of chefs, the size of the tables, the type of ovens, and the recipe. They run hundreds of tests, measure the results, and hope one combination works. This is called "grid-searching." It's expensive, wastes time, and often misses the real problem.

This paper argues: Stop guessing. Start calculating.

The Core Idea: Build the "Floor" First

The authors propose a new workflow called "Floor First."

Imagine the restaurant has a concrete floor. No matter how you arrange the furniture, the floor is the absolute lowest point the furniture can go. In the world of computer chips, this "floor" is the theoretical minimum time it takes to do a task based on physics (how fast electricity moves, how much data fits in memory, etc.).

The Workflow:

Calculate the Floor: Before you touch a single knob or run a test, you do a simple math calculation to find the "speed limit" of your hardware.
Measure the Reality: You run your system and see how fast it actually is.
Check the Gap:
- Small Gap: If your actual speed is very close to the theoretical floor, you are doing great. Stop. Don't waste time profiling. The hardware is already working as hard as it physically can.
- Big Gap: If your actual speed is much slower than the floor, then you open the "profiler" (the fancy diagnostic tool) to find out why. Is the chef dropping ingredients? Is the door stuck?

The Analogy:
Think of it like a car. If your car is doing 60 mph on a road with a 60 mph speed limit, you don't need a mechanic to tell you the engine is fine. You just know you're at the limit. But if you're doing 20 mph, then you need to check the engine. This paper gives you the speed limit sign so you know when to stop checking.

The "Five-Dimensional" Scorecard

To calculate this floor, the authors break the problem down into five simple buckets of resources, like a shopping list for a trip:

Memory Traffic: How much data has to move? (Like how many suitcases you need to carry).
Computing Power: How much math needs to be done? (Like how many miles you need to drive).
Network Traffic: How much data is sent between computers? (Like how many phone calls you make).
Network Messages: How many times do you have to say "Hello"? (Like the time it takes to start a call).
Storage Capacity: How much space do you have for the "memory" of the conversation? (Like the size of your trunk).

By adding up the time it takes to fill these buckets, you get a "Floor." The paper introduces a clever trick: it calculates an Optimistic Floor (assuming everything happens perfectly at the same time) and a Pessimistic Floor (assuming everything happens one by one). If your real-world speed falls between these two numbers, you know exactly how well your system is overlapping tasks.

The Case Study: The "H20" Chip

The paper tests this idea on a specific, tricky computer chip called the NVIDIA H20.

The Situation: This chip is like a truck with a huge cargo hold (memory) but a weak engine (computing power).
The Conflict: Two different teams built restaurants using these trucks.
- Team A arranged the kitchen so the chefs (processors) worked together in one big group.
- Team B arranged it so the chefs worked in smaller, separate groups.
- They argued about which was better, relying on "folklore" and trial-and-error.

The Paper's Verdict:
Using the "Floor First" math, the authors showed that the answer depends entirely on how many customers are waiting.

Few Customers: Team A's layout is faster.
Many Customers: Team B's layout is faster because it handles the "trunk space" (memory capacity) better, even if the engine is slightly slower.

The math proved that both teams were right for their specific situation. You don't need to guess; you just calculate the "wall" (the limit) where your specific number of customers hits the ceiling.

The "Agent" Skill

The paper also mentions that this logic can be taught to AI coding agents. Instead of an AI agent blindly running tests and wasting money, it can be programmed to:

Do the Math First: Calculate the floor.
Ask Permission: "My math says this test is a waste of time. Can I skip it?"
Only Profiling When Needed: "My math says there's a huge gap. I need to open the diagnostic tool now."

Summary

This paper is a call to stop "brute-forcing" optimization.

Old Way: Try everything, measure everything, hope for the best.
New Way (Floor First): Do the math to find the speed limit. If you are near the limit, stop. If you are far from it, find the leak.

It turns a chaotic, expensive guessing game into a clean, logical process where you know exactly when to stop working and when to dig deeper.

Technical Summary: Floor-First Triage for LLM Serving

Problem Statement
Production teams optimizing Large Language Model (LLM) serving face a vast configuration space involving parallelism layouts (TP, EP, DP), batching, quantization, and kernel-level work. The prevailing operational response to missed latency targets is to benchmark numerous configurations and deploy heavy profilers (e.g., Nsight Systems) indiscriminately. This approach often degenerates into inefficient grid search, where measured performance numbers are ambiguous: a slow Time Per Output Token (TPOT) could stem from fundamental bandwidth limits, implementation gaps, exposed communication, host overhead, or underperforming kernels. The paper argues that the engineering community lacks a systematic decision procedure to determine when profiling is warranted and where to look first, leading to unpredictable deployment choices (e.g., conflicting parallelism strategies for DeepSeek models on identical hardware) based on folklore rather than explicit resource accounting.

Methodology: Floor-First Triage
The paper proposes "Floor First," a residual-driven triage workflow that treats analytical estimation as the primary layer of profiling. The methodology consists of four distinct stages:

Analytical Floor Construction: Before any benchmarking, the system computes a "two-sided floor" for a given configuration. This model treats each decode step as a five-dimensional resource vector: HBM bytes, FLOPs, network bytes, network messages, and KV capacity.
- Optimistic Floor ( $t_{opt}$ ): Calculated as the maximum of independent resource times ( $max(t_{HBM}, t_{compute}, t_{network})$ ), assuming perfect overlap.
- Pessimistic Floor ( $t_{sum}$ ): Calculated as the sum of all resource times ( $t_{HBM} + t_{compute} + t_{network}$ ), assuming no overlap.
- Diagnostic Interval: The position of a measured benchmark within the $[t_{opt}, t_{sum}]$ interval serves as a zero-cost diagnostic for system overlap. If the measurement is near $t_{opt}$ , overlap is good; if near $t_{sum}$ , scheduling/overlap is the bottleneck; if above $t_{sum}$ , time is leaking to unmodeled costs (host gaps, stragglers).
Benchmark Discipline: Benchmarks must be conducted under strict rules: open-loop load generation (to avoid self-throttling bias), realistic length distributions, steady-state operation, and tail percentile (P99) reporting.
Reconciliation and Triage: The measured time is compared against the floor.
- Small Residual (< 1.3× floor): The investigation stops. The system is near the theoretical limit for that axis; further optimization requires changing the architecture (e.g., sparse attention, quantization) rather than tuning kernels.
- Large Residual: Profiling is escalated. Nsight Systems is used only to answer three specific questions: Are there gaps between kernels? Is communication exposed? Which kernel class exceeds its analytical budget? Nsight Compute is reserved strictly for kernel classes that exceed their budget after other causes are excluded.
Wall Ordering: Instead of comparing single-batch latency, deployments are evaluated by "wall ordering"—determining which resource constraint (KV capacity, KV bandwidth, weight bandwidth, or compute) binds first as load increases. Capacity is treated as a first-class resource dimension, not a post-hoc memory check.

Key Contributions

Residual-Driven Triage Workflow: A decision loop that separates target definition, floor construction, and profiler escalation, with explicit stopping rules that dispatch optimization efforts to either pipeline work (layout/scheduling) or kernel work.
Two-Sided Floor Model: An analytical model that provides both optimistic and pessimistic bounds, using the gap between them to diagnose overlap efficiency without running a profiler.
Wall Ordering as Output: A method to compare deployment alternatives based on which resource wall binds first as concurrency grows, revealing that a layout with worse single-batch latency can dominate the goodput frontier if it has a higher capacity wall.
Metric Discipline: A systematization of utilization metrics (MFU vs. MBU) based on the "ridge identity" ( $MFU/MBU = I_{work}/I_{ridge}$ ), ensuring the correct metric is reported for the binding resource.
Artifact and Agent Skill: A zero-dependency Python artifact (floor.py) and an agent-readable skill (SKILL.md) that enforce this discipline in agentic optimization loops, preventing agents from reflexively reaching for profilers.

Case Study Results
The paper applies this methodology to a DeepSeek-V3.2-style 671B MoE model on 16 NVIDIA H20 GPUs.

Hardware Characterization: The H20 GPU is identified as having a ridge point of ~74 FLOP/byte (vs. ~590 for H100), making it an extreme "decode-oriented" part where the memory-bound region is 8× wider.
Layout Judgment:
- TP16: At batch size 64 and 8K context, TP16 is limited by KV capacity to ~70 concurrent requests. The KV bandwidth term is significant, and the compute wall is unreachable within the feasible region.
- EP16 + DP Attention: This layout trades slightly worse weight traffic for a capacity wall of ~644 requests (an order of magnitude higher). While TP16 has lower single-stream latency (2.4× faster at B=1), EP16+DP dominates the high-concurrency goodput frontier.
- Sparse Attention (DSA): Removes the KV-bandwidth wall but not the capacity wall, as the full KV must remain addressable.
Resolution of Disagreement: The study explains why different teams deployed opposite layouts (TP vs. DP) on the same hardware: the optimal choice is a computable function of the operating point (concurrency). High concurrency favors DP; low concurrency favors TP.

Significance and Claims
The paper claims that production disagreements in LLM serving are often predictable from a small resource account (five numbers per GPU plus model dimensions) rather than requiring extensive experimentation. By making this account systematic, the workflow provides a general triage procedure that:

Prevents wasted profiling effort when measurements are close to the analytical floor.
Identifies the correct optimization axis (pipeline vs. kernel) before expensive tools are engaged.
Explains deployment choices through "wall ordering" rather than point benchmarks.
Offers a robust method for hardware with extreme characteristics (like the H20) where standard heuristics fail.

The authors modestly note that the current draft relies on analytical floors and calibrated constants, with a planned measurement campaign (Section 8) required to validate the residual thresholds and escalation rules empirically. The method is presented as a decision procedure for engineering, not a replacement for simulators or profilers, which remain essential for residuals that survive the floor check.