Technical Summary: Dooly

Problem Statement

Deploying Large Language Model (LLM) inference systems requires navigating a vast configuration space comprising hardware, serving engines, attention backends, model architectures, and workload characteristics. No single configuration is optimal across all workloads; performance rankings often invert based on sequence length or model type, making pre-deployment evaluation essential.

Current profile-based simulators, the standard tool for this evaluation, face two critical limitations that hinder efficient exploration:

1. Static Configuration Dependencies: Existing simulators hardcode the set of operations and input arguments for specific configurations. Any change in the software stack (e.g., a new serving engine or attention backend) or model architecture requires manual rework to restore coverage.
2. High Profiling Overhead: Simulators fail to recognize when different configurations invoke semantically identical computations. Consequently, they re-profile every operation from scratch for each new model or backend. For example, profiling four similar-sized models with a single backend can take nearly 28 wall-clock hours on a single A100 GPU, scaling linearly with the number of models.

These limitations stem from a lack of structural understanding of LLM inference: specifically, that input dimensions are derived from either fixed model configurations or variable request characteristics, and that many model-configuration values (e.g., head size, layer count) recur across different models, leading to significant redundancy in profiling.

Methodology

The authors propose Dooly, a framework for configuration-agnostic, redundancy-aware profiling. Dooly exploits the structural recurrence of operations across configurations to eliminate redundant profiling runs. The system operates through three core components:

1. Tainted Runner (Semantic Annotation)

To distinguish between fixed model parameters and variable request inputs, Dooly employs taint propagation.

• Source Tainting: The system intercepts the serving engine's entry points (e.g., ModelConfig and request processing logic) to label values. Model configuration parameters are tagged with MODEL_CONFIG, while request-dependent values (token counts, batch sizes) are tagged with NUM_TOKS and NUM_REQS.
• Propagation: As the inference pass executes, these taints propagate through arithmetic and tensor operations. Rules are defined for how taints combine (e.g., multiplying a batch size by a head dimension results in a composite MIX taint).
• Trace Annotation: The resulting trace annotates every tensor dimension with its origin, enabling the system to identify which dimensions are invariant across workloads and which are variable.

2. Operation Set Finder (Granularity Resolution)

This component parses the tainted trace into a hierarchical call graph to determine the finest granularity at which operations can be profiled in isolation.

• Hierarchy Construction: The trace is pruned to collapse structurally identical subtrees (e.g., repeated transformer layers) into canonical representatives.
• Runnability Testing: The system attempts to execute leaf operations (individual kernels) in isolation. If an operation fails due to missing runtime state (e.g., KV cache memory or attention metadata), the system falls back to its parent module.
• Context Resolution: For stateful operations that cannot run standalone, Dooly reuses the serving engine's own initialization code to reconstruct the necessary runtime context, eliminating the need for manual instrumentation.

3. Duplication-Aware Profiler (Signature Matching)

This component eliminates redundant profiling by identifying semantically equivalent operations across different configurations.

• Signature Generation: An operation is uniquely identified by a signature hash comprising:
  1. The operation name and its MODEL_CONFIG-tainted dimensions (fixed by architecture).
  2. The set of GPU kernel symbols dispatched (capturing backend-specific kernel selection).
  3. A digest of module-level attributes (e.g., sliding window size) for runtime branching.
     Note: Workload-dependent dimensions (NUM_TOKS, NUM_REQS) are excluded from the signature as they parameterize the sweep, not the operation's identity.
• Selective Profiling: Before dispatching a measurement, the profiler checks the latency database. If a signature already exists, the operation is skipped. Only unique signatures are profiled across user-defined ranges of sequence lengths and batch sizes.
• Latency Database: Results are stored in a SQLite database keyed by signature hash. This database serves as a drop-in backend for existing simulators, which train per-signature regression models to predict latency.

Key Contributions

• Configuration-Agnostic Profiling: Dooly dynamically extracts operations and profiles them without hardcoding templates, allowing it to support new models, backends, and serving engines without manual rework.
• Redundancy Elimination: By using taint propagation to distinguish model-derived dimensions from request-derived ones, Dooly detects equivalent operations across different models and configurations, reducing profiling effort.
• Stateful Operation Isolation: Through hierarchical context resolution, Dooly isolates complex, state-dependent operations (like attention kernels) by reusing the serving engine's internal initialization logic, avoiding the need for manual context reconstruction.
• Drop-in Integration: The resulting latency database trains regression models that existing simulators can adopt as a backend to perform end-to-end simulation.

Results

The authors evaluated Dooly across two GPU platforms (NVIDIA A100 and H100), three attention backends (FlashInfer, FlashAttention, TritonAttention), and 12 diverse model architectures (including dense, MoE, and heterogeneous families).

• Simulation Accuracy: When integrated into a simulator (DoolySim), the system predicts Time to First Token (TTFT) with a Mean Absolute Percentage Error (MAPE) within 5% and Time Per Output Token (TPOT) within 8% compared to live measurements. It also reproduces batch scheduling decisions with a MAPE below 0.5%.
• Profiling Overhead Reduction: Across 12 models profiled with three attention backends (36 total configurations), Dooly reduced total profiling GPU-hours by 56.4% compared to a naive, non-deduplicating approach.
  • Specifically, profiling Llama-3.1-8B and Command-R7B took 5.2 GPU-hours with Dooly versus 8.1 hours naively.
  • The system identified only 100 unique signatures out of 612 total operator occurrences, with attention signatures accounting for 95% of the time saved.
• Scalability: Once a few models exercise the distinct attention configurations in a corpus, subsequent models add negligible overhead (under 0.2 GPU-hours) as they inherit existing signatures.

Significance and Claims

The paper claims that Dooly fundamentally changes the economics of LLM inference simulation by shifting from a "profile-everything" approach to a "profile-what-is-unique" approach. By leveraging the structural recurrence of model configurations, Dooly enables researchers and practitioners to evaluate serving strategies across a wide range of configurations with significantly reduced cost.

The authors emphasize that their approach is modest regarding broader impacts, stating the work has no significant ethical or societal impact beyond its technical utility. They acknowledge limitations, such as the inability to trace operations that bypass PyTorch's dispatcher (though enclosing modules remain covered) and the assumption of random expert routing for MoE models, noting that data-dependent routing modeling is outside the scope of the paper. The primary contribution is a practical framework that makes cross-configuration exploration feasible and efficient without requiring manual engineering for every new stack update.