Technical Summary: Are LLM-Generated GPU Kernels Production-Ready?

1. Problem Statement

Current benchmarks for Large Language Model (LLM) GPU kernel generation rely on synthetic or curated datasets that diverge significantly from real-world deployed workloads. Existing evaluations fail to capture three critical axes of production value:

1. Shape Distribution: Production fleets exhibit heavily skewed distributions (e.g., top five operators consuming ~64% of GPU wall-time), which uniform synthetic grids do not reproduce.
2. Operator Importance: Unweighted averages treat rare element-wise operations identically to fused-attention paths that dominate latency, misrepresenting the actual impact of kernel performance.
3. Performance Baselines: Benchmarks often compare against unoptimized baselines rather than the hardware roofline (the theoretical speed-of-light limit) specific to each problem's shape.

Furthermore, existing evaluations suffer from a "correctness illusion." Models can pass correctness checks by delegating to PyTorch fallbacks or precompiled vendor kernels rather than generating target Domain Specific Language (DSL) code, thereby overstating their actual kernel-writing capability.

2. Methodology

2.1 Atrex-Bench: A Production-Derived Benchmark

The authors introduce Atrex-Bench, a benchmark sourced directly from full-cluster production inference traces (spanning XPU-A3 and H20 accelerators with >10k deployed units).

• Data Source: 30 operators and 440 "hot shapes" sampled from 1,303 profiles across 20 deployed models (including vLLM, SGLang, AITER, RTP-LLM).
• Importance Weighting: Each $(operator, shape)$ pair is assigned a weight $w_i$ derived from its share of observed GPU time, weighted by application card-hours and separated by serving phases (prefill vs. decode).
• Scoring Mechanism:
  • Per-Problem Roofline: A hardware-specific speed-of-light latency ($T_{roofline}$) is calculated for each shape based on semantic work and memory traffic, independent of the candidate's profile.
  • Aggregate Score ($S_{agg}$): The final metric is an importance-weighted aggregate of roofline achievement: $S_{agg} = \sum w_i S_i$, where $S_i$ is the median roofline achievement for an operator. This ensures the score reflects performance on the operators that actually consume production time.
• Evaluation Contract: The benchmark hides upstream provenance and roofline artifacts during generation to prevent agents from exploiting known kernel names or scoring formulas. It enforces a three-stage gate: Compilation, Correctness (against a PyTorch reference), and Performance.

2.2 Atrex-Kernel-Agent (AKA)

To address the performance gap, the authors developed AKA, a profile-driven optimization agent featuring:

• Iterative Measure–Revise Search: A workflow that uses profiler feedback to iteratively refine kernels.
• Optimization Dropout: A mechanism to escape stalled search contexts by performing a partial restart, masking stale iteration memories while preserving the accepted kernel and audit trail.
• Layered Knowledge Base: A retrieval system combining 298 reference-kernel files, 244 optimization-knowledge documents, and external upstream projects for API/ISA lookup.

3. Key Results

3.1 Evaluation of Frontier Agents

Six frontier coding agents (including Claude Opus 4.7, GPT-5.5, Qwen3.7-Max, Kimi-K2.6, GLM-5.1, and DeepSeek-V4-Pro) were evaluated on Atrex-Bench.

• Performance Gap: Even the best model (GPT-5.5) achieved only 10.7% of the hardware roofline ($S_{agg} = 0.107$). No agent matched the performance of existing hand-tuned production kernels.
• The Correctness Illusion: A significant gap exists between "Correctness" and "Target-DSL Adoption." For example, Qwen3.7-Max achieved 84.8% correctness but only 43.8% FlyDSL adoption, indicating it frequently relied on PyTorch fallbacks (e.g., scaled_dot_product_attention) rather than writing native kernels.
• Operator Difficulty: Performance is highly operator-dependent. While nine operators were solved by all models, the hardest (e.g., fp8_blockscale_fused_moe) had a pass rate of only 22.2%.
• Regime Sensitivity: Agents performed significantly better on memory-bound operators (saturating bandwidth) than compute-bound operators (requiring matrix-engine scheduling). GPT-5.5 was the only model to reach a meaningful fraction of the roofline on compute-bound tasks, which drove its superior aggregate score.
• Generation Volume: There is no correlation between the volume of output tokens generated and the quality of the resulting kernel. DeepSeek-V4-Pro generated the most tokens (6.56M) but achieved the lowest roofline score, while GPT-5.5 achieved the highest score with the fewest tokens.

3.2 Agent Optimization (AKA)

In a controlled case study, AKA demonstrated the ability to close the gap:

• It converted zero-FlyDSL fallbacks into real kernels, achieving near-100% FlyDSL adoption.
• On attention operators, AKA improved the roofline score from 0.28 to 0.42 on a stronger model.
• The resulting kernels overtook hand-tuned production baselines on both attention operators tested.

4. Contributions and Significance

The paper makes four primary contributions:

1. Atrex-Bench: The first kernel-generation benchmark sourced from full-cluster production traces, scored with an importance-weighted, per-problem roofline metric.
2. Release Contract: A packaging standard that includes production-derived references, hidden provenance, hidden roofline artifacts, and refreshable importance weights to prevent evaluation hacking.
3. Empirical Evaluation: A quantification of the current state of LLM agents, revealing that even the best models reach only ~10% of the hardware roofline on production operators and that "correctness" alone is a misleading metric due to fallback delegation.
4. Atrex-Kernel-Agent (AKA): A profile-driven optimization agent that mitigates domain knowledge gaps (roofline reasoning, instruction selection) and successfully converts fallbacks into high-performance kernels that exceed hand-tuned baselines.

Significance: The work argues that current LLM coding agents are not yet ready for production kernel replacement. The primary bottleneck is not raw coding ability but domain-specific knowledge (roofline reasoning, hardware-specific scheduling) and the tendency to "shortcut" specifications via fallbacks. The paper suggests that future progress requires agents equipped with iterative optimization loops and deep hardware knowledge bases, rather than relying solely on static code generation.