CUDABench: Benchmarking LLMs for Text-to-CUDA Generation

This paper introduces CUDABench, a comprehensive benchmark designed to evaluate Large Language Models' text-to-CUDA generation capabilities across diverse domains by assessing compilation correctness, functional consistency, and performance via a novel roofline-based metric.

The Gist

Imagine you have a brilliant, hyper-intelligent robot assistant (an LLM) that can write code. You ask it to write a recipe for a complex dish, and it does a great job. But now, you ask it to write a recipe specifically for a high-performance racing engine that needs to run on a specific, incredibly fast, and temperamental machine (a GPU).

This is the challenge the paper CUDABench tackles.

Here is the breakdown of the paper using simple analogies:

1. The Problem: The "Chef" vs. The "Race Car Mechanic"

Previously, researchers tested these AI "chefs" by asking them to translate a recipe written in one language (like Python) into another (CUDA, the language for GPUs). It's like asking a chef to translate a French menu into English. They just have to swap the words; the cooking instructions are already there.

But in the real world, you often just say, "Make me a fast engine for this specific car," without giving them the blueprint. This is Text-to-CUDA. The AI has to invent the engine from scratch based only on a text description.

The paper argues that previous tests were too easy because they didn't test if the AI could actually design the engine, only if it could translate the instructions. Also, just because an engine starts doesn't mean it's fast. A slow engine is useless for a race car.

2. The Solution: CUDABench (The Ultimate Driving Test)

The authors built a new, massive testing ground called CUDABench. Think of it as a driving school with three specific types of tests:

• Breadth (The Variety of Roads): They didn't just test on a straight highway. They tested the AI on six different "terrains":
  • Linear Algebra (Doing math puzzles).
  • Deep Learning (Teaching the car to recognize cats).
  • Computer Vision (Seeing the road).
  • Data Analytics (Counting millions of cars in traffic).
  • Signal Processing (Listening to radio waves).
  • Science & Finance (Predicting weather or stock prices).
• Depth (The Traffic Density): They tested the AI with tiny amounts of data (a single car) and massive amounts (a traffic jam of millions). The AI needs to handle both a quiet street and a gridlock.
• Difficulty (The Instructions):
  • Level 1 (The Guided Tour): "Build an engine. Here is the blueprint, the tools, and the exact steps."
  • Level 2 (The Schematic): "Build an engine. Here is the blueprint, but you figure out the tools and steps."
  • Level 3 (The Whisper): "Build an engine. That's it. Good luck." (The AI has to remember everything from its training).

3. The Scorecard: The "Roofline" Metric

How do you know if the AI's engine is good?

• Old Way: "How fast did it finish the lap?" (Execution Time).
  • Problem: If you test on a Ferrari vs. a Toyota, the Ferrari wins even if the driver is bad. It's unfair.
• New Way (CUDABench-Score): They use a Roofline Model. Imagine a ceiling (the roof) representing the absolute maximum speed that specific car could possibly go.
  • If the AI's engine hits 90% of the ceiling, it's a genius.
  • If it only hits 10% of the ceiling, it's a disaster, even if it technically "works."
  • This score is hardware-independent. It tells you how well the AI utilized the car's potential, regardless of whether the car is a Ferrari or a Toyota.

4. The Results: The AI is a "Good Talker, Bad Builder"

The authors tested the smartest AI models available (like GPT-5, Claude, etc.) and found some surprising things:

• The "Syntax" Trap: The AIs are amazing at grammar. They can write code that looks perfect and compiles (starts up) 99% of the time. It's like a mechanic who can assemble a car perfectly, but the engine doesn't actually run.
• The Logic Gap: Once the code starts running, the AIs fail often. They get the words right but the logic wrong. They forget to synchronize the workers (threads) or manage the memory correctly.
• The Knowledge Hole: When asked to do complex, niche tasks (like financial modeling or specific scientific simulations) without hints (Level 3), the AIs crumble. They know general math, but they don't know the specific "tricks of the trade" for high-performance GPU programming.
• The Speed Issue: Even when the code works, it's slow. The AIs are leaving about 60% of the GPU's power on the table. They aren't using the car's full potential.

The Bottom Line

CUDABench is a wake-up call. It shows that while AI is great at writing code that looks right, it still struggles to write code that is fast, efficient, and truly optimized for the complex hardware of the future.

It's like having a robot that can write a beautiful poem about a race car, but when you ask it to build the race car, it builds something that sputters and barely moves. We need to teach these robots not just to speak the language of engineers, but to think like them.

Technical Summary

1. Problem Statement

While Large Language Models (LLMs) have shown promise in general code generation, their ability to generate high-performance CUDA kernels from natural language descriptions (Text-to-CUDA) remains under-evaluated. Existing benchmarks suffer from three main limitations:

• Scope: Most focus on translating high-level languages (e.g., PyTorch) to CUDA (Code-to-Code), rather than generating kernels directly from natural language specifications (Text-to-CUDA).
• Domain Coverage: They are often restricted to Machine Learning workloads, ignoring critical domains like scientific computing, finance, and signal processing.
• Evaluation Metrics: Current metrics rely heavily on execution time, which is hardware-dependent and fails to capture the theoretical efficiency of a kernel. A kernel may produce correct output but perform poorly due to suboptimal memory or compute utilization.

The paper argues that accurately assessing LLMs for GPU programming requires a benchmark that addresses breadth (diverse domains), depth (input scale), and difficulty (prompt complexity), along with a hardware-independent performance metric.

2. Methodology: CUDABench Framework

The authors propose CUDABench, a comprehensive benchmarking framework consisting of three core components:

A. CUDABench-Set (The Dataset)

A dataset of 1,500 prompts constructed from 500 unique tasks, organized into a 3D Evaluation Space:

1. Breadth (Domains): Covers six diverse domains:
   • Fundamental Linear Algebra (e.g., GEMM).
   • Deep Learning Operators (e.g., Loss functions, Optimizers).
   • Computer Vision & Image Processing (2D/3D).
   • Data Analytics (Sorting, Reduction).
   • Signal Processing (FIR, Wavelets).
   • Scientific Simulation & Finance (Monte Carlo, PDEs).
2. Depth (Input Scale): Tasks are tested across five input scales (Tiny to Huge, ranging from 1 KB to >1 GB) to stress-test memory bandwidth and compute units. Each scale has dedicated reference kernels and validators.
3. Difficulty (Prompt Complexity):
   • Level 1 (Guided): Includes task name, algorithm description, and specific CUDA implementation hints (memory hierarchy, thread mapping).
   • Level 2 (Algorithmic): Includes task name and algorithm, but no hardware-specific guidance.
   • Level 3 (Concept Retrieval): Zero-shot setting with only the task name; requires the LLM to retrieve domain knowledge and implementation details internally.

B. Generative Verification Pipeline

An automated end-to-end pipeline to validate LLM outputs:

1. Data Generation: Creates randomized input data and reference outputs.
2. Compilation: Checks if the generated code compiles using NVCC.
3. Functional Verification: Executes the kernel and compares output against the reference (tolerance-based).
4. Performance Profiling: Uses NVIDIA Nsight Compute to measure execution time, FLOPs, and data movement.

C. CUDABench-Score (The Metric)

To overcome hardware dependency, the authors introduce a Roofline-based metric:

• Performance-Score: Calculates the ratio of the kernel's Achieved GFLOPs/sec to its Attainable GFLOPs/sec (the theoretical limit defined by the Roofline model).
  • For Memory-Bound kernels, this effectively measures Memory Bandwidth Utilization.
  • For Compute-Bound kernels, it measures Compute Utilization.
• CUDABench-Score: A unified scalar metric combining compilation success, functional correctness, and the Performance-Score.

3. Key Contributions

1. CUDABench-Set: The first text-to-CUDA benchmark covering a Breadth-Depth-Difficulty space across non-ML domains (Science, Finance, etc.) with production-scale input sizes.
2. Generative Verification Pipeline: A robust framework ensuring both functional correctness and hardware-aware performance evaluation.
3. CUDABench-Score: A novel, hardware-independent metric that normalizes performance across different GPU architectures (e.g., A40 vs. RTX 4090) by utilizing the Roofline model.
4. Comprehensive Evaluation: Extensive testing of 7 state-of-the-art LLMs (including GPT-5, Claude 4.5, Gemini 3, DeepSeek, Qwen3, Kimi, MiniMax).

4. Experimental Results & Findings

The authors evaluated top-tier LLMs on the A40 and RTX 4090 GPUs. Key findings include:

• Finding 1: Significant Challenge for LLMs.
  • Performance drops sharply as difficulty increases. For example, GPT-5.2's functional accuracy drops from 79.8% (Level 1) to 60.8% (Level 3). This contrasts with general code benchmarks (e.g., HumanEval) where models perform much better, indicating CUDA generation requires specialized knowledge.
• Finding 2: High Compilation, Low Functionality.
  • There is a massive gap between syntactic validity and semantic correctness. Models like Claude 4.5 achieve near-perfect compilation rates (~99-100%) but struggle with functional logic.
  • Functional Inconsistency is the primary failure mode (e.g., 47% of Level 3 generations for DeepSeek-V3.2 failed functionally, while compilation errors were only ~3.6%). Models often fail at thread synchronization and memory boundary conditions.
• Finding 3: Lack of Domain Knowledge.
  • In Level 3 (Zero-shot), performance collapses, especially in niche domains like "Scientific Simulation & Finance" (failure rates up to 85% for some models). This suggests LLMs lack intrinsic expertise in complex, domain-specific algorithms.
• Finding 4: Suboptimal Hardware Utilization.
  • Even the best-performing models achieve a Performance-Score of only ~40% on Level 1. This implies that ~60% of GPU resources (compute or memory bandwidth) are wasted.
  • The performance gap between models is negligible, suggesting a systemic inability to implement hardware-aware optimizations (e.g., tiling, shared memory usage).
• Hardware Independence:
  • The CUDABench-Score remained stable across different GPUs (A40 vs. RTX 4090), confirming its effectiveness as a hardware-agnostic metric.

5. Significance

• Bridging the Gap: CUDABench moves beyond simple code translation to evaluate the true capability of LLMs in generating optimized, domain-specific GPU code.
• Realistic Evaluation: By introducing the Roofline-based metric, the paper shifts the focus from "does it run?" to "how efficiently does it run?", which is critical for high-performance computing.
• Research Direction: The results highlight that current LLMs, despite strong general coding abilities, lack the specific parallel programming intuition and domain knowledge required for CUDA. This sets a clear agenda for future research: improving domain-specific training, enhancing reasoning for hardware constraints, and developing better prompting strategies for GPU optimization.
• Open Resource: The benchmark is open-sourced, providing a standardized foundation for the community to advance LLM-driven GPU programming.