Hunting CUDA Bugs at Scale with cuFuzz
This paper presents cuFuzz, the first practical CUDA-oriented fuzzer that overcomes key obstacles like false positives and tool incompatibility through whole-program fuzzing, device-side coverage instrumentation via NVBit, and decoupled sanitization, successfully discovering 43 previously unknown bugs across 14 CUDA programs.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have a massive, high-speed factory. On one side, you have the Manager's Office (the CPU), where the plans are drawn up and resources are allocated. On the other side, you have the Assembly Line (the GPU), where thousands of robots work simultaneously to build things at incredible speeds.
For years, people thought the Assembly Line was so simple and well-organized that it rarely made mistakes. But as the factory grew more complex, the robots started making subtle, dangerous errors: grabbing the wrong parts, stepping on each other's toes, or reading instructions that hadn't been written yet. These errors were hard to catch because they only happened under very specific, chaotic conditions.
This paper introduces cuFuzz, a new "safety inspector" designed specifically to hunt down these hidden bugs in the GPU factory.
Here is how it works, broken down into simple concepts:
1. The Problem: Why Old Inspectors Failed
Before cuFuzz, safety inspectors tried to check the robots in two ways, but both had major flaws:
The "Isolated Robot" Check (Kernel-Level Fuzzing):
Imagine an inspector grabbing a single robot off the line, giving it random parts, and seeing if it breaks.- The Flaw: In the real factory, the Manager (CPU) checks the parts before handing them to the robot. If the parts are bad, the Manager stops the process. But the isolated inspector doesn't know this! It creates "impossible" scenarios (like giving a robot a part that doesn't exist) and screams, "ERROR!" when the robot crashes. These are False Alarms.
- The Missed Danger: Conversely, some bugs only happen because the Manager made a mistake before handing the parts to the robot. The isolated inspector never sees the Manager, so it misses these bugs entirely.
The "Blind Spot" Problem (Lack of Device Coverage):
The Manager's office has a map showing which rooms are visited. But the Assembly Line is a giant, dark warehouse with its own secret paths. The old inspectors could only see the Manager's map. They had no idea which specific paths the robots were taking on the line. Without this map, the inspector couldn't guide the robots to try new, dangerous combinations.The "Tool Clash" Problem:
To find bugs, you need two types of tools: a Map Maker (to track which paths are taken) and a Safety Scanner (to detect crashes or memory leaks).- The Flaw: In the GPU world, these two tools hate each other. If you turn on the Safety Scanner, the Map Maker breaks, and vice versa. So, inspectors had to choose: either map the factory (but miss the bugs) or scan for bugs (but have no idea where to look).
2. The Solution: Enter cuFuzz
cuFuzz is the first inspector that solves all three problems at once. Think of it as a Super-Inspector with a special toolkit.
It Checks the Whole Factory (Whole-Program Fuzzing):
Instead of pulling robots off the line, cuFuzz watches the entire process: the Manager planning, the robots building, and the hand-off between them.- The Benefit: It knows that if the Manager says "No," the robot never gets the bad part. This stops the False Alarms. It also catches bugs where the Manager's bad planning causes the robot to crash later.
It Lights Up the Dark Warehouse (Device-Side Coverage):
cuFuzz uses a special tool called NVBit (like a swarm of tiny, invisible drones) that flies over the Assembly Line in real-time.- The Benefit: It maps exactly which paths the robots take on the GPU. It combines this map with the Manager's map, giving the inspector a complete picture of the factory. Now, it can guide the robots to try the specific, tricky paths that are most likely to break.
It Separates the Tools (Decoupled Sanitization):
Since the Map Maker and Safety Scanner can't work in the same room, cuFuzz builds a parallel processing system.- The Benefit: It runs the Map Maker on the main line to keep things moving fast. Meanwhile, it sends the most interesting, tricky inputs to a separate "Safety Room" where the Safety Scanners can work without interfering. This way, it gets the best of both worlds: speed and deep error detection.
3. The Results: A Safety Revolution
The team tested cuFuzz on 14 different "factories" (software programs), including some used by major companies for image processing and AI.
- The Scorecard: cuFuzz found 43 brand-new bugs that no one knew about.
- The Impact: 19 of these were in commercial libraries used by real businesses. Some were memory leaks (like a robot hoarding parts until the warehouse runs out), others were data races (robots stepping on each other), and some were crashes.
- The Efficiency: By using a "Persistent Mode" (keeping the factory running in a loop instead of shutting down and restarting for every test), cuFuzz became much faster, finding bugs in record time.
The Big Picture
Before cuFuzz, testing GPU software was like trying to find a needle in a haystack while wearing blindfolded gloves. You might find a few needles, but you'd mostly find hay (false alarms) or miss the needles hidden in the dark.
cuFuzz takes off the blindfold, gives you a metal detector, and lets you walk through the whole factory at once. It proves that even in the chaotic, super-fast world of modern GPUs, we can build tools to keep our software safe, reliable, and bug-free.
In short: cuFuzz is the ultimate quality control system for the AI and graphics revolution, ensuring that the powerful computers of tomorrow don't crash because of a tiny, overlooked mistake today.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.