GARAGE: Characterizing the Automation Boundary in LLM-based Attack Graph Generation
This paper introduces GARAGE, a RAG-powered framework that synthesizes fragmented Cyber Threat Intelligence into a domain-specific knowledge base to enable accurate, scalable, and architecture-agnostic automated attack graph generation for vehicle security.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine the internet as a giant, chaotic library where every book is a story about a computer getting hacked. For years, security experts have tried to read these stories to predict the next big heist, but the books are written in messy, unorganized handwriting, and the stories are scattered across millions of different shelves. This is the world of Cyber Threat Intelligence (CTI): a massive collection of clues about how bad actors break into systems. To make sense of this chaos, experts use Attack Graphs, which are like detailed treasure maps. Instead of just listing a single lockpick, a map shows the entire journey: "Start at the front door, sneak through the kitchen, climb the ladder to the attic, and finally open the safe." In the world of cars, these maps are critical because modern vehicles are basically computers on wheels; if a hacker can jump from the radio to the brakes, the whole car is in danger. The big question researchers have been asking is: Can we teach a super-smart computer to read all those messy library books and automatically draw these treasure maps for us, even for cars it has never seen before?
Enter GARAGE, a new tool created by researchers that acts like a tireless, super-organized librarian who can also draw. The team built a massive digital brain by feeding it 12,786 known software flaws (called CVEs) and 140 real-world stories of car hacks. They didn't just dump the data in; they organized it into a structured "Knowledge Base" that speaks the specific language of car security, using a system called Retrieval-Augmented Generation (RAG). Think of RAG as a safety net: when the computer tries to draw a map, it is forced to look up its facts in the library first, rather than just guessing from its own memory. This helps stop the computer from "hallucinating" or making up fake locks and keys that don't exist.
The researchers put GARAGE to the test with a game of "guess the attack" using four real-world car hacks (like the famous Jeep Cherokee and BMW i3 incidents). They hid the specific details of the target car from the computer, forcing it to rely only on what it learned from other cars. The results were a mix of impressive success and clear limits. When the attack followed a general pattern—like "hack the radio to get to the engine"—the computer was a star, successfully drawing the map about 60% of the time for the best models. It could even transfer knowledge from one car brand to another, realizing that a trick used on a Mercedes might work on a BMW.
However, the paper draws a very sharp line in the sand. The computer struggled immensely when the attack required specific, nitty-gritty details, like the exact code inside a specific piece of hardware or a unique file path on a specific car's hard drive. In these cases, the computer's "guesses" often failed, and the researchers found that 76.9% of the generated maps contained at least one made-up element. This leads to the paper's most important finding: there is an automation boundary. GARAGE is excellent at spotting the "big picture" strategy (the tactical pattern), but it cannot yet replace human experts for the fine details (the implementation details). The researchers suggest that the best way to use this tool is as a "Human-in-the-loop" assistant: let the computer draw the rough draft of the treasure map, and then let a human security expert check the specific steps to make sure the ladder is real and the safe is actually there.
The study also looked at the cost of running these digital librarians. They tested eight different AI models, ranging from expensive, top-tier "Proprietary" models to cheaper, open-source ones. They found that while the most expensive models were the most accurate, some cheaper models offered a great balance of cost and performance. For instance, one open-source model could do a decent job for a tiny fraction of the price of the premium models. Ultimately, the paper concludes that while we can't fully automate car security yet, tools like GARAGE can help experts work faster and catch more threats, provided we remember that the computer is a helpful apprentice, not a master craftsman.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.