Emerging Challenges in Threat Modeling for GenAI-Augmented Systems: A View from the Trenches
This paper presents an exploratory assessment of GenAI-aware threat modeling methods in a Small and Medium Enterprise setting, revealing that while existing techniques identify different threats, they currently lack sufficient support for GenAI-specific risks related to software supply chains and human-centered security, alongside notable challenges in usability and workflow integration.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are building a massive, high-tech castle. Before you lay a single brick, you need a "threat model." Think of this as a special kind of treasure map where, instead of marking where the gold is, you draw every possible way a villain could sneak in, break a wall, or trick the guards. For decades, security experts have used a standard checklist called STRIDE to draw these maps. It's like having a master list of common burglaries: someone pretending to be a guard (Spoofing), smashing a window (Tampering), or stealing the keys (Information Disclosure). It works great for traditional software, which is like a rigid, predictable machine.
But now, we are building a new kind of castle: one powered by Generative AI (GenAI). This isn't a rigid machine; it's more like a brilliant, chaotic, and sometimes hallucinating wizard living inside your walls. This wizard can write code, answer questions, and create art, but it can also be tricked by a clever whisper (a "prompt injection") or fed bad ingredients that make it spit out poison. The big question is: Can our old, rigid treasure maps (the STRIDE checklist) find the sneaky ways to trick this new wizard? If we try to use a map designed for a stone fortress to find a spy in a magic library, will we miss the real dangers? This paper dives into that exact problem, asking if our current security tools are ready for the age of AI wizards.
The Paper's Story: Testing Old Maps on New Wizards
The authors of this paper, a team of security researchers and industry practitioners, decided to put this question to the test. They didn't just sit in a lab and guess; they went into the "trenches" of a real-world company (a Small and Medium Enterprise, or SME) that was building a system using AI to process customer emails and generate business reports. They wanted to see how well three different "threat modeling" methods would work on this AI-augmented system.
To do this, they first did a quick scan of the latest research to find the best new methods designed specifically for AI. They picked three contenders:
- M1 (AIaaS Framework): A method that looks at the system like a set of assets (data, models, processes) and asks, "How could someone mess with these?"
- M2 (ADMIn Framework): A method that focuses on three main attack surfaces: the data, the model, and the inputs.
- M3 (ThreatFinderAI): A fancy, tool-supported method that uses a special language to draw the AI parts of the system and connects to a giant database of known AI attacks to find threats automatically.
They applied all three methods to the company's system, which used a Large Language Model (LLM) to read emails and write reports. Then, they checked their findings against a famous "Top 10" list of AI risks (the OWASP Top 10 for LLMs) to see which method caught the most important dangers. Finally, they asked the actual developers at the company what they thought about using these methods.
What They Found: The Good, The Bad, and The Missing
The results were a mix of "not bad" and "we need to do better."
The Good News:
All three methods were pretty good at spotting the "classic" AI tricks. They successfully identified Prompt Injection (where a user tricks the AI into ignoring its rules), Data Poisoning (where bad data corrupts the model), and Sensitive Information Disclosure (where the AI accidentally spills secrets). The tool-supported method, M3, was the clear winner. It found the most threats, was the easiest for the developers to use, and felt the least like a chore. It's like having a metal detector that beeps loudly when it finds a buried coin, compared to digging with a spoon.
The Bad News (The Gaps):
Here is where the old maps struggled. The methods showed limited support for spotting risks related to the software supply chain (like if the AI model itself was built with stolen or broken parts) and Excessive Agency (when the AI is given too much power to make decisions without a human checking its work).
- Supply Chain: None of the methods offered a specific way to model or identify risks coming from the "ingredients" the AI used. If the AI model was downloaded from a sketchy website, the methods lacked the specific syntax or checklists to flag it effectively.
- Human-Centered Risks: They also missed the danger of humans trusting the AI too much. If the AI writes a piece of code that looks perfect but is actually dangerous, and a human developer just copies it without checking, that's a huge risk. The methods didn't have a good way to map out how that "lazy trust" could break the system.
What the Developers Said:
The developers who tried these methods agreed with the data. They loved M3 because it had a tool that did the heavy lifting. They felt that the other methods were too manual and took too much time. However, they all pointed out a major hurdle: these methods don't fit easily into their daily workflow. They want checklists, templates, and a way to connect the security findings directly to their task management software (like a "ticket" system). One developer said, "Templates would make it so much easier," while another noted that they wouldn't have caught the data leakage risks on their own without these tools.
The Takeaway: We Need New Maps for New Castles
The paper suggests that while our current tools are a good start for catching the obvious AI tricks, they are not enough for the full picture. We are currently blind to the risks coming from the supply chain (where the AI comes from) and the risks of humans trusting the AI too blindly.
The authors propose that the future of security modeling needs to change in three ways:
- Include the Supply Chain: We need maps that show not just the castle, but the roads bringing in the bricks and the people who made them.
- Use Tools and Prompts: We need automated tools that can help us draw these maps quickly, perhaps even using AI to help us find the threats in the first place.
- Watch the Humans: We need to model how humans interact with the AI, specifically looking for moments where a human might get lazy and trust a wrong answer, turning a small mistake into a big disaster.
In short, the paper suggests that we can't just patch the old maps; we need to draw entirely new ones that account for the magic, the supply chain, and the human tendency to trust the wizard a little too much. Until we do, our AI castles might look secure on paper, but they could still have secret doors we haven't even thought to lock.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.