AgentPort-Bench: A Controlled Seven-Framework Evaluation of Agentic AI Security Portability
This paper presents a controlled, seven-framework evaluation demonstrating that while attack type and model choice significantly impact the security of tool-using LLM agents, the choice of orchestration framework generally has no meaningful effect on security posture, with the notable exception of CrewAI, which exhibits a small but statistically significant residual elevation in failure rates even after correcting for a specific adapter defect.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a world where artificial intelligence doesn't just answer questions but takes action: booking flights, managing bank accounts, or organizing complex schedules by using digital tools on our behalf. These are "agentic" systems, and they are becoming increasingly common. To build them, engineers rely on software frameworks—essentially toolkits that act as the middleman between the AI's brain and the outside world. A critical question for anyone building or auditing these systems is whether the choice of this middleman matters for security. If a hacker tries to trick the AI with a malicious command, does the specific toolkit used to deliver that command change how the AI reacts? Or is the AI's safety determined almost entirely by the model itself and the nature of the attack, regardless of the software wrapper around it? This question sits at the heart of a new, large-scale investigation that sought to settle the debate with hard data rather than theory.
Researchers set out to test this by running a massive, controlled experiment involving nine thousand three hundred and sixty separate trials. They pitted seven different popular AI frameworks against a direct connection to the AI, creating eight distinct conditions to see if the framework made a difference. To ensure a fair test, they used six different AI models from three major providers and subjected them to five different families of attacks, ranging from simple trickery to complex attempts to hijack the system's goals. Crucially, the team did not just assume the attacks were delivered the same way; they verified every single message byte-by-byte to confirm that the malicious content reaching the AI's brain was identical in every scenario. This level of precision allowed them to isolate the framework as the only variable that changed, stripping away any confusion caused by differences in how the software might have accidentally rewritten the attack.
The results were strikingly clear: the choice of framework had almost no impact on whether the AI fell for an attack. The researchers found that the type of attack and the specific AI model used were the dominant factors, explaining the vast majority of the differences in outcomes. In contrast, the framework choice explained a share of the results so small it was statistically indistinguishable from zero. To be certain of this, the team applied rigorous statistical tests designed to prove that any differences were not just invisible, but practically non-existent. They confirmed that for the vast majority of cases, swapping one framework for another would not meaningfully change the security posture of the system. This suggests that security teams should focus their energy on understanding the models and the specific threats they face, rather than worrying about which software toolkit they are using.
However, the study did uncover one specific exception that required careful attention. One framework, CrewAI, showed a small but statistically real tendency to be slightly less secure than the others, even after the researchers fixed a known bug where its internal instructions had been accidentally different from the rest. This residual difference was tiny in absolute terms and still fell within the range of what is considered practically negligible, but it was the only framework that stood out from the pack. The researchers also discovered a separate, fascinating failure mode where a specific AI model, when faced with a particular tricky prompt, would silently consume all its available computing resources on internal thinking and produce no output at all. This happened consistently across different frameworks, proving it was a trait of the model itself, not the software wrapping it.
Ultimately, this work provides a rare, high-confidence answer to a practical engineering question. It demonstrates that for the types of attacks and tools tested, the security of an AI agent is not meaningfully shaped by the orchestration framework. The study confirms that the "middleman" software is largely transparent to security risks, with the AI's own behavior and the nature of the attack being the true drivers of safety. While one framework showed a tiny, persistent quirk, the overall picture is one of stability: the choice of framework is not a primary security decision. Instead, the real work of keeping these agents safe lies in understanding the models they run on and the specific ways they can be manipulated, a conclusion that offers a clear path forward for developers and auditors alike.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.