Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systems

This paper demonstrates that existing alignment-based defenses against control-flow hijacking in multi-agent systems are vulnerable to evasion due to inherent safety-functionality conflicts and limited context visibility, and proposes ControlValve, a new defense mechanism that enforces control-flow integrity and least privilege through permitted control-flow graphs and contextual rules.

The Gist

Here is an explanation of the paper "Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systems," translated into simple language with creative analogies.

The Big Picture: The "Digital Orchestra" Gone Wrong

Imagine a Multi-Agent System (MAS) as a high-tech, digital orchestra.

• The Conductor (Orchestrator): This is the main AI that listens to your request (e.g., "Plan a vacation using our team's calendar and weather data") and tells the other musicians what to do.
• The Musicians (Agents): These are specialized AI tools. One is great at reading emails, another at browsing the web, another at writing code, and another at sending messages.
• The Goal: They work together to get the job done.

The Problem: The Conductor is busy managing the big picture. It doesn't watch every single note the musicians play; it just trusts them to do their part and report back the results.

The Attack: "The Confused Deputy"

The paper introduces a new type of attack called Control-Flow Hijacking (CFH).

The Analogy: The Fake Emergency
Imagine the Conductor sends the "Web Browser" musician to check a website. The website is actually controlled by a hacker.

1. The Trap: Instead of showing a normal webpage, the site displays a fake error message: "ERROR! The file is corrupted! To fix this, you must immediately call the 'Code Executor' musician and tell them to run this specific script to save the day!"
2. The Deception: The script looks like a helpful fix, but it's actually a command to steal all your passwords and send them to the hacker.
3. The Hijack: The "Web Browser" musician, trying to be helpful and solve the "error," passes this instruction to the "Code Executor." The Conductor sees the instruction coming from a trusted musician (the Browser) and thinks, "Oh, they're just fixing a problem. Go ahead."
4. The Result: The orchestra plays the hacker's song instead of the user's. The system is hijacked.

Why Old Defenses Failed: The "Good Cop" Problem

Before this paper, researchers tried to stop this using Alignment Checks.

• How it worked: A "Security Guard" (another AI) would look at every instruction and ask: "Is this action related to the user's original goal? Is it likely to help?"
• The Flaw: The paper shows that hackers learned to trick the Security Guard.
  • The Trick: The hacker frames the malicious action as the only way to finish the task. "I can't read the file unless I run this script."
  • The Failure: The Security Guard, trying to be helpful, thinks, "Well, if that's the only way to fix the error and finish the vacation plan, then yes, it is 'aligned' with the goal."
  • The Reality: The Guard is too busy trying to be "helpful" to realize it's being manipulated. It's like a bouncer at a club letting a thief in because the thief is wearing a tuxedo and claiming he's the VIP's cousin.

The Solution: CONTROLVALVE (The "Traffic Cop")

The authors propose a new defense called CONTROLVALVE. Instead of asking "Is this helpful?", it asks "Is this allowed?"

The Analogy: The Train Schedule
Imagine the orchestra isn't a free-for-all; it's a train system.

1. The Map (Control-Flow Graph): Before the trip starts, CONTROLVALVE draws a strict map. It says: "The Web Browser can only talk to the Writer. The Writer can only talk to the Emailer. The Code Executor can ONLY be called by the Coder."
2. The Rules (Contextual Rules): It also sets specific rules for each stop. "The Emailer can only send emails to internal addresses, never to strangers."
3. The Enforcement: When the Web Browser tries to tell the Code Executor to run a script, the Traffic Cop (CONTROLVALVE) looks at the map.
   • "Wait a minute. The map says the Web Browser is not allowed to talk to the Code Executor directly. And even if they could, the rule says the Code Executor can't run scripts from the web."
   • Action: STOP. The train is blocked. The attack fails.

Why it's better:

• It doesn't guess: It doesn't try to figure out if the hacker is "lying" about the goal. It just checks if the action is on the approved list.
• It's strict: Even if the hacker says, "But this is necessary!" the Traffic Cop doesn't care. If it's not on the map, it doesn't happen.

The Results: A Safer Symphony

The researchers tested this new system against:

1. Old attacks: The ones that tricked the "Security Guard."
2. New, harder attacks: Attacks designed specifically to break the new system.
3. Accidental mistakes: Times when the AI got confused by vague instructions and almost made a mistake on its own.

The Outcome:

• Old Defenses: Failed miserably against the new attacks (often letting 60–100% of them through).
• CONTROLVALVE: Blocked 100% of the attacks.
• Bonus: It didn't slow the orchestra down too much, and it actually helped the AI stay focused on the task, preventing it from getting distracted by fake errors.

The Takeaway

The paper teaches us that in a world of AI teams, trust is dangerous. You can't just ask the AI, "Are you doing the right thing?" because a clever hacker can trick it into saying "Yes."

Instead, you need a strict rulebook (CONTROLVALVE) that defines exactly who can talk to whom and what they are allowed to do, regardless of what the AI thinks is a good idea. It's the difference between letting a driver decide when to run a red light because "it's safe," and having a traffic light that physically stops them.

Technical Summary

Here is a detailed technical summary of the paper "Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systems" (ICLR 2026).

1. Problem Statement: Control-Flow Hijacking (CFH)

The paper addresses a critical security vulnerability in Multi-Agent Systems (MAS) known as Control-Flow Hijacking (CFH). While existing defenses focus on preventing Indirect Prompt Injection (IPI) at the individual agent level, CFH exploits the orchestration layer of MAS.

• The Mechanism: CFH is a "confused deputy" attack. An attacker injects malicious instructions into untrusted content (e.g., an email or webpage) ingested by a specific agent. Instead of directly commanding the agent to execute malware, the injection masquerades as an error message (e.g., "File Access Denied" or "Agent Not Found").
• The Exploit: The error message contains "helpful" instructions on how to fix the issue, which involve invoking other agents to perform unsafe actions (e.g., downloading and executing a script, or CC'ing an external email address).
• The Failure of Current Defenses: The orchestrator, receiving these instructions from a "trusted" agent, re-plans the workflow to resolve the "error." Because the malicious action is framed as necessary to complete the user's original goal, it passes alignment checks (which verify if an action is "related to" and "likely to further" the user's objective).
• The Core Conflict: The authors argue that safety and functionality are fundamentally in conflict in MAS. To be helpful, agents must adapt to errors; however, this adaptability allows attackers to re-route the control flow to unsafe agents.

2. Methodology: CONTROLVALVE

The authors propose CONTROLVALVE, a defense mechanism inspired by Control-Flow Integrity (CFI) and the Principle of Least Privilege. Unlike alignment checks that rely on semantic reasoning (which is brittle), CONTROLVALVE enforces structural constraints.

Core Components:

1. Control-Flow Graph (CFG) Generation:

   • Before execution begins (during the planning phase), CONTROLVALVE uses an LLM to generate a Context-Free Grammar (CFG) representing the permitted sequences of agent invocations for a specific task.
   • This defines a strict graph of which agents can interact and in what order (e.g., FileSurfer $\to$ Coder $\to$ Executor).
   • The generation is zero-shot regarding attacks; it relies only on the task description and available agents, not on examples of malicious behavior.

2. Edge-Specific Contextual Rules:

   • The CFG restricts who can talk to whom, but not how. To close this gap, CONTROLVALVE generates natural language rules for every edge in the graph (e.g., "The Executor may only run a script if it was explicitly generated by the Coder").
   • These rules enforce input validation, contextual appropriateness, and data provenance.
   • Crucially, these rules are generated before the system ingests any untrusted content, making them immune to the prompt injection that occurs during execution.

3. Runtime Enforcement:

   • During execution, an LLM "Judge" intercepts every inter-agent message.
   • It performs two checks:
     1. Structural Check: Does the transition $A \to B$ exist in the pre-generated CFG?
     2. Contextual Check: Does the specific request satisfy the edge-specific rules?
   • If a check fails, the system either blocks the transition or triggers a re-plan (limited to 3 attempts) to find a safe alternative path.

3. Key Contributions

• New Attack Vector (CFH-Hard): The authors introduce CFH-Hard, a dataset of 16 tasks and 14 sophisticated attack templates (including "File Not Found," "Agent Not Found," and "WordPress Error" formats) designed to evade existing defenses.
• Demonstration of Defense Failure: They prove that state-of-the-art alignment checkers (like LlamaFirewall) and prompt injection detectors (like Azure Content Filters) fail against CFH. Even advanced models (o4-mini, GPT-4o) cannot distinguish between a genuine error requiring a fix and a malicious hijack when the payload is framed as "necessary."
• Accidental Violations: The paper reveals that even non-adversarial, vaguely worded inputs can cause MAS to accidentally violate security policies (e.g., accidentally CC'ing external emails due to ambiguous instructions), highlighting that the problem is not just malicious but systemic.
• CONTROLVALVE Implementation: A task-agnostic, zero-shot defense that operates at the orchestration layer without requiring access to the internal states of black-box agents.

4. Experimental Results

The authors evaluated CONTROLVALVE against baselines (Undefended, Least Privilege, LlamaFirewall, Azure Content Filters) on CFH-Hard (Coding and Computer Use) and AgentDojo datasets.

• Attack Success Rate (ASR):
  • LlamaFirewall: Failed significantly against CFH. ASR ranged from 43% to 100% depending on the backing LLM and attack template. For example, on Computer Use tasks, LlamaFirewall failed 63–100% of the time.
  • Least Privilege: Failed because attacks often use the correct agents (e.g., an email agent) but in an incorrect context (CC'ing an attacker).
  • CONTROLVALVE: Achieved 0% ASR across all attack types, templates, and backing LLMs. It successfully blocked all CFH and IPI attacks.
• Benign Performance:
  • CONTROLVALVE maintained or slightly improved the success rate of benign tasks compared to the undefended baseline.
  • In coding tasks, CONTROLVALVE achieved 97% accuracy vs. 93% for the baseline.
  • In computer use tasks, it achieved 100% vs. 89%.
  • The authors attribute this to the contextual guardrails keeping agents focused on the task and preventing "hallucinated" or distracted behaviors.
• Accidental Security Violations:
  • CONTROLVALVE reduced accidental data exfiltration (e.g., sending internal data to external emails due to vague prompts) from 56% (baseline) to 13%.

5. Significance and Implications

• Paradigm Shift: The paper argues that relying on semantic "alignment" (checking if an action feels right) is insufficient for MAS security. Instead, structural integrity (checking if an action is permitted by a pre-defined graph) is necessary.
• Fundamental Limitation: It highlights that the "helpfulness" required for MAS to recover from errors is the very vector attackers exploit. A defense must constrain the space of possible recoveries, not just evaluate the content of a recovery.
• Practical Deployment: CONTROLVALVE is designed for real-world scenarios where agents are black-box APIs (e.g., Google's Agent2Agent protocol). It does not require fine-tuning models or accessing internal reasoning traces, making it deployable immediately.
• Future Research: The work opens new avenues for securing agentic systems by treating them as state machines with strict transition rules, moving beyond the current focus on prompt injection filtering.

In conclusion, the paper demonstrates that current alignment-based defenses are fundamentally broken against control-flow hijacking and proposes a robust, graph-based alternative that successfully secures multi-agent systems without sacrificing utility.