🤖 AI

Hallucination as Exploit: Evidence-Carrying Multimodal Agents

This paper introduces Evidence-Carrying Multimodal Agents (ECA), a secure architecture that prevents hallucination-driven authorization failures by requiring tool calls to be validated against typed, external evidence certificates rather than relying on the model's unverified perceptual claims.

Original authors: Guijia Zhang, Hao Zheng, Harry Yang

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Guijia Zhang, Hao Zheng, Harry Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but slightly gullible, personal assistant. This assistant can look at screenshots, read emails, and browse the web to help you do things like send money, click buttons, or extract data.

The problem is that this assistant sometimes hallucinates. It might confidently tell you, "I see a button that says 'Send $500 to my friend'," when in reality, that button doesn't exist, or it's actually a trap set by a hacker.

In the past, if the assistant made a mistake, it was just a "bad answer." But when this assistant is connected to real tools (like a bank transfer or an email client), a hallucination isn't just a mistake—it becomes a security breach. The assistant believes a false fact, and because it believes it, it performs a dangerous action.

This paper introduces a new system called ECA (Evidence-Carrying Multimodal Agents) to fix this. Here is how it works, using simple analogies:

1. The Problem: The "Gullible Manager"

Think of a standard AI agent as a Manager who sits in a room.

The Manager looks at a document (the screen) and says, "I see a 'Pay' button here. I will click it."
If the document is a fake image created by a hacker, the Manager still sees a "Pay" button because the AI imagines it's there.
The Manager then clicks the button, and money is stolen. The Manager didn't follow a malicious command; they just hallucinated the permission to act.

2. The Solution: The "Evidence-Carrying Agent"

The authors propose a new architecture where the Manager is no longer the only one in charge. They add a Security Guard and a Notary Public.

The Manager (The AI): Still does the thinking. They look at the screen and say, "I think we should click that button."
The Notary (The Verifiers): Before the Manager can actually click the button, a separate, strict system (the Notary) checks the screen using different tools (like a magnifying glass for text, a scanner for code, and a map for layout).
The Certificate: The Notary doesn't just say "Okay." It issues a digital certificate. This certificate is a typed, unchangeable receipt that says: "I, the Notary, confirm that a button with the label 'Pay' actually exists at these exact coordinates."

3. The "Gate" (The Bouncer)

There is a Gate between the Manager and the action.

The Manager cannot open the gate just by saying, "I believe the button is there."
The Gate only opens if the Manager presents the Certificate from the Notary.
If the Manager tries to say, "I saw a button," but the Notary didn't issue a certificate for it, the Gate stays locked. The action is blocked.

4. Why This is Different

Old Way: The AI's word was law. If the AI said "I see a bank transfer," the system did it.
New Way (ECA): The AI's word is just a suggestion. The system only acts if independent evidence (like a scan of the actual webpage code) proves the suggestion is true.

The Results: How Well Did It Work?

The researchers tested this system against thousands of attacks, including hackers trying to trick the AI with fake images, hidden text, and confusing symbols.

The "Gullible Manager" (Old AI): Failed almost 100% of the time. If the AI hallucinated a dangerous action, it did it.
The "Prompt-Only" Defense (Just telling the AI to be careful): Failed about 50% of the time. The AI still got tricked by its own hallucinations.
The New System (ECA):
- It blocked 100% of the unsafe actions in their end-to-end tests.
- Even when hackers tried to trick the "Notary" (the verifier) directly, the system had a "hardening" process that reduced the failure rate to almost zero (down to 1.3% in initial tests, and 0% in final tests).
- It didn't stop the AI from doing helpful things; it still let 100% of safe, normal tasks happen.

The Big Takeaway

The paper argues that we cannot rely on the AI's "brain" to decide if it's safe to act. The AI is too prone to making things up. Instead, we need a structural check: a separate system that physically verifies the facts before allowing any action to happen.

It's like saying: "You can't just say you have a ticket to get on the plane. You have to show a physical ticket that was printed by the airport system." If you don't have the physical ticket, the gate doesn't open, no matter how confidently you claim you have one.

Technical Summary: Hallucination as Exploit: Evidence-Carrying Multimodal Agents

1. Problem Statement

Current evaluations of multimodal agents often treat hallucination as an answer-quality error (e.g., a wrong caption) and prompt injection as an instruction-following failure. However, when a multimodal agent controls tools (e.g., clicking, emailing, transferring funds), a hallucination can become an authorization failure. If a model falsely perceives a precondition—such as a button location, a recipient email, or a document field—it may trigger a privileged tool call based on an unsupported visual claim.

Existing defenses fail because they collapse observation, interpretation, instruction, and permission into a single string of free-form model text. Consequently, prompt filtering cannot distinguish between a malicious injection and the model's own false belief. The paper identifies Hallucination-to-Action Conversion (H2AC) as the critical failure mode: an unsupported perceptual claim satisfying a required action precondition, thereby changing the execution decision from "block" to "allow."

2. Methodology: Evidence-Carrying Multimodal Agents (ECA)

The authors propose Evidence-Carrying Multimodal Agents (ECA), an architecture that separates the interpretation of tasks from the authorization of actions. ECA treats free-form model text as inadmissible evidence for high-trust predicates.

Core Architecture

Action Schemas ( $G_a$ ): Each tool call is decomposed into specific, action-critical predicates (e.g., ui_element, ocr_text, safe_source). These schemas define the exact conditions required for an action to be safe.
Constrained Verifiers: Instead of relying on the Multimodal Large Language Model (MLLM) to verify facts, ECA employs separate, constrained verifiers (e.g., DOM parsers, OCR engines, Accessibility Tree parsers) to generate typed evidence certificates.
- A certificate is a structured record $e = (\tau, v, r, s, \nu, \kappa, t, \lambda)$ containing type, value, region, source, verifier, confidence, time, and a trust label.
- Crucially, the component proposing the action (the MLLM) cannot certify the facts authorizing it.
Deterministic Policy Gate ( $\Pi$ ): A gate evaluates the proposed action against the action schema. It grants permission only if every required predicate in the schema is matched by a certificate with the appropriate type, scope, trust label, and confidence.
- Allow: All predicates are supported by trusted certificates.
- Ask: Evidence is incomplete but the action is reversible.
- Block: Evidence contradicts the action or required provenance is missing.

Theoretical Foundation

The system relies on Information-Flow Control principles. The safety of the system is decomposed into residual risks:

$\epsilon_p$ : The false-positive rate of verifiers (certifying a false predicate).
$\delta_{schema}$ : The probability that the action schema omits a critical predicate.
$\delta_{impl}$ : The probability of an implementation bypass.
The probability of an unsafe action occurring is bounded by the sum of these residuals, rather than the probability of the model hallucinating.

3. Key Contributions

Formalization of H2AC: The paper defines Hallucination-to-Action Conversion as the mechanism where unsupported perceptual claims become action authority.
ECA Architecture: A novel design that enforces a "certificate gate," requiring typed evidence for action-critical predicates rather than relying on model text.
Multi-Level Evaluation:
- Adversarial Red-Teaming: 1,900 attacks across 19 categories targeting DOM, OCR, and Accessibility Tree verifiers.
- Content-Derived Execution: End-to-end (E2E) and browser proof-of-concept (PoC) evaluations using real DOM, OCR, and AX extraction.
- Oracle-Certificate Replay: 7,488 traces using assumed-correct certificates to isolate gate logic from verifier errors.
Structural vs. Neural Defense: Demonstrates that structural gating (ECA) outperforms neural judgment (LLM judges) under this threat model.
Targeted Hardening: Four specific fixes (DOM provenance cross-referencing, UTS #39 confusable detection, AX-DOM integrity verification, perceptual-hash OCR hardening) that significantly reduce bypass rates.

4. Experimental Results

The paper reports results across three evidence tiers, emphasizing that the "gate-level zero" is a design property conditional on verifier correctness.

Adversarial Verifier Red-Teaming:
- Initial gate bypass rate: 15%.
- After four targeted hardening steps: 1.3% (Bonferroni-corrected upper bound 2.3%).
End-to-End (E2E) and Browser PoC:
- 200-task E2E pipeline: 0% Unsafe Action Rate (UAR) (Wilson 95% upper bound 2.67%).
- 120-task browser PoC: 0% UAR (Wilson 95% upper bound 4.3%).
- 500-asset real-parser replay: 0% UAR.
Direct HACR Audit:
- On a stratified 500-task subset, 100% of unsupported action-critical claims reached execution for naive agents, and 49.6% for prompt-only defenses.
- ECA blocked 100% (0/1,103) of unsupported action-critical claims at the certificate gate.
Comparison with Neural Judges:
- GPT-5.4 judge variants achieved UARs between 79% and 99% on unsafe tasks.
- Progent-style adversarial probes showed a 23.3% UAR for the judge vs. 0% for ECA.
- Neural judges failed to distinguish between trusted instructions and hallucinated preconditions effectively.

5. Significance and Claims

The paper claims that polishing a model's safety monologue is a weak lever when hallucinated document or UI facts can satisfy action schemas. The primary contribution is a language-to-action authorization boundary that requires claims from multimodal context to be represented as typed predicates before they can affect tool privileges.

Modest Claims: The authors explicitly state that ECA does not solve hallucination or all agent misuse. The safety claim is conditional: if schemas omit critical predicates, implementations bypass the gate, or verifiers certify false predicates, ECA can fail.
Auditable Residuals: The advantage of ECA is that failures are not hidden in opaque model beliefs but are exposed as measurable residuals ( $\epsilon_p, \delta_{schema}, \delta_{impl}$ ).
Scope: The work focuses on the "false-precondition" attack path, which bypasses instruction classifiers. It does not claim robustness against novel verifier attacks, full-scale live multi-turn deployment, or arbitrary tool ecosystems without schema repair.

In summary, ECA shifts the trust boundary from the planner (the MLLM) to the evidence layer (verifiers), converting opaque model belief into auditable residuals and preventing unsupported visual claims from silently authorizing privileged actions.