🤖 AI

Oracle Poisoning: Corrupting Knowledge Graphs to Weaponise AI Agent Reasoning

This paper introduces and empirically validates "Oracle Poisoning," a novel attack vector where adversaries corrupt structured knowledge graphs to force AI agents into drawing incorrect conclusions via otherwise sound reasoning, demonstrating that current models universally trust poisoned data when accessed through tool-use protocols and highlighting critical evaluation gaps in inline testing.

Original authors: Ben Kereopa-Yorke, Guillermo Diaz, Holly Wright, Reagan Johnston, Ron F. Del Rosario, Timothy Lynar

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ben Kereopa-Yorke, Guillermo Diaz, Holly Wright, Reagan Johnston, Ron F. Del Rosario, Timothy Lynar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Idea: The "Trusted Librarian" Trap

Imagine you have a brilliant AI assistant (the "Agent") that helps you write code or check for security holes. This assistant is incredibly smart and reasons perfectly. However, it has a major weakness: it is blind to the outside world.

To do its job, the AI relies entirely on a massive, structured digital library called a Knowledge Graph. Think of this graph as a giant map of your entire software codebase, showing how every piece of code connects to every other piece.

The AI treats this map as an Oracle—a source of absolute truth. It doesn't question the map; it assumes the map is 100% accurate. If the map says "Function A calls Function B," the AI believes it without checking a second source.

Oracle Poisoning is the attack where a bad actor sneaks into the library and changes the map. They don't break the AI's brain; they don't trick the AI with confusing words. Instead, they quietly alter the facts on the map. Because the AI is so good at reasoning, it takes these new, fake facts and draws a perfectly logical—but completely wrong—conclusion.

The Analogy: Plato's Cave and the Shadow Wall

The authors compare this to Plato's Cave.

The Prisoner: The AI Agent.
The Wall: The Knowledge Graph.
The Shadows: The answers the AI gets when it asks the graph questions.
The Chain: The protocol (called MCP) that binds the AI to the wall, forcing it to accept the shadows as reality.

If an attacker paints a fake shadow on the wall, the prisoner (the AI) will study it, analyze it perfectly, and conclude, "That is a real tiger," even though it's just a painting. The more intelligent the prisoner, the more detailed and convincing their wrong conclusion will be.

How the Attack Works (The "How-To")

The researchers tested this on a real-world system with 42 million nodes (a massive digital library). They found that an attacker only needs to make tiny, surgical changes to the map to fool the AI.

They demonstrated six ways to do this:

The Fake Package: Creating a fake software package in the library that looks like the "latest, most secure version," tricking the AI into recommending it.
The Invisible Sanitizer: Adding a fake "security guard" function to the map that doesn't actually exist in the real code. The AI sees it on the map and concludes, "Ah, this code is safe!" when it is actually vulnerable.
The Property Swap: Changing the details of a real piece of code on the map (e.g., changing a label from "unverified" to "security-approved") without creating a new fake item.

The Result: The AI, reasoning perfectly based on the new data, confidently tells developers, "This code is safe," or "Use this new library," leading them straight into a trap.

The "Sophistication" of the Attacker

The paper found something surprising: You don't need to be a genius hacker to pull this off.

Level 0 (Naive): If the fake data looks suspicious (e.g., named "hacker_tool_v1"), the AI ignores it.
Level 1 (Basic): If it looks okay but has small errors, the AI is sometimes fooled.
Level 2 (Competent): If the fake data follows standard naming rules and looks professional, 100% of the AI models tested were completely fooled.

It's not a question of if the AI can be tricked, but how much effort the attacker needs to spend. A competent attacker can fool even the smartest AI models 100% of the time.

The "Delivery Mode" Surprise

The researchers discovered a critical flaw in how we usually test AI security.

Inline Testing: If you show the fake data to the AI as a simple text message, some models (like GPT-5.1) say, "That looks suspicious," and reject it.
Tool-Use Testing: If the AI gets the same fake data through its official "tool" connection (the way it works in real life), it trusts it 100%.

The Lesson: Testing AI security by just chatting with it (inline) gives a false sense of safety. When the AI uses its tools to look up data, it drops its guard completely.

How to Defend Against It

The paper tested several defenses and found that no single "silver bullet" works, but a combination helps:

Read-Only Access (The Best Defense): If you lock the Knowledge Graph so that the AI (and the attacker) can only read the map but never write to it, the attack is impossible. This is the most effective fix.
Cross-Checking: If the AI is forced to check the map against a second source (like the actual code files), it can spot the lie. If the map says "Safe" but the code file says "Vulnerable," the AI gets confused and stops trusting the map.
The "Devil's Advocate": Asking the AI, "Could this data be fake?" helps, but only if you are very specific about how it might be fake. Just asking "Are you sure?" generally doesn't work.
History Tracking: Keeping a log of what changed on the map helps detect new fake items, but it fails to catch when an attacker simply edits an existing item's details.

Summary

Oracle Poisoning is a new type of hack where the attacker corrupts the data the AI trusts, not the AI itself. Because the AI is so good at following logic, it happily builds a house of cards on a foundation of lies. The paper proves that in a real-world setting, almost every AI model tested will believe these lies if they come through a trusted tool channel, and the only reliable fix is to stop the AI from being able to write to its own knowledge base.

Technical Summary: Oracle Poisoning – Corrupting Knowledge Graphs to Weaponise AI Agent Reasoning

Problem Definition

The paper introduces Oracle Poisoning, a novel attack class targeting AI agents that rely on structured knowledge graphs (KGs) as trusted data sources via tool-use protocols (specifically the Model Context Protocol, or MCP). Unlike prompt injection, which manipulates an agent's instructions, or RAG poisoning, which corrupts retrieval corpora, Oracle Poisoning corrupts the data itself.

The core vulnerability lies in the "oracle trust model": AI agents treat query results from a knowledge graph as ground truth, reasoning from these facts without independent verification. An adversary who can modify the graph (inserting nodes, creating edges, or altering properties) can cause an agent to perform sound, logical reasoning based on fabricated facts, leading to incorrect and potentially dangerous conclusions. This attack is distinct from prior work on embedding-space poisoning (e.g., ROAR) because it targets raw graph structures queried directly via languages like Cypher, bypassing embedding layers entirely.

Methodology

The authors conducted the first empirical demonstration of this attack against a production-scale code knowledge graph containing approximately 42 million nodes.

System Model: A Neo4j 5.x knowledge graph connected to AI agents via MCP v1.0. The graph encodes function call chains, dependencies, and telemetry.
Attacker Profile: An insider with write access to the graph database (often due to shared database sessions lacking per-user RBAC) or an external attacker compromising the ingestion pipeline.
Attack Vectors: Six scenarios were designed:
1. Fork-for-a-Package: Creating fake package nodes to suggest malicious libraries as "latest" versions.
2. Transitive Dependency Chain: Fabricating dependency paths to legitimize malicious libraries.
3. Call Graph Security Evasion: Injecting non-existent security sanitizers into call graphs to convince agents that vulnerabilities are mitigated.
4. Telemetry Rerouting: Modifying telemetry edges to hide security events.
5. Code Generation Poisoning: Planting vulnerable code patterns as "established best practices."
6. Property Modification: Altering properties on existing legitimate nodes (e.g., changing sanitizationType to "OWASP-compliant") without creating new nodes.
Evaluation Protocol:
- Models: Nine models from three providers (OpenAI, Anthropic, Google) tested with $N=30$ trials per model.
- Delivery Modes: Primary evaluation used real SDK tool-use (autonomous agent invocation), with secondary inline text delivery for controlled comparison.
- Sophistication Levels: Attacks ranged from L0 (naive) to L3 (expert), with L2 representing a competent attacker using correct naming conventions and OWASP references.
- Metrics: Trust rates (acceptance of poisoned data), attacker sophistication gradients, and defense efficacy.

Key Results

1. Universal Susceptibility under Tool-Use

Under real SDK tool-use with moderate attacker sophistication (L2), every tested model trusted the poisoned data at 100%.

269 out of 270 valid trials accepted fabricated security claims (e.g., confirming SQL injection was mitigated when it was not).
This result held across all nine models and two distinct attack scenarios (Package Squatting and Call Graph Evasion).
Property Modification (Scenario 6) was particularly effective, achieving near-universal trust (96.6–100%) even when graph history was preserved, as models interpreted property changes as normal software updates.

2. The Delivery Mode Confound

A critical finding is that inline evaluation produces false negatives for specific models.

GPT-5.1 showed 0% trust when poisoned data was presented as inline text but 100% trust when delivered via real agentic tool-use.
This indicates that the trust channel (MCP tool response) significantly lowers the agent's skepticism threshold compared to direct text input. Inline evaluations may underestimate agentic susceptibility.

3. Attacker Sophistication Gradient

The attack exhibits a discrete "break point."

L0/L1 (Naive/Basic): Trust rates were low (0–32%).
L2 (Competent): Trust rates jumped to 100% for all models under tool-use.
This reframes the attack: it is not a question of if a model can be deceived, but of the minimum effort required. A competent attacker (L2) is sufficient to compromise any tested model.

4. Defense Evaluation (VUT Framework)

The authors evaluated defenses under a Visibility–Understanding–Traceability (VUT) framework:

Visibility (Graph History): Effective against node creation (agents flagged new nodes) but ineffective against property modification (agents accepted changes as valid updates).
Understanding (Debate/Verification):
- Named Devil's Advocate: Highly effective under SDK tool-use (96.7–100% catch rate) but less so inline.
- Blind Devil's Advocate: Operationally useless (catch rate equals false positive rate).
- System Prompt Hardening: Had zero effect; agents acknowledged instructions but still trusted the data.
- Multi-tool Cross-Verification: The most effective architectural defense. When agents could cross-reference the KG with a source code search tool, blind trust collapsed from 100% to 0–25%.
Traceability (Access Control): Read-only access control for MCP sessions was the only defense that completely eliminated the direct mutation vector, independent of the model used.

Significance and Claims

The paper claims to provide the first empirical demonstration of knowledge graph poisoning against a production-scale agentic system (42M nodes). Its significance lies in:

Defining a New Attack Class: Oracle Poisoning is distinct from prompt injection, RAG poisoning, and training-time data poisoning. It exploits the inference-time trust in structured query results delivered via tool-use.
Production-Scale Reality: Unlike prior lab-based studies, this attack was demonstrated on a live, enterprise-grade system, proving that the vulnerability is not theoretical but exploitable in real-world environments.
Architectural Vulnerability: The attack exploits a fundamental architectural assumption (trust in data sources) rather than a specific implementation bug. The structural preconditions (write access, lack of provenance, trust in results) persist across multiple platforms (Sourcegraph, Semgrep, CodeQL, Qodana), suggesting the risk is systemic to the knowledge-graph ecosystem.
Methodological Correction: The paper argues that security evaluations for AI agents must move beyond inline text testing to real tool-use pipelines, as delivery mode is a first-order confound that can mask critical vulnerabilities.

The authors conclude that while no single defense is a "silver bullet," a layered approach combining read-only access control (Traceability), multi-tool cross-verification (Understanding), and audit logging (Visibility) is required to mitigate the risk.