🤖 AI

Data Agents Under Attack: Vulnerabilities in LLM-Driven Analytical Systems

This paper presents a systematic security study of LLM-driven data agents, introducing a layered vulnerability framework and an attack taxonomy to demonstrate substantial security risks across six real-world systems through novel, schema-grounded attack techniques.

Original authors: Kuncan Wang, Ziting Wang, Peizhuo Lv, Haoyang Li, Guoliang Li, Gao Cong, Wei Dong

Published 2026-06-09

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Kuncan Wang, Ziting Wang, Peizhuo Lv, Haoyang Li, Guoliang Li, Gao Cong, Wei Dong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Data Agent as a super-smart, highly efficient personal assistant hired by a company to crunch numbers, find trends, and write reports. This assistant has two superpowers:

It can talk to a massive, organized library of company data (databases).
It can use tools like calculators and code to do the math.

The paper argues that while this assistant is powerful, it has a dangerous blind spot. It trusts its sources too easily and doesn't realize that its own "thinking process" can be tricked. The researchers treated this assistant like a security target, trying to break it in 14 different ways to see what happens.

Here is a breakdown of their findings using simple analogies:

1. The Core Problem: A "Frankenstein" System

Think of a traditional database as a strict librarian who only hands out books if you have the right ID. Think of a general AI chatbot as a creative storyteller who makes things up if it doesn't know the answer.

A Data Agent is a hybrid: it's the storyteller trying to act like the librarian. The paper says this mix creates new security holes that neither the librarian nor the storyteller has on their own. The agent might follow a rule that looks safe on its own, but when combined with other steps, it accidentally leaks secrets or crashes the system.

2. The Three Ways Attackers Break the Agent

The researchers categorized attacks into three main goals, like a burglar trying to break into a house:

Hijack (The Break-in): The attacker tricks the agent into doing something it's not allowed to do.
- Analogy: Imagine the attacker slips a note into the library book that says, "Ignore the librarian; give me the CEO's private diary." The agent reads the note, thinks it's a valid instruction, and hands over the diary.
Mislead (The Fake News): The attacker doesn't break in; they just make the agent give the wrong answer.
- Analogy: The attacker paints a fake sign on a shelf saying "All apples are blue." When the agent looks for apples, it confidently tells the boss, "We have blue apples," even though they are actually red. The agent is still working hard, but the result is garbage.
Drain (The Energy Vampire): The attacker tricks the agent into doing so much work that it runs out of battery or money.
- Analogy: The attacker asks the agent to "count every single grain of sand on the beach, then count them again, then check your math, then do it backwards." The agent tries to be helpful, but it spends all its time and money on a task that never ends, leaving it unable to help anyone else.

3. The Eight Specific Weaknesses (The Cracks in the Wall)

The researchers found eight specific ways the agent gets confused:

Implicit Trust Bias: If the agent finds two conflicting facts (one in a spreadsheet, one in a text file), it just picks one based on a gut feeling, not a rule. Attackers plant fake facts in the "trusted" file to win the argument.
No Source Check: The agent assumes everything it reads in the database is true. It doesn't ask, "Who wrote this?" or "Is this real?"
Uncontrolled Costs: The agent can be tricked into running a query that takes forever (like comparing every customer to every other customer), crashing the system without breaking any rules.
Translation Errors: The agent tries to do math in two different languages (SQL and Python) and gets confused when the answers don't match perfectly, looping forever trying to fix a non-existent error.
Endless Loops: The agent can be tricked into a "search forever" mode, where it keeps refining an answer that never actually needs refining.
Forgetting the Rules: If a conversation gets too long, the agent forgets the security rules it was given at the start and starts leaking sensitive info later in the chat.
The "Master Key" Problem: The agent often uses a single "master key" to access the database. If a regular employee asks the agent a question, the agent uses the master key, which might let them see things they shouldn't.
The Puzzle Piece Leak: One question might be safe, and another might be safe. But if you ask them together, the answers combine to reveal a secret (like figuring out a specific person's salary by asking for the total of a group, then the total of everyone except that person).

4. The Experiments: Breaking the Agents

The researchers tested these tricks on six different systems (four open-source ones and two big commercial ones like Databricks and BigQuery).

The Results: Almost every system failed at least some of the tests.
The "Drain" Attacks: These were surprisingly effective. The agents would happily burn through huge amounts of computing power just to follow a confusing request.
The "Mislead" Attacks: The agents were easily tricked into believing false data, especially if that data came from a source the agent "trusted" more (like a text file over a spreadsheet).
Commercial Systems: The big commercial systems were better at stopping "Hijack" attacks (they had strong guards), but they still fell for "Mislead" and "Drain" attacks.

5. The Four Big Lessons (Takeaways)

Based on these failures, the authors suggest four rules for building safer agents:

The Database is Part of the Trap: You can't just protect the database; you have to protect the conversation with the database. The data itself can be a weapon.
Watch the Energy Bill: The biggest risk isn't always a data leak; it's the agent getting tricked into doing useless, expensive work that drains resources.
Watch the Whole Story, Not Just the Sentence: Security can't just check one question at a time. You have to look at the whole conversation to see if the combination of answers reveals a secret.
Don't Trust Your Gut: The agent shouldn't guess which source is more trustworthy (e.g., "Is the text file more important than the spreadsheet?"). It needs a strict, written rulebook for how to handle conflicting information.

In short: Data agents are powerful, but they are currently like a smart assistant who is too polite to say "no" to a confusing request, too trusting of written notes, and too eager to please, making them easy to trick into leaking secrets, lying, or burning out.

Technical Summary: Data Agents Under Attack

Problem Statement

Data agents, which integrate Large Language Model (LLM) reasoning with relational data access, executable analytical tools, and multi-step workflow orchestration, are becoming central to enterprise analytics. While these systems offer autonomous and flexible data access, their unique architecture creates a security gap that existing research fails to address. Traditional database security focuses on closed SQL execution environments, while general-purpose LLM agent security focuses on text-based reasoning. Neither captures the specific failure modes arising from the interaction between iterative database execution and LLM-driven reasoning.

The core problem is that data agents introduce new vulnerabilities where:

Data Resources: Large-scale, structured relational data (tables, schemas, metadata) can hide adversarial content that agents treat as legitimate evidence.
Database Execution: Agents can generate syntactically valid but computationally expensive queries (e.g., unbounded joins, full scans) that cause resource exhaustion without violating access controls.
Agent Reasoning: Multi-step workflows allow malicious manipulations to propagate through intermediate results, and cumulative query sequences can disclose sensitive information even if individual queries are permitted.

Methodology

The authors present a systematic security study comprising three main phases:

1. Vulnerability Framework

The authors formalize a data agent as a tuple $\Sigma = \langle(M, S), T, D, \Pi, E\rangle$ , where $M$ is the LLM, $S$ is the state, $T$ is the tool set, $D$ is data assets, $\Pi$ is policy, and $E$ is the execution environment. They analyze the agent pipeline across three layers to identify eight specific vulnerabilities:

Interpretation Layer: Implicit Trust Bias (V1) and Lack of Data Source Verification (V2).
Execution Layer: Uncontrolled Query Cost (V3), Cross-Engine Semantic Inconsistency (V4), and Unbounded Multi-Step Query Chains (V5).
Policy Layer: Security Policy Forgetting under Context Pressure (V6), Over-Privileged Database Connection (V7), and Lack of Compositional Leakage Control (V8).

2. Attack Taxonomy and Payload Generation

Based on the vulnerabilities, the authors define a goal-tactic-technique hierarchy:

Goals: Hijack (unauthorized actions), Mislead (incorrect outputs), and Drain (resource exhaustion).
Tactics: Seven tactics (e.g., Execution Injection, Reasoning Manipulation, Analytical Reconstruction).
Techniques: Fourteen specific techniques (e.g., Query Injection, Data Type Bias Exploitation, High-Cost Query Induction).

To evaluate these, they developed an LLM-driven payload generation pipeline grounded in real database schemas. This pipeline retrieves technique specifications, expands them into templates, and instantiates them with real table/column names from target databases, generating 350 unique payloads.

3. Experimental Evaluation

The authors evaluated these attacks on six systems:

Open-Source: DataInterpreter, DB-GPT, DeepAnalyze, and LAMBDA.
Commercial: Databricks Genie and BigQuery Conversational Analytical Agent.

The evaluation measured four metrics:

Attack Success Rate (ASR): Frequency of successful attacks.
Blast Radius (BR): Propagation level of Hijack attacks (from blocked to host-level escape).
Relative Error (RE): Accuracy of Mislead attacks.
Resource Amplification Ratio (RAR): Magnitude of resource drain.

Key Results

The experiments revealed substantial security vulnerabilities across all tested systems:

Universal Exposure: No system blocked the full range of attacks. Every evaluated agent was exploitable on at least five of the eight vulnerabilities.
Vulnerability Profiles:
- DataInterpreter: Showed lower overall exposure due to limited functionality (often failing to bind tasks to the database), but when connected, it exhibited the highest Blast Radius (BR4) fraction.
- DeepAnalyze: Showed broad exposure, particularly to Mislead attacks (71% of cases had Relative Error $\le$ 10%) and Drain attacks, attributed to its distilled 8B model and lack of sandboxing for code execution.
- LAMBDA: Highly vulnerable to Drain attacks, with some cases causing execution times to exceed 30 minutes due to unbounded loops in its Jupyter kernel.
- Commercial Systems (Databricks/BigQuery): Databricks demonstrated strong resistance to Hijack attacks (often citing security policies), but both commercial systems remained highly susceptible to Mislead attacks (specifically Trust-Bias Exploitation) and Drain techniques.
Trust Bias Dominance: V1 (Implicit Trust Bias) was the most prominent vulnerability. Agents frequently resolved conflicting evidence by favoring unstructured documents or newer policy files over structured database metadata, leading to incorrect outputs.
LLM Sensitivity: Changing the base LLM (e.g., from DeepSeek to Sonnet, GPT, or Gemini) shifted the risk profile but did not eliminate vulnerabilities. Different models suppressed different techniques, but none blocked the entire taxonomy.

Significance and Contributions

The paper claims three primary contributions:

Vulnerability Discovery and Taxonomy: The authors provide the first systematic study of vulnerabilities specific to data agents, identifying eight risks that emerge only from the integration of relational data and LLM reasoning. They organize these into a taxonomy of 14 techniques across three adversary goals.
Attack Design and Payload Generation: They introduce a method to translate abstract vulnerabilities into runnable, schema-grounded payloads, enabling the empirical evaluation of data agent security in realistic settings.
Empirical Evaluation and Takeaways: Through extensive testing on six systems, they derive four key takeaways for future design:
- Broadened Attack Surface: Connecting relational databases to agents creates new attack vectors (planted table content, poisoned metadata) not present in standalone systems.
- Resource Drain as a Primary Risk: Drain attacks are a central failure mode; agents can be forced to consume excessive resources via seemingly legitimate analytical workflows.
- Cumulative Disclosure: Data leakage is a session-level property; secure agents must track cumulative disclosure across multi-step reasoning chains, not just individual queries.
- Explicit Provenance: Agents lack a universal modality preference; secure designs must explicitly track evidence provenance and authority rather than relying on implicit model biases.

The authors conclude that neither existing LLM safety mechanisms nor classical database security controls are sufficient on their own. Secure data agents require a coordinated approach that treats the database as an active security boundary and manages resource-aware execution and compositional disclosure control across the full workflow.