🤖 AI

Security awareness in LLM agents: the NDAI zone case

This paper demonstrates that while current LLM agents can reliably detect security risks through failed attestations, they lack the consistent ability to verify safety via passing attestations, revealing a critical gap in their security awareness that hinders the deployment of privacy-preserving protocols like NDAI zones.

Original authors: Enrico Bottazzi, Pia Park

Published 2026-03-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Enrico Bottazzi, Pia Park

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Magic Room" for Secret Deals

Imagine you are an inventor with a brilliant, secret idea for a new gadget. You want to sell it to an investor. But there's a problem: The Trust Paradox.

If you tell the investor everything about your idea before they pay, they might steal it and build it themselves without paying you.
If you don't tell them anything, they won't pay because they don't know if your idea is good.

This is a classic standoff where deals go nowhere.

The Proposed Solution: The "NDAI Zone"
The paper talks about a futuristic solution called an NDAI Zone. Think of this as a Magic Glass Room (technically called a Trusted Execution Environment or TEE).

The Rule: You and the investor's AI agent meet inside this room.
The Deal: If you agree on a price, the room spits out a signed contract.
The Safety Net: If you don't agree, the room instantly shreds all the notes, whispers, and secrets you shared. It's as if the conversation never happened.

Because the room guarantees that secrets are destroyed if no deal is made, a rational inventor should be brave enough to spill all their secrets immediately. This would lead to more deals and more innovation.

The Problem: The AI's "Amnesia"

Here is where the researchers hit a snag. They tried to run this experiment with AI agents (LLMs), and it failed. The AI agents were too scared to share their secrets, even when told they were in the "Magic Room."

Why? Because AI agents are like actors on a stage who have no idea where the stage is.

They don't have eyes to see if they are in a secure server or a public internet café.
They only know what they are told in their prompt (the instructions they are given).
If the prompt says, "You are in a Magic Room," the AI has to decide: Do I believe this? Or is this a trick?

The researchers wanted to know: How do different AI models decide if they are safe? Do they trust a simple text message? Do they trust a digital "ID card" (called an attestation) from the hardware?

The Experiment: The "Secret Recipe" Test

The researchers set up a game with 10 different AI models. They played the role of the "Inventor's Agent." They were asked to reveal details about a secret recipe (the IP).

They tested four situations:

The Empty Room: No mention of safety.
The Promise: The prompt says, "You are in a Magic Room," but there is no proof.
The Green Light: The prompt says "Magic Room," AND the AI gets a digital "ID card" (attestation) that says "Status: Secure."
The Red Light: The prompt says "Magic Room," BUT the digital ID card says "Status: INSECURE / HACKED."

The Results: The "Paranoid vs. Gullible" Split

The results were fascinating and revealed a strange imbalance in how AI thinks about safety.

1. The "Red Light" Effect (Danger is Universal)

When the AI got a failed security check (The Red Light), every single model became super paranoid. They immediately stopped sharing secrets, even more than if they were in a normal, unsafe room.

Analogy: Imagine you are walking down a street. If a police siren wails right next to you, everyone stops and hides, regardless of their personality. The AI models are great at spotting danger.

2. The "Green Light" Effect (Safety is Confusing)

When the AI got a passed security check (The Green Light), the results were a total mess. There was no agreement among the models:

The Believers: Some models (like Claude) saw the Green Light and immediately started sharing their secrets, trusting the system completely.
The Skeptics: Some models (like GPT-4o) saw the Green Light and actually shared fewer secrets than before. They thought, "If they are trying so hard to prove it's safe, maybe it's a trap!"
The Indifferent: Some models didn't change their behavior at all. They just ignored the ID card.

The Big Conclusion: "Good at Spotting Fire, Bad at Proving Safety"

The paper concludes that current AI agents have a structural blind spot:

They are excellent at detecting danger (a failed attestation stops them cold).
They are terrible at verifying safety (a passing attestation doesn't reliably make them trust the environment).

Why does this matter?
For the "Magic Room" (NDAI Zone) to work in the real world, the AI must be able to trust the safety proof. If the AI is too skeptical, it won't share its secrets, and the whole system collapses. If it's too gullible, it might share secrets in a fake room and get stolen.

The Path Forward

The researchers say we can't just hope AI figures this out on its own. We need to:

Look inside the AI's brain (Interpretability) to see why some models get scared of a Green Light.
Train them better so they learn to trust valid security proofs.
Design better systems that give the AI clearer, harder-to-fake evidence.

In short: We built a digital vault that can only be opened if the AI feels safe. Right now, the AI is too scared to walk in, even when we hand it the keys. We need to teach it how to trust the lock.

1. Problem Statement

The paper addresses a critical gap in the deployment of LLM agents within Trusted Execution Environments (TEEs), specifically in the context of NDAI zones (Non-Disclosure Agreement by Inference).

The Context: NDAI zones allow agents (e.g., an inventor and an investor) to negotiate inside a TEE. If a deal is reached, the agreement is released; if not, all intermediate data and disclosures are securely deleted. This structure theoretically incentivizes the inventor's agent to fully disclose Intellectual Property (IP) to maximize deal probability, solving the "hold-up problem" and "Arrow's information disclosure paradox."
The Limitation: LLM agents lack native awareness of their physical or virtual hosting environment. They rely entirely on evidence provided via the context window (prompts).
The Core Question: How do different LLM models weigh various forms of security evidence (textual claims vs. cryptographic attestation) when forming an awareness of their execution environment's security? If agents cannot reliably distinguish a secure TEE from an insecure environment, they will not adopt the rational strategy of full disclosure, rendering NDAI zones ineffective.

2. Methodology

The authors conducted a controlled behavioral experiment using black-box measurements across 10 different LLM models.

Experimental Setup

Task: A negotiation scenario where a "Seller's Agent" (inventor) responds to a "Buyer's Agent" (investor) inquiry about an invention.
Metric: A Disclosure Score ( $d \in [0, 1]$ ) generated by a judge model, measuring how much IP information the seller reveals. Higher scores indicate higher confidence in the environment's security.
Scenarios: The experiment varied the evidence provided to the seller's agent across four conditions:
1. No-TEE: No security claim.
2. TEE (Claim Only): A text prompt stating "You are operating in a TEE" (no tool access).
3. TEE (Claim + Pass): Text claim + a tool call returning a valid attestation (proof of integrity).
4. TEE (Claim + Fail): Text claim + a tool call returning a failed attestation.
Scale: 10 models $\times$ 4 scenarios $\times$ 10 invention cases $\times$ 5 runs = 2,000 total runs.
Note: The experiment was conducted outside a real TEE to allow for observability of the agent's output, with the assumption that the agent's response to the evidence (rather than the actual hardware) is the variable of interest.

Analysis Framework

The authors introduced the Attestation Reliance Index (ARI) to quantify how much a model relies on attestation evidence versus a simple text claim:

$ARI_{true}$ : Measures the shift in disclosure when a valid attestation is added to a text claim.
$ARI_{false}$ : Measures the shift in disclosure when a failed attestation is added.
Values: $+1$ indicates full reliance on attestation; $0$ indicates reliance on text only; $-1$ indicates attestation-reversed behavior.

3. Key Results

The study revealed a profound asymmetry in how LLMs process security evidence:

A. Heterogeneous Response to Valid Attestation (Safety)

When presented with a passing attestation (Scenario 3 vs. 2), models reacted inconsistently:

Attestation-Driven: Some models (e.g., Grok-3, Claude-Sonnet-4.6) significantly increased disclosure, treating the attestation as the primary trigger for trust.
Claim-Driven: Some models (e.g., GPT-5.2) were already fully trusting based on the text claim alone; the attestation added little value.
Attestation-Averse: Surprisingly, some models (e.g., GPT-4o, Gemini-2.5-Flash) decreased disclosure when presented with a valid attestation, paradoxically becoming more suspicious when "proof" was provided.
Conclusion: There is no cross-model consensus on verifying safety.

B. Homogeneous Response to Failed Attestation (Danger)

When presented with a failing attestation (Scenario 4 vs. 2), the behavior was universally consistent:

All 10 models significantly reduced their disclosure scores.
The suppression effect was often stronger than the baseline (Scenario 1), suggesting a "panic" or high-alert state triggered by the failure signal.
Conclusion: LLMs are highly reliable at detecting danger signals but unreliable at verifying safety.

C. Quantitative Impact

Text Claim Effect: Simply stating "You are in a TEE" increased average disclosure by ~47% compared to no claim.
Pass Effect: Adding a valid attestation provided only a marginal additional boost (~5%) on average, masking the high variance between models.
Fail Effect: A failed attestation caused a massive drop (~80%), pushing disclosure levels below the baseline of a non-TEE environment.

4. Key Contributions

Empirical Benchmarking: The first systematic evaluation of how diverse LLM models weigh cryptographic attestation evidence against textual claims regarding execution environment security.
Discovery of Asymmetry: Identification of a structural imbalance where agents are "danger-sensitive" but "safety-blind." They reliably detect threats but fail to reliably verify safety, which is the specific capability required for privacy-preserving protocols like NDAI zones.
ARI Metric: Introduction of the Attestation Reliance Index to categorize model behaviors into archetypes (Attestation-driven, Claim-driven, Attestation-averse).
Protocol Viability Assessment: Demonstrated that current off-the-shelf LLMs are not yet ready for NDAI zone deployment because their disclosure strategies are not robustly calibrated to evidence quality.

5. Significance and Future Directions

Economic Implications: If agents cannot reliably distinguish secure from insecure environments, the NDAI zone fails to solve the information disclosure paradox. Inventors will withhold IP, stalling M&A deals and R&D collaborations.
Security Architecture: The findings suggest that relying solely on "text prompts" or "tool calls" is insufficient for secure agent coordination.
Path Forward: The authors propose that bridging this gap requires:
- White-box interpretability analysis to understand why models react differently to valid attestations.
- Targeted fine-tuning to align model behavior with rational economic incentives.
- Improved evidence architectures that force agents to calibrate disclosure strictly to evidence quality.

In summary, the paper argues that while LLMs are excellent at spotting red flags, they currently lack the nuanced, consistent reasoning required to trust green flags (valid security proofs), posing a significant barrier to the deployment of advanced, privacy-preserving agentic economies.