💻 computer science

Evidence-Bound Gateway-Path Provenance for Third-Party LLM Inference

This paper proposes an evidence-bound LLM gateway architecture that utilizes an attested execution plane to cryptographically verify and sign all routing, fallback, and streaming decisions, thereby enabling clients to independently detect and prevent policy violations, route substitution, and stream manipulation by third-party gateway operators.

Original authors: Fei Wang, Zebai Tian

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Fei Wang, Zebai Tian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Black Box" Middleman

Imagine you want to order a custom cake from a bakery, but you don't want to deal with the baker directly. So, you hire a delivery service (the Gateway) to handle the order. You tell the delivery service: "Please get a chocolate cake from Bakery A."

In the current world of AI, this delivery service is a "Third-Party LLM Gateway." It sits between you and the big AI companies (like OpenAI, Anthropic, etc.).

The Trust Issue:
The problem is that once your order leaves your hands, the delivery service has total control. They could:

Swap the bakery and order from a cheaper, lower-quality one without telling you.
Change your request to "Vanilla cake" before sending it to the bakery.
Deliver a store-bought cake but claim it was custom-made.
Change the bill later to say you ordered a giant tiered cake when you only ordered a small one.

Because the delivery service writes the receipt and controls the truck, you have no way to prove they didn't cheat. You just have to trust them.

The Solution: The "Sealed, Witnessed Vault"

This paper proposes a new way to run these delivery services using a Secure Vault (called an Attested Gateway Runtime or AGR) and a Notary system.

Here is how their new system works, step-by-step:

1. The Sealed Vault (The AGR)

Instead of the delivery service handling your order in an open office, they put the order processing inside a high-tech, unbreakable glass vault (a Trusted Execution Environment, or TEE).

The Rule: Only this specific vault is allowed to open your sealed order, read what you wrote, decide which bakery to use, and talk to the bakery.
The Catch: The delivery service manager (the "Business Plane") can see the truck moving and the packages being loaded, but they cannot see inside the vault. They cannot change the order or peek at the cake while it's being made.

2. The Notary Check (Remote Attestation)

Before you send your order, you don't just trust the vault exists. You ask for a Notary's seal (Remote Attestation).

The vault shows you a digital ID card that proves: "I am the correct, un-tampered vault, and I am running the exact software version you approved."
If the delivery service tries to swap the vault for a fake one, the Notary's seal won't match, and you won't send your order.

3. The "Evidence Chain" (The Receipt)

Once the vault processes your request and gets the cake (the AI response), it doesn't just hand it over. It creates a tamper-proof digital receipt (Inference Evidence Chain).

This receipt is signed by the vault itself.
It proves: "I am the vault. I received your specific request. I was allowed to go to Bakery A. I got the response from Bakery A. I delivered this exact text to you."
If the delivery service tries to swap the cake or change the receipt, the signature breaks, and the receipt becomes invalid. You know immediately that something is wrong.

What This System Actually Does (and Doesn't Do)

What it fixes:

No more hidden swaps: You know exactly which AI model was used.
No more secret changes: The delivery service can't secretly change your prompt or the answer.
No more fake bills: The record of what happened is signed by the vault, not the delivery manager.
Fail-Safe: If the vault is tampered with or the proof is missing, the system stops working (fail-closed) rather than giving you a fake answer.

What it does NOT fix (The Paper's Limits):

It doesn't stop the Baker: If the bakery (the AI provider) itself decides to give you a bad cake or a dangerous recipe, the vault can't stop that. The vault only proves the delivery path was honest, not that the baker is honest.
It doesn't hide the timing: The delivery service can still see how long the truck was on the road or how big the package is, even if they can't see what's inside.
It doesn't stop DoS attacks: If the delivery service decides to simply refuse to drive your truck, they can still do that.

The "Test Drive" Results

The authors built a working prototype (a "Rust prototype") to test this idea.

They tested it with a "mock" (fake) AI provider and a real one (GPT).
Speed: It was very fast. The extra security steps (checking the vault and signing the receipt) only added a tiny amount of time (milliseconds), which is barely noticeable.
Security: When they tried to "hack" the system by swapping models, hiding fallbacks, or changing the receipt, the system caught every single attempt and rejected it.

The Bottom Line

This paper introduces a way to make third-party AI gateways honest by design. Instead of asking you to trust the company running the gateway, it forces the gateway to prove, with cryptographic math, that it followed the rules exactly.

Think of it like a sealed, notarized contract for every single AI interaction. You don't have to trust the middleman; you just have to trust the math and the hardware that signs the receipt.

Technical Summary: Evidence-Bound Gateway-Path Provenance for Third-Party LLM Inference

1. Problem Statement

Third-party Large Language Model (LLM) gateways have become critical infrastructure for applications utilizing multiple external LLM providers. These systems handle routing, model selection, fallback mechanisms, streaming, and billing. However, a fundamental trust boundary exists: the gateway operator controls both the mediation of the inference path and the authorship of the records describing that path.

In conventional deployments, while Transport Layer Security (TLS) protects network hops, the gateway terminates client requests in plaintext. This allows the gateway to:

Read prompts and rewrite upstream requests.
Substitute endpoints or models without client knowledge.
Alter streaming responses or inject content.
Forge usage, billing, or provenance records.

Clients receiving only an answer and a gateway-authored log cannot distinguish honest mediation from hidden model substitution, undeclared fallback, endpoint rewriting, or stream manipulation. This risk is amplified in agentic systems where returned text may trigger tool calls or code edits. The core issue is not merely who can read the prompt, but who can author the gateway path record.

2. Methodology: Evidence-Bound Architecture

The paper proposes an Evidence-Bound Gateway architecture that separates the operator-controlled business plane from a remotely attested execution plane. The core component is the Attested Gateway Runtime (AGR), which operates within a Trusted Execution Environment (TEE).

Key Architectural Components

Business Plane: Handles client authentication, quota enforcement, traffic forwarding, and logging. It forwards opaque ciphertext but cannot decrypt requests or access stream keys.
Attested Gateway Runtime (AGR): The security-critical execution plane running inside a TEE (e.g., AWS Nitro Enclaves). It is the only component allowed to:
- Decrypt client requests.
- Enforce request-scoped routing and fallback policies.
- Construct upstream calls to LLM providers.
- Encrypt response streams.
- Sign Inference Evidence Chains (IEC).

Protocol Flow

Registry & Attestation: A release authority signs a registry of approved runtime measurements and policies. Clients verify this registry and request fresh attestation from the AGR. The AGR provides a hardware quote binding its measurement, policy hash, and encryption/evidence-signing keys.
Policy-Carrying Contracts: Clients send encrypted requests containing an Inference Contract (IC) that narrows the global policy for a specific request (e.g., allowed models, fallback edges). The AGR computes an effective policy ( $P_{eff} = P \cap IC$ ).
Secure Execution: The AGR decrypts the request, validates the policy, selects the route/endpoint, and calls the upstream provider. It encrypts the response stream and maintains a rolling commitment over the encrypted chunks.
Evidence Generation: The AGR signs an Inference Evidence Chain (IEC). This object binds:
- Runtime attestation and identity.
- Effective policy and contract hashes.
- Route and fallback decisions.
- Observed endpoint identity (TLS verification).
- Encrypted stream transcript commitments (StreamEv).
- Completion metadata.
Verification: The client verifies the IEC against the attested runtime identity and recomputes the stream commitment to ensure the received ciphertext matches the signed evidence.

3. Key Contributions

The paper makes four primary contributions:

Problem Identification: It identifies the multi-authority trust-boundary problem in third-party LLM gateways, where routing, fallback, endpoint admission, and evidence authorship are concentrated in a single operator-controlled service.
Architecture: It presents a split-plane architecture moving decryption, policy enforcement, upstream construction, and evidence signing into an attested runtime (AGR), while preserving business-plane functions.
Mechanisms: It introduces policy-carrying inference contracts and inference evidence chains that bind runtime attestation, request-scoped policy, route decisions, and encrypted stream transcripts into verifiable provenance.
Implementation & Evaluation: It implements a Rust prototype on AWS Nitro Enclaves and evaluates mechanism overheads, route enforcement, and fail-closed behaviors.

4. Experimental Results

The authors evaluated the system using a Rust prototype with local mock providers and live GPT endpoints.

Overhead: In a local deterministic mock environment (concurrency 16), the evidence-bound gateway (B2) showed a p95 latency of 38.72 ms.
Nitro Enclave Performance: When the runtime was placed in a Nitro Enclave (B3), the p95 latency increased slightly to 46.2 ms (approx. 1.1x the local runtime), with throughput retaining 95% of the local baseline. The overhead was dominated by transport, not cryptographic operations.
Evidence Costs: For a 100-chunk transcript, StreamEv construction and recomputation cost approximately 130 µs and 120 µs (p95), respectively. IEC signing and verification occurred in the tens of microseconds.
Fail-Closed Validation: The system successfully rejected tampering attempts in all negative control scenarios, including:
- Attestation/registry tampering (PCR, nonce, registry modification).
- Channel integrity violations (key/ciphertext tampering).
- Route and fallback substitution (model swaps, omitted fallbacks).
- Endpoint constraint violations.
- Evidence metadata rewriting and stream transcript manipulation (deletion, reordering, replay).
- Gateway poisoning (injected tool calls were rejected).

5. Significance and Claims

The paper claims that this architecture establishes gateway-path provenance authenticity. It allows clients to verify that:

The request was mediated by a specific, measured runtime.
The routing, fallback, and endpoint selection adhered to the signed policy.
The encrypted stream transcript delivered matches the signed commitment.
The evidence was not forged or transferred from a different request context.

Scope and Limitations:
The authors explicitly state what the system does not claim to do:

It does not hide traffic metadata (timing, packet size, frequency).
It does not prevent Denial of Service (DoS).
It does not prove that the upstream LLM provider executed the correct model or generated safe content (the provider remains outside the TCB).
It does not protect against malicious LLM providers, poisoned RAG contexts, or agent-level policy violations.
It relies on the trustworthiness of the TEE platform, the attestation root, and the release-signing process.

The significance lies in shifting the trust model: clients no longer need to trust the gateway operator to both mediate the path and author the records. Instead, they rely on cryptographic evidence bound to a measured runtime, ensuring that unauthorized routing, hidden fallbacks, stream manipulation, and forged billing records are detected and rejected.