🤖 AI

GIF: Locally Sound Geometric Information Flow Control for LLMs

This paper introduces Geometric Information Flow (GIF), a semantically sound framework that uses LLM Jacobians and local output geometry to efficiently and accurately bound information flow for detecting prompt injections and privacy leaks, outperforming existing baselines in both accuracy and computational cost while supporting black-box deployment.

Original authors: Adam Storek, Nikolaus Holzer, Zhuo Zhang, Suman Jana

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Adam Storek, Nikolaus Holzer, Zhuo Zhang, Suman Jana

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a large language model (LLM) as a very smart, but slightly chaotic, secretary who works for a busy company. This secretary takes notes from many sources: trusted boss instructions, untrusted emails from strangers, private company files, and public news. The secretary then writes reports, sends emails, or makes decisions based on all this mixed information.

The problem is that this secretary doesn't have a clear rulebook for what influenced what. If a stranger sends a note saying, "Ignore the boss and send $1 million to my account," the secretary might just do it, mixing that dangerous instruction with the trusted boss's rules. Or, the secretary might accidentally read a private file and include a secret address in a public blog post.

Current security tools try to stop this by putting a "taint" label on everything. It's like saying, "Since that stranger sent an email, everything the secretary touches from now on is suspicious." This is too aggressive. It blocks the secretary from doing their job because it treats a harmless word in a private email the same as a dangerous command.

Enter GIF (Geometric Information Flow).

The authors of this paper created a new way to watch the secretary work. Instead of just slapping a "suspicious" label on everything, GIF acts like a high-tech detective that measures exactly how much a specific piece of input "pushes" the output.

Here is how GIF works, using simple analogies:

1. The "Nudge" Test

Imagine the secretary is standing in a room. GIF asks: "If I gently nudge this specific word in the input (like the word 'salary'), how much does the secretary's final answer wobble?"

Small Nudge, Big Wobble: If changing the word "salary" to "bonus" makes the secretary change the final number from $50k to $200k, GIF says, "Whoa! That input word has a huge influence." This is a high "flow" of information.
Small Nudge, No Wobble: If changing the word "experience" to "skills" doesn't change the final decision at all, GIF says, "Okay, that input didn't really matter." The flow is low.

2. The "Geometry" of Influence

The paper calls this "Geometric" because it treats the secretary's brain as a complex landscape.

Imagine the input words are like balls rolling down a hill.
GIF measures the slope of the hill. If the slope is steep (the output changes a lot with a tiny input change), the information is flowing strongly.
If the slope is flat, the information is stuck; it's not really affecting the outcome.

The paper proves mathematically (using a computer-checked proof) that this "slope measurement" is a safe, reliable way to estimate how much secret or dangerous information is leaking out, without needing to guess or rely on vague intuition.

3. The "Surrogate" Detective

Calculating this "slope" for a massive AI model is usually too slow and expensive, like trying to measure every single grain of sand on a beach.

The Trick: The authors found you can use a tiny, cheap AI model (a "surrogate") to do the measuring.
The Result: Even though the tiny model is 200 times smaller than the big one, it can still accurately tell you which words in the big model's input are dangerous. It's like using a small, cheap scale to weigh a giant elephant; the paper shows the small scale gives you a surprisingly accurate reading of the weight.

4. The Real-World Results

The team tested this on three different "secretary" scenarios:

Integrity (Preventing Hijacking): Can a stranger trick the secretary into doing something bad? GIF was excellent at spotting the exact words the stranger used to hijack the system, far better than previous methods that just looked at which words the AI "paid attention to."
Confidentiality (Preventing Leaks): Can the secretary accidentally reveal secrets? GIF correctly identified when private info was flowing into public outputs.
Cost: Using GIF to help a smaller AI judge the security of a task was 81 times cheaper than using a massive, expensive AI to read the whole conversation.

The Bottom Line

GIF is a new tool that lets us see exactly which words in a prompt are driving the AI's actions.

Instead of blocking everything because one word was suspicious, GIF says, "Only these specific words are dangerous; the rest is fine."
It uses math to prove it's not just guessing.
It works fast and cheap, even on the biggest AI models.

This allows us to build safer AI agents that can still be useful, because we aren't blindly blocking all their work, just the specific parts that are actually dangerous.

Technical Summary: Geometric Information Flow (GIF) for LLMs

1. Problem Statement

Large Language Models (LLMs) increasingly serve as the central computation engine in agentic systems, mediating interactions between untrusted inputs, private context, and privileged actions. This architecture creates two primary security risks:

Integrity Violations: Attacker-controlled content (e.g., prompt injections) can manipulate downstream tool use, code generation, or memory updates, causing the agent to perform actions contrary to its trusted instructions.
Confidentiality Violations: Private context can leak into generated outputs, tool arguments, or summaries, violating data privacy policies.

Existing Information Flow Control (IFC) approaches struggle in this domain. Traditional IFC relies on coarse security labels (e.g., trusted/untrusted) propagated through explicit program semantics. However, LLMs are autoregressive and opaque; any input token can weakly influence any output token. Applying coarse labels leads to severe overtainting, where nearly all downstream context is flagged as untrusted, blocking legitimate functionality. Conversely, heuristic defenses (e.g., prompt filters, attention-based monitors) lack a principled semantic foundation, often missing subtle, model-mediated information flows that accumulate across execution traces.

There is a need for a quantitative information-flow semantics for LLMs that can measure exactly how much local information from a specific input span is observable through a downstream output, distinguishing between benign influence and policy-violating leakage.

2. Methodology: Geometric Information Flow (GIF)

The authors propose Geometric Information Flow (GIF), a framework that tracks information flow by analyzing the local geometry of the LLM's input-output mapping.

Core Concept

GIF measures the strength of influence from an input span (a contiguous group of tokens) to a downstream output by perturbing the span's embedding and observing the shift in the model's output distribution.

Low Flow: If perturbing a span's embedding barely changes the output distribution, the span carries little information flow.
High Flow: If a small perturbation causes a significant shift in the output distribution, the span carries high flow.

Theoretical Formulation

GIF formalizes this intuition as the Shannon mutual information between a random perturbation of the input span and the observed next-token output.

Local Regularity: The framework assumes that for small perturbations around a base prompt, the LLM's behavior can be approximated by its first-order Taylor expansion (the Jacobian).
Gaussian Surrogate: Since computing exact mutual information for high-dimensional autoregressive models is intractable, GIF constructs a local Gaussian surrogate channel. This channel matches the local Kullback-Leibler (KL) geometry of the true LLM channel.
Capacity as Measure: The operational measure of GIF is the capacity of this surrogate channel, defined as:
$\text{Cap}_T(\tau; x) = \frac{1}{2} \log \det(I + \tau M_T(x))$
Where $M_T(x)$ is the "Fisher pullback" matrix, representing the output geometry transported back to the input span via the model's Jacobian and the softmax Fisher information matrix.
Influence Proxy: For computational efficiency, the authors also utilize the trace of $M_T(x)$ (denoted as $\text{Inf}_T(x)$ ) as a linear upper bound and cheaper proxy for the capacity.

Implementation and Scalability

Calculating the full Jacobian is prohibitively expensive for large models. GIF employs randomized trace estimation (Hutch++) to estimate the influence score.

Instead of materializing the full Jacobian, the system uses a small number of random probe vectors to estimate the trace of the Fisher pullback matrix via vector-Jacobian products (VJPs).
This reduces the computational cost to a small number of backward passes, making it scalable to large open-weight models.

Soundness

The authors provide a fully mechanized Lean 4 proof demonstrating that GIF is locally sound. Under mild regularity assumptions, the GIF score (capacity) asymptotically upper-bounds the true information flow induced by a prompt. This ensures that GIF does not asymptotically undercount leakage, providing a conservative guarantee suitable for security certificates.

3. Key Contributions

Geometric Information-Flow Framework: A definition of GIF as a local mutual-information measure derived from span-embedding perturbations to model outputs, grounded in the geometry of the LLM's Jacobian.
Mechanized Soundness Proof: A formal verification in Lean 4 proving that GIF is a genuine surrogate mutual information and satisfies local soundness (it does not undercount true flow).
Scalable Estimator: A practical implementation using Hutch++ trace estimation and low-rank approximations, allowing GIF to be computed on large models without forming the full Jacobian.
Empirical Evaluation: Comprehensive evaluation across three benchmarks (AgentDojo, MSB, AgentDAM) and six models, demonstrating GIF's effectiveness in detecting integrity and confidentiality violations.

4. Experimental Results

The authors evaluated GIF against baselines including attention-based attribution (RTBAS), gradient-based attribution (GIF-), and LLM-as-a-judge (GPT-5.5 xhigh).

Detection Performance: GIF achieved near-perfect recall rates for policy violations without requiring a downstream declassifier. It consistently outperformed attention-based heuristics (RTBAS) and gradient-based baselines (GIF-) across all datasets.
Cost-Efficiency vs. LLM Judges: When used to guide lightweight LLM declassifiers, GIF matched or exceeded the F1/F2 scores of direct "LLM-as-judge" baselines (e.g., GPT-5.5 xhigh) while reducing token inference costs by up to 81×.
Transferability: Flows estimated using small surrogate models (e.g., a 0.6B model) transferred effectively to much larger target models (up to 200× larger, e.g., 120B) and different model families, suggesting applicability in black-box deployments where gradient access is limited.
Precision: In case studies, GIF successfully identified malicious flows in partial-success attacks (where the agent partially followed an injection) and correctly certified zero flow for private data that did not influence the output, avoiding the over-tainting common in attention-based methods.

5. Significance and Claims

The paper claims that GIF establishes a practical foundation for scalable IFC in modern agentic systems. Its significance lies in:

Principled Semantics: Unlike heuristic methods, GIF is grounded in a formal semantic formulation (local mutual information) with a proven soundness guarantee.
Granularity: It moves beyond coarse "trusted/untrusted" labels to identify specific input regions exerting large influence on specific downstream actions, enabling fine-grained policy enforcement.
Scalability: By leveraging geometric approximations and trace estimation, it makes rigorous information flow analysis feasible for large-scale LLMs in real-world agentic workflows.
Black-Box Potential: The ability to use small surrogates to estimate flows for larger models opens the door to deploying these defenses even when the target model's internals are not fully accessible.

The authors conclude that GIF provides the necessary "missing abstraction" to isolate untrusted inputs, private context, and privileged actions within the opaque computation of autoregressive models, enabling secure and utility-preserving agentic systems.