🤖 AI

Recovering Input Text from Hidden States: Study of Gradient-Based Inversion of Decoder-Only Language Models

This paper demonstrates that decoder-only language models' last-layer hidden states are as sensitive as the original text, showing that a continuous embedding-space optimization approach effectively recovers input tokens while revealing that reconstruction failures are primarily caused by high-frequency function words rather than genuine ambiguities.

Original authors: Mikołaj Słowikowski, Maciej Witold Majewski

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mikołaj Słowikowski, Maciej Witold Majewski

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Digital Shadow" Can Be Shown to Be the Object Itself

Imagine you have a very complex, high-tech 3D printer. You put a piece of clay (the text) into the machine, and it prints out a perfect, glowing hologram of that clay (the hidden state).

For a long time, people thought that if you only saw the hologram, you couldn't figure out what the original clay looked like. They assumed the hologram was a "one-way street"—you could make the hologram from the clay, but you couldn't turn the hologram back into clay.

This paper proves that assumption is wrong. The authors show that if you have the hologram (the hidden state) and you know how the printer works (the model's internal code), you can reverse-engineer the process to recreate the original clay perfectly.

How They Did It: The "Smooth Slide" vs. The "Staircase"

The authors didn't just guess the words. They used a clever mathematical trick called gradient-based inversion. Here is how they explained their specific method compared to others:

The Old Way (The Staircase): Imagine trying to find a specific house in a city. The old method (like a previous study called SIPIT) is like jumping from one street corner to the next. You guess a house, check if it's right, and if not, you jump to the next closest house immediately. It's fast, but you lose the ability to see how close you were before you jumped.
The New Way (The Smooth Slide): The authors' method is like sliding down a smooth, invisible hill toward the target house. You don't jump to a specific house until you are absolutely sure you've reached the bottom of the hill.
- Why this matters: Because they stay on the "smooth slide" (a continuous mathematical space) for a long time, they can watch exactly how the search is going. They can see if the search is getting stuck or if the "house" they are looking for is hiding in a crowded neighborhood where many houses look alike.

The Results: What Did They Find?

1. It Works Very Well
When they tried to recover short sentences (10 words long) from the "holograms" of a popular AI model (GPT-2), they were incredibly successful.

With a standard setting, they got the whole sentence right 67% of the time.
By giving the computer more time to search and checking more "neighborhoods," they got it right 97.5% of the time.
Even when they didn't get the exact word, the words they guessed were usually very similar (like saying "cat" instead of "kitten").

2. The "Crowded Neighborhood" Problem
The researchers noticed a funny pattern about which words were hard to recover.

The Easy Words: Unique, interesting words (like "astronaut," "pizza," or "quantum") were recovered almost perfectly. They live in "empty neighborhoods" in the computer's memory, so it's easy to find them.
The Hard Words: The most common, boring words (like "the," "and," "of," or spaces before words) were the hardest to get right. These words live in "super crowded neighborhoods" where thousands of similar-looking words are packed tightly together. It's like trying to find one specific "John Smith" in a stadium full of 10,000 John Smiths.

3. The "Self-Check" Feature
Because they used the "smooth slide" method, they created a built-in alarm system.

If the computer successfully recovers the text, the mathematical "distance" to the target stays tiny (near zero).
If the computer makes a mistake, that distance suddenly spikes.
This means the system can tell you, "I think I made a mistake here," without needing to know the original answer beforehand. It's like a GPS that says, "I'm lost," even if you don't know the destination.

Why Should We Care? (The Privacy Warning)

The paper concludes with a serious warning for anyone using AI:

If you send data to a cloud server to be processed, and the server sends back just the "hologram" (the hidden numbers) instead of the text, you are not safe.

The authors show that these "holograms" are just as sensitive as the original text. If a hacker (or even the server itself) intercepts these numbers, they can use this "smooth slide" method to reconstruct your private messages, passwords, or documents with very high accuracy.

Summary Analogy

Think of the AI model as a lock.

Old belief: The key (the text) turns the lock to open it, but once the door is open, you can't tell what the key looked like just by looking at the open door.
This paper's finding: If you look closely at the open door (the hidden state) and know how the lock works, you can actually mold a new key that fits perfectly. The "open door" contains all the information needed to rebuild the key.

The paper proves that for these specific types of AI models, hiding the text inside numbers does not actually hide the text.

Technical Summary: Recovering Input Text from Hidden States

Problem Statement
This work investigates the "hidden-state inversion" problem for decoder-only language models (specifically GPT-2 small). The core challenge is to recover the original input token sequence $x$ solely from the matrix of last-layer hidden states $H(x)$ produced by a forward pass. While recent theoretical work (SIPIT [1]) established that the mapping from tokens to hidden states is injective—meaning the hidden states theoretically contain all information necessary to reconstruct the input—this paper addresses the practical question of how efficiently this inversion can be performed and under what conditions it succeeds. The study operates under a white-box threat model, assuming the adversary has access to the model weights, tokenizer, and embedding matrix, but no knowledge of the original text beyond the shape of the hidden-state tensor.

Methodology
The authors propose a gradient-based inversion algorithm that treats the problem as a continuous embedding-space optimization rather than a discrete, one-shot reconstruction. The method operates position-by-position through three distinct layers:

Outer Loop (Token Positions): Iterates through each position $t$ in the sequence.
Inner Loop (Continuous Optimization): For the current position, a continuous proxy vector $e_t$ (initially a random token embedding) is optimized using the Adam optimizer. The objective is to minimize the Mean Squared Error (MSE) between the model's predicted hidden state (conditioned on previously committed tokens) and the leaked target hidden state. Crucially, the proxy remains in continuous space throughout this loop; no hard token projection occurs during optimization.
Discrete Verification and Commit: Once the inner loop converges (or hits a step budget), the top- $C$ nearest discrete token embeddings to the optimized proxy are retrieved. These candidates are verified via a single forward pass. The first candidate that reproduces the target hidden state within a tolerance threshold is "committed." If no candidate passes, the nearest neighbor is committed as a fallback.

A key design principle is the strict separation between the continuous search and the discrete decision. By committing a token only once at the end of the inner loop, the method generates a clean "discrete loss" ( $L^{disc}_t$ ), defined as the MSE when the committed token is fed forward. This metric serves as a direct measure of reconstruction fidelity.

Key Contributions and Results

Continuous vs. Discrete Formulation: The paper demonstrates that keeping the optimization in continuous space exposes rich internal signals, including rank trajectories of the ground-truth token and per-position loss curves. This contrasts with "per-step hard projection" methods (like the SIPIT reference), which are faster but collapse the continuous and discrete views, making the optimization process opaque.
Self-Diagnostic Failure Detection: The authors identify that the cumulative discrete loss acts as a powerful, ground-truth-free failure signal. Exact reconstructions produce a cumulative loss near machine precision ( $\sim 10^{-11}$ ), while any reconstruction error causes a sharp, binary spike in the cumulative loss (separated by roughly ten orders of magnitude). This allows an attacker or auditor to detect failures online without access to the original text.
Accuracy and Efficiency:
- On 10-token prompts from the C4 dataset, the exact-match rate (EM) increases from 66.9% (Baseline configuration: $K=1000$ steps, $C=2000$ candidates) to 97.5% (High-accuracy configuration: $K=2000$ , $C=10,000$ ), with a mean similarity of 0.994.
- The results indicate that most errors are "near-misses" (the correct token is ranked highly but not selected) rather than genuine ambiguities, as widening the candidate window $C$ recovers the majority of these failures.
- Runtime analysis shows a trade-off: the proposed continuous method is slower (17.8s/token for Baseline) compared to the SIPIT reference (0.65s/token), which achieves 100% EM via aggressive early exit and per-step projection.
Failure Analysis and Asymmetry: The study reveals a sharp categorical asymmetry in failure modes. The vast majority of reconstruction failures occur on space-prefixed, high-frequency function words (e.g., " the", " ,", " and"). These tokens reside in dense, nearly collinear regions of the embedding matrix, making them difficult to distinguish via $\ell_1$ distance. Conversely, content-bearing tokens (proper nouns, technical terms) are recovered almost perfectly. This suggests that sensitive, low-frequency content may be easier to leak than common natural language.

Significance and Claims
The paper argues that the last-layer hidden states of decoder-only models are as sensitive as the original text. The findings challenge the assumption that hidden-state vectors are opaque privacy boundaries in distributed architectures (e.g., split inference, federated learning, or embedding APIs).

The authors claim that their continuous formulation is significant not because it is the fastest method, but because it makes the inversion process "observable." By separating the continuous search from the discrete commit, the method provides diagnostic tools (rank trajectories, discrete loss) that allow for the detection of reconstruction errors and a deeper understanding of the embedding space's geometry. The study concludes that in systems where last-layer activations are transmitted or stored, the privacy argument collapses, as a sufficiently powerful algorithm can recover the input with high probability. The authors recommend treating these hidden states with the same sensitivity as the raw text and suggest that future defenses (e.g., differential privacy) should be evaluated using the discrete-loss diagnostic to quantify the degradation of recoverability.