🤖 machine learning

The Reservoir Attention Network: Cross-Pass State in Pretrained Transformers via Content-Addressable Reservoir Injection

This paper investigates the feasibility of the Reservoir Attention Network (RAN), an architecture that injects a fixed, untrained random reservoir into pretrained transformers to carry cross-pass state, demonstrating through minimal probes on models ranging from GPT-2 to Qwen2.5 that untrained recurrent dynamics alone can suffice for maintaining usable state across forward passes.

Original authors: Emma Leonhart

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Emma Leonhart

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Problem: Stateless Transformers

Standard large language models (like GPT-2 or Qwen) are stateless between separate interactions. When you send a prompt, the model processes it and generates a response, but once that process finishes, its internal memory is wiped clean. It has no record of previous conversations unless you explicitly paste that history into the current context window. The authors wanted to give the model a persistent memory that survives between these separate "forward passes" without retraining the entire neural network.

The Solution: A Fixed Reservoir

To solve this, the authors added a component called a Reservoir. In computer science, this refers to a specific type of recurrent neural network (often called an Echo State Network).

What it is: It is a pool of artificial neurons with random, fixed connections. Crucially, the weights connecting these neurons are never trained. They are set randomly at the start and left alone.
How it works: Because the connections are recurrent (loops), the reservoir maintains an internal state vector. When new information enters, it changes this state. The state doesn’t reset immediately; it holds a fading "echo" of recent inputs. This allows information to persist over time, even after the main model has finished processing a specific sentence.

The Mechanism: Injecting Memory into a Frozen Model

The authors did not train the main transformer model. Instead, they kept the pretrained transformer frozen (unchanged) and inserted the reservoir at a mid-depth layer of the network.

Reading: At this specific layer, the reservoir reads the transformer’s attention outputs. It uses a fixed, random projection to convert the transformer’s data into a format the reservoir can process.
Updating: The reservoir updates its internal state vector based on this input.
Writing: The reservoir then sends its updated state back into the transformer.
Training: Only a small "readout" layer (which interprets the reservoir’s state) and lightweight adapters (LoRA) are trained. The massive transformer backbone remains untouched.

The Key Discovery: How to Inject the State

The most critical finding of the paper is how the reservoir’s state is fed back into the transformer. The authors tested two methods with drastically different results:

1. Additive Injection (Failed)

Method: The reservoir’s state vector was simply added to the transformer’s internal activations (like adding noise to a signal).
Result: This failed. The transformer learned to treat the reservoir’s input as irrelevant noise and ignored it. The model could not recall information from previous passes.

2. Content-Addressable Injection (Succeeded)

Method: The reservoir’s state was converted into pseudo-tokens (fake words) and placed at the beginning of the input sequence as "key/value" pairs. The transformer’s attention mechanism was forced to attend to these tokens just like it attends to real words.
Result: This worked perfectly. Because the attention mechanism explicitly looked at these tokens, the model successfully recalled information from previous conversations that were not present in the current text.

Tuning the Dynamics

For the reservoir to work, its internal behavior had to be carefully tuned:

Edge of Chaos: The random connections in the reservoir must be balanced. If they are too stable, the memory fades instantly. If they are too chaotic, the signal becomes noise. The reservoir must sit at the "edge of chaos" to maintain useful, fading echoes of input.
Input Scaling: The signal coming from the transformer into the reservoir was too strong, causing the reservoir’s neurons to saturate (max out). The authors had to scale down this input signal to 10–25% of its original strength to keep the reservoir functioning correctly.

Scaling Results

The effectiveness of this method depended heavily on the size of the model and the reservoir:

GPT-2 Small: Worked well with a standard-sized reservoir.
GPT-2 Medium: Failed. The model could not effectively utilize the memory injection.
Qwen-1.5B (Large): Worked again, but only when the reservoir was significantly enlarged (e.g., 2048 neurons instead of 512) and the input scaling was applied. This suggests that larger models require larger reservoirs to handle their more complex internal representations.

Capabilities and Limits

This is a proof-of-concept study, not a finished product.

What it CAN do: It can carry simple state across conversations, such as remembering a specific trigger word or counting how many times it has been called.
What it CANNOT do: It cannot yet remember complex stories, long lists of facts, or perform multi-step reasoning across sessions. There is a hard capacity ceiling: the reservoir can reliably hold about 20–48 items before performance degrades.
Agent Tasks: In tests where the AI had to "wait" or "stay silent," it succeeded, but often by using standard contextual tricks rather than true long-term memory. The reservoir’s true value was proven only when recalling specific tokens absent from the current input.

Safety and Monitoring

The persistent state offers potential safety benefits:

Inspection: Since the reservoir holds a running state of recent interactions, an operator could theoretically inspect this state to monitor the model’s "mental" status or detect agitation before it generates output.
Immediate Interruption: Because the model continuously attends to the reservoir’s state (via the pseudo-tokens), it can react to a stop signal immediately, rather than waiting to finish its current generation cycle.

Summary

The paper demonstrates that you can add persistent memory to a frozen transformer by inserting a fixed, random recurrent network (reservoir) at a mid-layer. The success depends entirely on how the memory is injected: adding it as noise fails, but injecting it as attendable pseudo-tokens succeeds. While currently limited to simple state retention (like remembering a word or count), this approach proves that stateless models can be upgraded to remember across sessions without full retraining.

Technical Summary: The Reservoir Attention Network (RAN)

Problem Statement

Standard transformer architectures are stateless across forward passes; they possess no endogenous variable that evolves between calls, relying solely on position within the context window for temporal awareness. This study investigates whether a fixed, randomly-initialized reservoir (in the Echo State Network sense) injected into the mid-layer attention of a pretrained transformer can endow the model with usable, persistent state between independent forward passes without retraining the backbone. The core research question is whether such a mechanism can carry state across passes to enable tasks structurally impossible for stateless models, and under what injection and dynamic regimes this state becomes a usable signal rather than noise.

Methodology

The Reservoir Attention Network (RAN) architecture grafts a fixed-random recurrent substrate onto a pretrained transformer.

Architecture: A reservoir of size $N$ with fixed random weights ( $W_r$ , $W_{in}$ ) and a leak rate $a$ is injected at a mid-depth layer $L_k$ .
Mechanism: The reservoir reads the layer's attention output via a fixed random projection $W_{in}$ , updates its state $r(t)$ , and writes the state back as content-addressable key/value (KV) prefix pseudo-tokens via a trained readout $W_{out}$ .
Training: The backbone transformer is frozen. Only the readout $W_{out}$ and light LoRA adapters on upper layers are trained. The reservoir itself remains untrained and fixed-random.
Experimental Scope: Experiments were conducted on GPT-2 (124M, 355M) and Qwen2.5 (0.5B, 1.5B) on a single consumer GPU. Tasks were minimal probes designed to isolate specific mechanisms (e.g., secret-word recall, silence policies) rather than demonstrate complex agentic reasoning.

Key Contributions and Results

1. Injection Design is Decisive

The study establishes that the method of injecting the reservoir state determines whether the carried state is usable.

Additive Injection Failure: Writing the reservoir state additively into the residual stream results in the model learning to ignore the recurrent state. Cross-context recall remains at chance levels (0.17), indistinguishable from a state-reset baseline.
Content-Addressable (KV-Prefix) Success: Re-injecting the state as KV-prefix pseudo-tokens that upper layers attend to yields 100% cross-context recall (1.00 vs. 0.17 control) on GPT-2-small. This proves that the reservoir state can carry usable information if the model is given a mechanism to attend to it.

2. Reservoir Dynamics and Scaling

Edge-of-Chaos Regime: The reservoir operates effectively in the edge-of-chaos regime (spectral radius $\rho \approx 1$ ). However, real transformer activations over-drive a unit-scaled reservoir, causing saturation.
Input Scaling: To avoid saturation, the input scaling from the transformer to the reservoir must be tuned down to approximately one-quarter to one-tenth of unit scale.
Scaling to 1.5B: The previously observed "GPT-2-small-only" boundary was an artifact of an undersized reservoir and mismatched input scaling. By enlarging the reservoir to 2048 nodes and matching the input scaling (0.1 for Qwen-1.5B), cross-pass recall recovers to 0.83–1.00 across the Qwen family.
Model Specificity: The success is not a simple function of parameter count. GPT-2-medium (355M) failed across a seven-point scaling sweep, while Qwen-0.5B succeeded with different scaling (0.5) than Qwen-1.5B (0.1). Input scaling matched to the model is necessary but not sufficient; the ability of a backbone to learn to read the prefix remains model-specific.

3. Capacity and Limits

Capacity Ceiling: The reservoir demonstrates a capacity ceiling of tens of items. Recall is perfect for ~6 secret words, degrades to ~0.42 at 24 words, and falls to chance by 48 words.
N-Seed Selection: The study found that at current training budgets, selecting a "better" fixed reservoir seed via untrained metrics (e.g., participation ratio) is ineffective. Outcomes are dominated by training noise and initialization variance rather than inherent reservoir quality.

4. Stateful vs. Stateless Behaviors (The Battery)

An eight-task "battery" separating temporal/agency metrics from symbolic content recall revealed a critical split:

Temporal/Agency Metrics: Metrics like "silence," "timed," and "self-initiation" were matched by a stateless ablation (LoRA adapters on current-pass features only). These behaviors did not rely on the carried reservoir state.
Content Recall: Symbolic content recall (recalling specific tokens) is the true test of statefulness. While initially failing at scale, it can be retained in the battery only under specific conditions: denying the stateless shortcut its adapter capacity (small LoRA rank) and decaying the learning rate. Without these, the optimizer finds a stateless shortcut, and the reservoir-driven solution collapses.

5. Safety and Monitoring Implications

The paper proposes that the fixed reservoir offers a low-cost, stable monitoring surface:

Interruptibility: A per-tick agent with a reservoir registers "STOP" signals with zero latency and retains them for several passes via fading memory, unlike turn-based agents.
Monitorability: A linear readout can recover internal process variables (e.g., elapsed time) from the reservoir state with high fidelity ( $R^2 \approx 0.995$ ) without sparse autoencoders. This mapping is resilient to moderate fine-tuning drift because the reservoir weights remain fixed.

Significance and Claims

The paper explicitly frames its contributions as a feasibility and dynamics study at small scale, avoiding claims of general intelligence or complex agentic reasoning.

Core Finding: The primary contribution is the injection-design finding: that a fixed, untrained reservoir can carry usable cross-pass state in a pretrained transformer, provided it is injected as content-addressable KV-prefix tokens and the reservoir is sized and scaled correctly.
Negative Results as Boundaries: The paper emphasizes controlled negatives, such as the failure of GPT-2-medium and the stateless nature of temporal metrics in the battery, to bound the contribution.
Theoretical Stance: The authors state that while cross-pass state is motivated by complexity-theoretic arguments (lifting the TC0/FO(M) bound), they do not claim to have proven such a lift. The work treats the finite-precision reservoir as an open theoretical question.
Future Work: The "always-alive agent" vision is treated as compute-limited future work. The current study isolates whether untrained recurrent dynamics suffice to carry state, leaving trained recurrence as a complementary, more expensive direction.

In summary, the paper demonstrates that how the reservoir is injected is the deciding factor for usability, characterizes the necessary dynamics (scaling, input scaling) for larger models, and provides a recipe for retaining stateful content recall in multi-task settings, while acknowledging significant limitations in capacity and model-specific transferability.