🤖 machine learning

Echo: KV-Cache-Free Associative Recall with Spectral Koopman Operators

Echo is a KV-cache-free architecture that integrates Spectral Koopman Attention into State-Space Models to achieve constant-memory, perfect associative recall over long sequences, effectively overcoming the retrieval limitations of pure SSMs and the memory bottlenecks of traditional Transformers.

Original authors: Anupama Sridhar, Alexander Johansen

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anupama Sridhar, Alexander Johansen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Memory Cliff"

Imagine you are reading a very long story, maybe 10,000 pages long. You need to remember a specific fact mentioned on page 10 to answer a question on page 10,000.

The Old Way (Transformers): To remember that fact, the computer keeps a giant "notebook" (called a KV Cache) where it writes down every single word it has read so far. As the story gets longer, this notebook gets huge. Eventually, it runs out of space on the computer's hard drive, causing a crash. This is the Memory Bottleneck.
The "Efficient" Way (State Space Models like Mamba): To save space, these models don't keep a notebook. Instead, they try to compress the whole story into a tiny, fixed-size "mental summary" (a Recurrent State). They update this summary as they read.
- The Problem: This summary is a bit like a fading echo. If you hear a whisper on page 10, by the time you get to page 10,000, the echo has faded so much that you can't hear it anymore. The paper calls this the "Memory Cliff." If the gap between the fact and the question is too big, the model forgets everything and guesses randomly.

The Solution: Echo

The authors created a new model called Echo. It combines the best of both worlds: it keeps the tiny, space-saving "mental summary" but adds a special tool to retrieve specific facts without needing a giant notebook.

Think of Echo as a Librarian with a Magic Index Card System.

1. The Magic Index Card (Spectral Koopman Attention)

Instead of writing down every word in a notebook (which takes too much space), Echo writes down a few summary statistics on a small, fixed-size index card.

How it works: As the model reads, it doesn't store the words. Instead, it updates a few numbers on the card that represent "how often this word appeared" and "what usually comes after it."
The Magic: Because these numbers are just simple additions (like adding 1 to a score), the card never gets bigger, no matter how long the story is. It fits in your pocket (constant memory).

2. The Spectral Filter (The "Echo" Effect)

Here is where the "Spectral Koopman" part comes in. The authors realized that some information is "loud" and persistent (like a drum beat), while other information is "quiet" and fades away quickly (like a whisper).

The Problem: In a normal summary, the quiet facts get drowned out by the noise.
The Fix: Echo uses a mathematical "filter" (like a noise-canceling headphone for data). It looks at the patterns on the index card and asks: "Is this fact a persistent drum beat, or just a fleeting whisper?"
The Result: It amplifies the persistent facts (the ones you need to remember) and suppresses the noise. This allows it to retrieve a fact from page 10 even when you are on page 10,000, without ever having written page 10 down in a notebook.

3. No More "Guessing"

The paper tested this on a game called "Needle in a Haystack."

The Game: Hide a specific sentence (the needle) in a massive block of text (the haystack). Ask the model to find it.
The Results:
- Pure Mamba (The Fading Echo): Got it wrong almost 100% of the time when the text was long. It hit the "Memory Cliff."
- Standard Attention (The Giant Notebook): Got it right, but needed a massive amount of computer memory to do so.
- Echo (The Magic Index): Got it 100% correct, even with the longest texts, while using a tiny, fixed amount of memory. It didn't need a notebook; it just needed its index card.

Why This Matters (According to the Paper)

The paper claims that Echo proves you don't have to choose between being efficient (low memory) and being smart (good memory).

Efficiency: It uses the same tiny amount of memory whether the text is 100 words or 100,000 words.
Accuracy: It solves the "Memory Cliff" problem that plagues other efficient models.
Speed: It calculates the answer using a direct mathematical formula (like solving a math equation) rather than trying to "guess" the answer through trial and error (which is how standard attention models often work).

Summary Analogy

Imagine you are trying to remember a phone number from a long conversation.

Standard AI: Writes the whole conversation down in a book. It can find the number, but the book is heavy and hard to carry.
Mamba (Old Efficient AI): Tries to remember the conversation in its head. By the end, it's forgotten the number because the memory faded.
Echo: Keeps a tiny, fixed-size notepad. Instead of writing the whole conversation, it writes down a "code" for the number. When asked, it uses a special decoder (the Spectral Filter) to turn that code back into the number instantly, no matter how long the conversation was.

The paper concludes that this "Echo" architecture allows AI agents to handle very long tasks (like complex tool use or long reasoning chains) without running out of memory, solving a major bottleneck in current AI technology.

Technical Summary: Echo: KV-Cache-Free Associative Recall with Spectral Koopman Operators

1. Problem Statement

Long-context reasoning and agentic tool-calling generate traces spanning tens of thousands of tokens. While Transformer-based architectures rely on Key-Value (KV) caches to maintain content-addressed retrieval, these caches grow linearly with sequence length, creating a memory bottleneck on commodity hardware. Conversely, State Space Models (SSMs) like Mamba offer linear-time inference and constant memory per step by compressing the sequence into a fixed-size recurrent state. However, this compression is lossy: SSMs suffer from a "memory cliff," where retrieval accuracy collapses exponentially once the gap between a stored fact and its query exceeds the effective horizon of the recurrent state.

Current remedies involve hybrid architectures that interleave SSM layers with causal attention blocks. While this restores retrieval capabilities, it reintroduces the linear memory growth of the KV cache. The paper posits that content-addressed retrieval can be recast as kernel ridge regression, where sufficient statistics are accumulated as running sums in fixed memory, thereby eliminating the KV cache entirely without sacrificing retrieval performance.

2. Methodology: Echo and Spectral Koopman Attention (SKA)

The authors introduce Echo, a KV-cache-free architecture built around Spectral Koopman Attention (SKA), a drop-in replacement for standard attention layers. Echo retains SSMs (specifically Mamba-2) for local sequential processing but replaces global retrieval mechanisms with SKA.

2.1 Core Mechanism: Spectral Koopman Attention

SKA estimates a compact linear dynamical model from streaming sufficient statistics to retrieve values at query positions without materializing a $T \times T$ attention matrix or maintaining a KV cache.

Sufficient Statistic Accumulation: SKA accumulates three additive statistics in $O(r^2 + rP)$ memory (where $r$ is a small projection rank and $P$ is the head dimension):
1. A regularized Gram matrix $G = \sum \hat{z}_t \hat{z}_t^\top + \epsilon I$ .
2. A cross-temporal covariance $M = \sum \hat{z}_t \hat{z}_{t-1}^\top$ .
3. A value-key covariance $C_v = \sum v_t \hat{z}_t^\top$ .
  These statistics are exact sufficient statistics for the retrieval computation, meaning longer sequences improve the conditioning of the estimates rather than degrading them.
Operator Estimation and Retrieval:
- The system fits a whitened Koopman operator $A_w = L^{-1} M L^{-\top}$ (where $L$ is the Cholesky factor of the regularized Gram matrix) via ridge regression.
- The value readout is computed as $B_v = C_v \tilde{G}^{-1}$ , which is the unique minimizer of the regularized least-squares objective $\min_B \sum \|v_t - B\hat{z}_t\|^2 + \epsilon \|B\|_F^2$ .
- Unlike linear attention (which corresponds to a single gradient step from zero) or softmax attention (which implicitly approximates this solution via non-convex optimization), SKA computes the closed-form ridge solution directly via Cholesky factorization.
Spectral Filtering:
- The whitened operator $A_w$ is spectrally normalized to bound its spectral radius.
- A power-iterated filter ( $A_w^K$ ) is applied to the whitened query. This filter amplifies persistent eigenmodes (where $|\lambda| \approx 1$ , representing stable bindings) and suppresses transient modes (where $|\lambda| \ll 1$ , representing noise or distractors). This spectral selectivity is a capability not present in standard regression objectives alone.

2.2 Architecture Components

SSM Backbone: The majority of layers use Mamba-2 for local processing.
Spectral Koopman MLP: The standard SwiGLU feedforward layers are replaced with a spectral MLP that applies a block-diagonal rotation (constrained to the unit disk) to the input. This design preserves gradient rank and reduces parameter count by one-third compared to SwiGLU while maintaining norm preservation.
Inference: Echo maintains $O(1)$ memory per layer regardless of context length. The state consists of the accumulators ( $G, M, C_v$ ) and a scalar max norm, totaling approximately 77 KB for a 50M parameter model.

3. Key Contributions

Elimination of the Memory Cliff: The paper demonstrates that the retrieval failure of pure SSMs is a structural consequence of multiplicative recurrence, not a capacity limitation. Echo eliminates this cliff by using a closed-form ridge solver over streaming statistics.
KV-Cache-Free Associative Recall: Echo achieves perfect retrieval on benchmarks where pure SSMs fail and matches hybrid SSM+Attention models, but with constant inference memory.
Theoretical Connection to Regression: The work formalizes the link between content-addressed retrieval and ridge regression, showing that SKA solves this optimization problem explicitly via Cholesky factorization, bypassing the non-convex optimization landscape of attention.
Gradient Flow Preservation: The authors provide a theoretical analysis showing that SKA and the Spectral Koopman MLP introduce zero additional rank compression in the backward pass (unlike softmax attention, which compresses gradients via the entropy-dependent effective rank of the attention matrix). This suggests a mechanism for improved data efficiency.

4. Experimental Results

The authors evaluated Echo across three regimes: synthetic transfer tasks, Multi-Query Associative Recall (MQAR), and language modeling.

Synthetic Transfer (Sub-Million Scale): On five held-out retrieval tasks (including system prompt recall, tool trace, and needle-in-a-haystack), a 982K parameter Echo model (SSM+SKA) outperformed both a pure SSM (1.11M params) and an SSM+Attention hybrid (998K params). Echo achieved mean accuracies of 81.1% compared to 54.4% for pure SSM and 76.2% for the hybrid. Crucially, Echo maintained high accuracy (65% at 4,096 tokens) on length generalization tasks where baselines collapsed to near-chance levels.
MQAR (50M Scale): On the MQAR benchmark with distractor gaps up to 4,096 tokens and 32 KV pairs, a pure Mamba-2 scored ~3% (chance). The Echo model (Mamba-2+SKA) achieved 100% retrieval accuracy across all configurations, confirming that the memory cliff is structural and can be eliminated at this scale.
Language Modeling (180M Scale): An Echo-180M model trained on 10B tokens matched or exceeded the performance of other 100M–400M parameter models (including GPT-2 345M and Mamba-3 variants) on five of six zero-shot benchmarks (HellaSwag, PIQA, ARC, WinoGrande, LAMBADA), despite using 10x fewer training tokens than some baselines. This suggests the architecture is both memory-efficient and data-efficient.

5. Significance and Claims

The paper claims that Echo demonstrates that SSMs need not trade their efficiency advantages (linear time, constant memory) to recover the recall capabilities of attention. By recasting retrieval as a closed-form spectral regression problem, Echo closes the performance gap with attention-based models on retrieval tasks while maintaining a fixed memory envelope.

The authors emphasize that the retrieval gain is driven by the spectral operator and the closed-form solution, not by prefix masking strategies. They argue that this approach points toward a new class of architectures where recurrence handles local context and lightweight operator modules handle sparse, long-range lookups, all within a constant memory budget. The work suggests that the "memory cliff" is a solvable structural issue rather than an inherent limitation of state-space models.