Technical Summary: Raven - High-Recall Sequence Modeling with Sparse Memory Routing

1. Problem Statement

Linear-time sequence models, such as State-Space Models (SSMs) and linear Transformers, face a fundamental tradeoff in how they manage memory for long-context recall. Existing architectures generally fall into two extremes, both of which suffer from specific limitations in retrieving specific past tokens:

• Dense State Updates (SSMs): Models like Mamba-2, GLA, and GDN maintain a fixed-size state that updates densely at every step. Every memory slot is updated for every new token, relying on learned decay mechanisms to forget. While this allows for indefinite persistence in principle, the dense updates cause interference, making it difficult to preserve specific individual tokens without them being overwritten or diluted by subsequent information.
• Sparse, Position-Based Writes (Sliding-Window Attention - SWA): SWA writes sparsely by storing explicit token representations but restricts them to a fixed recent window. It uses a First-In-First-Out (FIFO) strategy where old tokens are hard-evicted based on position. While this allows for reliable retrieval within the window, recall drops to near zero once the relevant token is evicted, regardless of its importance.

The core problem is that existing models conflate where information is written with how long it persists. SSMs write everywhere and rely on decay to forget; SWA writes nowhere (except the current window) and relies on position to forget. Neither effectively combines sparse, content-dependent writing with explicit, controllable decay.

2. Methodology: Routing Slot Memories (RSMs)

The authors propose Routing Slot Memories (RSMs), a unified framework that decouples the location of writes from the duration of persistence. RSMs maintain a fixed set of $M$ memory slots and, at each time step $t$, use a router to determine which subset of slots receives new information and applies decay.

2.1 The RSM Framework

The general recurrence for an RSM is defined as:
$$S_t = (1 - r_t) \odot S_{t-1} + r_t \odot (D_t S_{t-1} A_t + U_t)$$
Where:

• $S_t \in \mathbb{R}^{M \times d}$ is the memory state with $M$ independent slots.
• $r_t \in \mathbb{R}^M$ is the routing vector determining write intensity for each slot.
• $D_t$ and $A_t$ control decay and feature mixing.
• $U_t$ is the write content derived from the input $x_t$.

The framework positions existing models on a spectrum:

• SSMs: Use a dense router ($r_t = \mathbf{1}_M$), updating all slots with gradual decay.
• SWA: Uses a time-dependent, one-hot router ($r_t = e_t$), updating exactly one slot with hard eviction (no decay).
• GSA/ABC: Use input-dependent routers but typically update all slots (dense writes), limiting their ability to isolate memories.

2.2 The Raven Architecture

Raven is a specific instantiation of RSMs designed to occupy the "middle ground" by combining sparse, input-dependent routing with explicit decay.

• Dual-State Structure: Like SWA, Raven maintains separate Key ($S^k_t$) and Value ($S^v_t$) states.
• Sparse, Input-Dependent Routing: Instead of a fixed window or dense update, Raven uses a router $g_t$ derived from the input $x_t$. It employs a Top-K selection strategy (adapted from DeepSeek MoE routing) to select a small subset of $K$ slots to update.
  • Scores are computed via a sigmoid function: $m_t = \sigma(W x_t)$.
  • Only the top $K$ entries are kept; others are set to zero.
  • The router is normalized to ensure the selected slots receive appropriate write intensity.
• Selective Decay: Decay is applied only to the selected slots. The recurrence is:
  $$S^k_t = \exp(a_t r_t) \odot S^k_{t-1} + (1 - \exp(a_t r_t)) \odot k^\top_t$$
  Here, $a_t$ is an input-dependent scalar decay (similar to Mamba-2). If a slot is not selected ($r_t[i] \approx 0$), the term $\exp(a_t r_t[i]) \approx 1$, meaning the slot is frozen and retains its content perfectly until selected again.
• Simplified Block Design: Unlike many linear models (e.g., Mamba, GLA), Raven omits short-range convolutions. It relies solely on the routing and decay mechanisms for local and long-range context integration, resulting in a simpler block structure.

3. Key Contributions

1. Routing Slot Memories (RSMs): A formal framework that unifies SSMs and SWA, explicitly separating the "write location" (routing) from the "forgetting mechanism" (decay).
2. The Raven Model: A linear-time sequence model that implements RSMs by using sparse, input-dependent routing to update a subset of memory slots while applying decay only to those updated slots.
3. Mechanism for Long-Range Recall: By routing retrieval-critical tokens (e.g., passkeys) to dedicated slots and freezing them, Raven prevents the destructive interference common in dense SSMs and the hard eviction of SWA.
4. Length Generalization: The model achieves strong performance on contexts up to 16$\times$ its training length without explicit length curriculum or positional interpolation techniques (like RoPE), as the variable effective sequence lengths across slots naturally expose the model to diverse temporal dynamics.

4. Experimental Results

The authors evaluated Raven on retrieval-intensive benchmarks and general language modeling tasks, comparing it against strong baselines including Mamba-2, GLA, GDN, SWA, and Transformer variants (FoX).

• Needle-in-a-Haystack (NIAH):
  • 400M Scale: Raven maintained near-perfect accuracy ($\ge 99\%$) up to 16K tokens and remained strong ($>91\%$) at 32K tokens (16$\times$ training length). In contrast, Mamba-2 and GDN degraded significantly beyond 8K.
  • 800M Scale: Raven achieved 91.0% accuracy on NIAH-1 at 64K tokens, significantly outperforming the next-best baseline (GDN at 45.2%).
• Real-World Recall Tasks:
  • On SWDE (web extraction), FDA (document extraction), and SQuAD, Raven consistently outperformed linear baselines. For instance, on FDA, it was the only linear model to exceed 22% accuracy, improving by ~8 points over the best SSM baseline.
• Language Modeling:
  • Raven matched or exceeded the performance of dense SSMs (Mamba-2, GLA) on standard zero-shot benchmarks (Lambada, PIQA, HellaSwag, etc.), demonstrating that the sparse routing does not compromise general language modeling capabilities.
• Hybrid Architectures:
  • When integrated into hybrid models (paired with full attention layers), Raven significantly outperformed hybrids using Mamba-2, GDN, or SWA-RoPE on long-context retrieval tasks, particularly in the 16K–64K range.

5. Significance and Claims

The paper claims that Raven addresses a fundamental architectural gap in linear sequence modeling. By explicitly separating where information is written from how long it persists, Raven achieves a level of long-range recall previously unattainable by linear models without resorting to quadratic attention or complex hybrid setups.

Key claims include:

• Superiority over SWA: Raven replaces position-based eviction with content-dependent routing and selective decay, enabling perfect recall on tasks where SWA fails completely.
• Superiority over Dense SSMs: Raven avoids the interference of dense state updates by writing selectively, allowing specific tokens to persist indefinitely in dedicated slots.
• Simplicity: The model achieves these gains without short-range convolutions, simplifying the architecture compared to prior linear models.
• Emergent Specialization: Analysis shows that Raven naturally learns to allocate specific memory slots for retrieval-critical tokens (e.g., passwords) while using other slots for general content, creating a structured, content-aware memory allocation without explicit supervision.

The authors conclude that the RSM framework suggests routing and forgetting are orthogonal design axes that should be treated as first-class design choices rather than byproducts of decay mechanisms.