QLAM: A Quantum Long-Attention Memory Approach to Long-Sequence Token Modeling

Technical Summary: QLAM: A Quantum Long-Attention Memory Approach to Long-Sequence Token Modeling

Problem Statement
Modeling long-range dependencies in sequential data remains a central challenge in machine learning. While Recurrent Neural Networks (RNNs) suffer from vanishing or exploding gradients that degrade long-term information retention, Transformers address this via self-attention mechanisms but incur quadratic computational complexity relative to sequence length, limiting scalability. State-Space Models (SSMs) offer a linear-time alternative by evolving a latent state through recurrent updates; however, their memory is typically formed via additive or linear transitions. This classical additive accumulation can limit the ability to capture complex global interactions and may lead to information loss as sequences grow, as historical data must be compressed into a fixed-dimensional vector.

Methodology: Quantum Long-Attention Memory (QLAM)
The authors propose QLAM, a hybrid quantum-classical memory mechanism that restructures memory representation using quantum superposition and unitary evolution. Unlike classical approaches that store memory as token-wise embeddings or key-value pairs, QLAM represents the hidden state as a quantum state $|\psi_t\rangle$ in a complex Hilbert space.

• Memory Representation: The memory state is a normalized vector $|\psi_t\rangle \in \mathbb{C}^{2^n}$, where $n$ is the number of qubits. This allows the memory to encode a coherent superposition of multiple contextual signals simultaneously. The authors note that this representation offers a memory complexity of $O(n) \sim O(\log_2 d)$, compared to the $O(d)$ complexity of classical state vectors.
• Memory Evolution: The state evolves through parameterized quantum circuits (PQCs) conditioned on the input token $x_t$. The update rule is defined as $|\psi_t\rangle = U(x_t, \theta)|\psi_{t-1}\rangle$, where $U(x_t, \theta) = U_{var}(\theta)U_{enc}(x_t)$. Crucially, because $U$ is a unitary operator, the transformation is norm-preserving ($\||\psi_t\rangle\| = \||\psi_{t-1}\rangle\|$). This ensures stable information propagation over long sequences, avoiding the exponential growth or decay often seen in classical dynamical systems.
• Readout Mechanism: To retrieve task-relevant information, QLAM employs a query-dependent measurement. Instead of explicitly computing pairwise interactions between tokens, a learnable Hermitian observable operator $O(q_t)$, parameterized by the query $q_t$, is applied to the quantum state. The attention weight $\alpha_{t,s}$ is derived from the expectation value $\langle \psi_t | O(q_t) | \psi_t \rangle$. This allows for the implicit capture of global dependencies and the extraction of specific information without storing all historical token representations.

Key Contributions

1. Novel Memory Abstraction: The paper introduces QLAM as one of the first studies to leverage quantum superposition to enhance state-based sequence modeling, effectively replacing classical additive memory with a quantum superposition-based representation.
2. Hybrid Framework: It proposes a hybrid quantum-classical architecture where contextual information is stored and processed directly on a quantum device (simulated in this work), evolving via unitary transformations and retrieved via measurement.
3. Generalization of Attention: The work generalizes the attention mechanism to the quantum setting, where retrieval is performed via query-dependent measurements rather than explicit similarity computations over stored keys and values.
4. Empirical Validation: The authors conduct an empirical exploration to evaluate the feasibility of quantum memory in practical learning settings, demonstrating that hybrid quantum memory is a promising direction for rethinking sequence modeling.

Experimental Results
The authors evaluated QLAM on sequential variants of standard image classification benchmarks: sMNIST, sFashion-MNIST, and sCIFAR-10. In these tasks, images were flattened into token sequences to remove spatial inductive biases and force reliance on sequential reasoning.

• Performance: Across all three datasets, QLAM consistently outperformed recurrent baselines (RNNs), Transformer-based models, and State-Space Models (SSMs).
  • On sMNIST, QLAM achieved 92.6% accuracy, surpassing Transformers (91.3%) and SSMs (91.2%).
  • On sFashion-MNIST, QLAM reached 81.4%, outperforming Transformers (80.2%) and SSMs (79.6%).
  • On the more complex sCIFAR-10, QLAM achieved 53.6%, compared to 53.0% for Transformers and 51.8% for SSMs.
• Stability: The results showed low standard deviation across 10-fold evaluations, indicating that QLAM provides not only higher accuracy but also greater training stability and robustness to data splits, particularly in complex datasets.
• Efficiency: The improvements were achieved without increasing model complexity, suggesting the gains stem from the efficacy of the memory mechanism itself.

Significance and Claims
The paper claims that QLAM provides a principled extension of state-space formulations where quantum superposition replaces classical additive memory. This approach enables richer global representations while retaining the favorable linear-time computational scaling of SSMs. By fundamentally rethinking memory as an actively evolving quantum system rather than passive storage, QLAM opens a new research direction for integrating quantum computation into the core design of memory mechanisms. The authors conclude that while the findings validate the potential of quantum-state-based memory, further work is required to deepen the theoretical understanding, explore implementations on actual quantum hardware, and extend the framework to large-scale language and multimodal tasks.