Phase-Associative Memory: Sequence Modeling in Complex Hilbert Space

1. Problem Statement

The paper addresses two primary challenges in current Large Language Model (LLM) development:

• The Limitations of Real-Valued Representations: Traditional models (like Transformers) rely on real-valued vector spaces and the principle of compositionality, assuming semantic meaning can be analyzed independently of context. However, recent empirical evidence (Bell violation tests, sheaf-theoretic analysis) suggests that natural language exhibits non-classical contextuality, where meaning is indeterminate prior to interpretation and depends on the interaction of components. Real-valued models attempt to approximate this complex structure by projecting it onto a real space, leading to inefficiencies and structural limitations.
• The Associative Recall Problem in Linear Attention: Linear attention mechanisms (without softmax) struggle with "associative recall" because real-valued dot products are non-negative. This means all stored associations contribute positively to the retrieval, causing signal dilution. Softmax attention solves this via exponential sharpening but incurs quadratic complexity ($O(T^2)$) and requires growing Key-Value (KV) caches.
• Holographic Binding Failure: Previous attempts to use complex-valued "holographic reduced representations" in vector-state models failed due to the $O(1/\sqrt{n})$ capacity degradation when superposing multiple associations in a single vector.

2. Methodology: Phase-Associative Memory (PAM)

The authors propose Phase-Associative Memory (PAM), a recurrent sequence model that operates natively in Complex Hilbert Space ($\mathbb{C}^d$).

Core Architecture

• Complex-Valued State: Unlike standard RNNs or Transformers, PAM maintains a matrix state $S_t \in \mathbb{C}^{d \times d}$ rather than a vector state. This upgrade from $\mathbb{C}^d$ to $\mathbb{C}^{d \times d}$ resolves the capacity degradation issue of superposed associations, providing $O(d^2)$ associative capacity per head.
• Operations:
  • Embedding & Mixing: Uses a ComplexGatedUnit (CGU) for channel mixing and Phase-Associative Memory (PAM) for sequence mixing.
  • Phase Preservation: All operations in the main signal path are complex-valued and phase-preserving. The activation function is modReLU (thresholds magnitude, preserves phase), and normalization is applied to magnitudes only.
  • Positional Encoding: Uses Complex Rotary Position Embeddings (RoPE) on Queries ($Q$) and Keys ($K$), encoding absolute position in the phase while preserving magnitudes.

The Retrieval Mechanism

The core innovation lies in how associations are accumulated and retrieved:

1. Accumulation: Associations are accumulated via outer products in the matrix state:
   $$S_t = \gamma_t S_{t-1} + V'_t \otimes K^*_t$$
   Where $K^*_t$ is the complex conjugate of the key, and $\gamma_t$ is a decay rate.
2. Retrieval: Retrieval is performed via the conjugate inner product between the query and the state:
   $$Y_t = S_t \tilde{Q}_t = \sum_{i \le t} \left( \prod_{j=i+1}^t \gamma_j \right) (K^*_i \cdot \tilde{Q}_t) V'_i$$
   • Phase Alignment: The term $K^*_i \cdot \tilde{Q}_t$ acts as a retrieval strength. If the phase of the stored key is coherent with the query, the association is retrieved strongly. If they are phase-incoherent, the result can be negative or imaginary, leading to destructive interference that actively suppresses irrelevant information. This mimics quantum measurement probabilities ($P = |\langle \psi | \phi \rangle|^2$).

Computational Efficiency

• Training: Uses a parallelizable dual form with $O(T^2)$ complexity (similar to attention) but without softmax.
• Inference: Operates as a recurrent model with $O(1)$ per-token cost. The state size is fixed ($H \times d^2$) and does not grow with sequence length, eliminating the need for a KV cache.

3. Key Contributions

1. Native Complex Hilbert Space Modeling: PAM is the first complete language model that operates entirely in complex space from embedding to output, demonstrating that complex arithmetic is a viable and efficient formalism for language modeling.
2. Matrix-State Resolution of Holographic Binding: The paper proves that upgrading the state from a vector ($\mathbb{C}^d$) to a matrix ($\mathbb{C}^{d \times d}$) resolves the capacity degradation of superposed associations, enabling effective holographic binding.
3. Destructive Interference for Retrieval: It introduces a mechanism where phase misalignment naturally suppresses irrelevant associations without needing softmax normalization, addressing the "signal dilution" problem of linear attention.
4. Theoretical Alignment: The architecture aligns with the "Quantum Semantic Framework," suggesting that language models should utilize the algebra of complex superposition and interference to handle the non-separable, contextual nature of human language.

4. Results

The model was evaluated on WikiText-103 with approximately 100M parameters (16 blocks, 6 heads, $d=64$).

• Performance:
  • PAM Validation Perplexity: 30.0 (after 10 epochs).
  • Matched Transformer Validation Perplexity: 27.1 (trained under identical conditions).
  • Gap: PAM is within ~10% of the Transformer, a significant result given the architectural differences.
• Efficiency & Overhead:
  • Arithmetic Cost: PAM has 4x arithmetic overhead compared to real-valued computation (due to complex matrix multiplications) and uses no custom CUDA kernels.
  • Throughput: PAM achieved 23k tokens/sec vs. the Transformer's 96k tokens/sec.
  • Inference Memory: PAM uses a fixed state size (49,152 floats per layer) regardless of sequence length, whereas the Transformer's KV cache grows linearly with context. At 2048 tokens, the Transformer's cache is ~56x larger than PAM's state.
• Ablation Studies:
  • Normalizing Query/Key magnitudes to unit vectors caused generation to collapse into repetition, proving that both magnitude and phase are necessary for the conjugate inner product to function as a retrieval mechanism.
  • Adding sparse windowed attention provided no benefit, suggesting PAM captures the necessary information via complex recurrence alone.

5. Significance and Implications

• Paradigm Shift: The paper challenges the dominance of real-valued, softmax-based architectures. It suggests that the "hallucinations" and "jailbreaks" in current LLMs may be inherent consequences of trying to force a non-classical, contextual system (language) into a classical, separable framework.
• Interpretability: By operating natively in a complex Hilbert space, PAM may produce representations that are more naturally described by sheaf-theoretic or quantum-mechanical frameworks, potentially offering a path toward better mechanistic interpretability than current sparse autoencoder approaches.
• Scalability: While currently slower due to unoptimized complex arithmetic, PAM offers a fundamentally more memory-efficient inference path (fixed state vs. linear KV cache) and a mathematically grounded approach to handling contextuality. The authors posit that as implementation optimizations (custom kernels) are applied, the performance gap may narrow significantly.

In conclusion, Phase-Associative Memory demonstrates that complex-valued sequence modeling is not just a theoretical curiosity but a competitive, structurally distinct approach to language modeling that aligns with recent findings on the non-classical nature of semantic context.