🤖 machine learning

The Quantum Shortcut: Complex Phase-State Dynamics Reduce the Optimization Steps of Sequence Models

This paper demonstrates that replacing the standard real-valued hidden state substrate with a complex-valued, quantum-inspired alternative significantly accelerates the training convergence of both state-space and attention-based sequence models, though the long-term performance advantage persists only in state-space architectures.

Original authors: Ahmed Nebli, Hadi Saadatdoorabi, Christopher Keibel, Kevin Yam

Published 2026-08-18

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ahmed Nebli, Hadi Saadatdoorabi, Christopher Keibel, Kevin Yam

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern artificial intelligence, particularly the systems that write text, translate languages, and predict the next word in a sentence, relies on a specific kind of mathematical engine. These engines process information by moving it through a series of layers, much like a signal traveling through a long wire. For decades, the standard way to build these engines has been to use real numbers, the familiar counting numbers and decimals we use in daily life. In this standard setup, the engine remembers past information by keeping the size of its internal signals strong. However, as information travels further back in time, these signals naturally tend to shrink, like a whisper fading down a long hallway. When the signal becomes too small, the machine effectively forgets what happened at the beginning of the sentence, making it difficult to learn from long sequences of data. This fading is a fundamental limitation of the current design, forcing researchers to use massive amounts of data and time to teach the machine to remember.

A team of researchers has discovered a way to bypass this fading problem by changing the very number system the machine uses to think. Instead of relying solely on real numbers, they built a version of these language models that uses complex numbers. While real numbers have only a single value, complex numbers carry two distinct pieces of information: a size and a direction, or angle. The researchers found that by storing the important information in the angle rather than the size, the machine could travel much further without losing its memory. Even if the signal's size shrinks as it moves through the network, the angle remains perfectly sharp and unchanged. This allows the machine to retain the context of the beginning of a sentence all the way to the end, without the signal ever fading into silence.

The researchers tested this new approach by building two different types of language models, one based on a standard design and another based on a newer, more efficient design. They created a "complex" version of each, swapping out the real numbers for complex ones, and then trained them on three different sets of text, ranging from a small collection of 100 megabytes to a massive 15 gigabytes. The results were striking. On the smaller, newer design, the complex model reached the same level of accuracy as the standard model in only about one-third of the training steps. On the larger, more traditional design, it reached the same level in about half the steps. This means the new models learned the same amount of information while consuming significantly less data and time.

What makes this discovery particularly robust is that the advantage appeared immediately, but the behavior of the two designs diverged as training continued. The researchers observed that the complex models pulled ahead right from the very first hundred steps of training. Crucially, they found that the nature of this speedup differed between the two architectures. In the case of the newer design (the state-space model), the gap between the complex model and the standard model actually grew wider as training continued, suggesting that the benefit is a permanent feature of the new design rather than a fleeting artifact of the start of the process. For the older design (the attention-based model), however, the advantage was strongest at the beginning and then slowly faded, decaying toward zero as training progressed. Despite this decay, the early boost was still substantial enough to cut the training time in half for that specific architecture.

The researchers were careful to ensure that this speedup was not caused by simply adding more computing power or changing the size of the models. They matched the two versions of each model so precisely that they had the exact same number of adjustable parts, differing by less than one-hundredth of a percent. They also used the exact same training schedule and computer hardware for both versions. This strict control confirmed that the difference in speed came entirely from the switch to complex numbers and the way the machine processed them. The study also ruled out the idea that the complex models were just getting lucky with their starting conditions; the consistent advantage across different sizes of text and different model architectures pointed to a fundamental improvement in how the machine learns.

One of the most interesting aspects of this work is how it changes the way the machine handles conflicting information. In a standard model, if the machine is unsure whether a sequence of numbers represents a year or a price, it has to actively suppress one of those possibilities by shrinking its internal signal until it disappears. In the complex model, the machine can instead rotate the angle of the signal. If two possibilities are in conflict, the machine can rotate them so that they cancel each other out through a process called interference, similar to how noise-canceling headphones work by creating a sound wave that is the exact opposite of the noise. This allows the machine to discard the wrong idea without losing the strength of the right one, a capability that standard models cannot perform.

The researchers noted that while the complex models learned faster, there were some practical trade-offs. The computer chips used for these tasks are currently optimized for real numbers, so running the complex models required a bit more processing power per step. For the newer model design, this extra cost meant that the total time to train the model was roughly the same as the standard version, even though it took fewer steps. However, the complex model still consumed far less data to reach the same quality, which is a major advantage for specialized tasks where data is scarce. For the older model design, the researchers did not have an optimized version of the standard model to compare against, so they could not yet say if the complex version would be faster in real-world clock time, but the data efficiency gain was clear.

This work suggests that the choice of number system is just as important as the architecture of the model itself. For years, researchers have focused on changing how the machine routes information, but this study shows that changing the language in which the machine thinks can yield even greater benefits. The complex models did not just learn a little faster; they fundamentally changed the efficiency of the learning process, reaching high levels of performance with a fraction of the data. The researchers concluded that while their current version is a practical adaptation that relaxes some of the strict mathematical rules of the theory, it is enough to prove that the complex approach works at a large scale. They have released all their code and training logs, inviting others to verify the results and explore how this new way of thinking might improve the next generation of artificial intelligence.

Technical Summary: The Quantum Shortcut in Sequence Models

1. Problem Statement

Sequence models are conventionally defined by their "backbone" (e.g., attention or recurrence) which routes information across positions. However, nearly all current models share a prior design choice regarding their substrate: the number system used to represent the hidden state and the functional form of the map from state to prediction. The prevailing substrate uses a real-valued state vector ( $\mathbb{R}^d$ ) with an affine transformation followed by a softmax readout.

This paper identifies two fundamental limitations of the real-affine-softmax substrate that are independent of the backbone:

The Rank Bottleneck: An affine map followed by softmax produces log-probability matrices of rank at most $d+1$ . If the data's conditional structure has higher rank, the model is misspecified regardless of how the state is computed.
The Vanishing Gradient/Forgetting Trade-off: Real linear recurrences with spectral radius $\rho < 1$ attenuate propagated gradients by $\rho^T$ over $T$ steps. To maintain stability and allow forgetting, models must attenuate the state, which simultaneously erodes the signal from early positions.

The authors propose that these limitations stem from the substrate itself. They investigate whether a complex-valued substrate, drawing on the mathematics of quantum theory (specifically phase dynamics and the Born rule), can overcome these issues and reduce the number of optimization steps required to reach a target loss.

2. Methodology

The Deployable Complex Substrate

The paper introduces a "deployable" version of a complex substrate, relaxing two properties of an idealized quantum-inspired model (Nebli et al. [10]) that previously prevented direct deployment:

Relaxation of Exact Unitarity: Instead of a strictly norm-preserving (unitary) recurrence, which forbids forgetting, the authors use a contractive, input-dependent recurrence. The state evolution is defined as $|\psi_{t+1}\rangle = a_t \odot |\psi_t\rangle + b_t$ $∣ ψ_{t + 1} ⟩ = a_{t} ⊙ ∣ ψ_{t} ⟩ + b_{t}$ , where the transition coefficient $a_{t,k}$ $a_{t, k}$ is a product of a real gate $\alpha_{t,k} \in (0,1)$ $α_{t, k} \in (0, 1)$ and a unit-modulus phase $\lambda_{t,k}$ $λ_{t, k}$ .
- Mechanism: The gate $\alpha$ handles forgetting (magnitude contraction), while the phase $\lambda$ (generated via a diagonal Cayley transform) transports information without distortion. This decouples forgetting from information transport.
Relaxation of the Born Vocabulary Readout: The idealized Born rule ( $p_v = |\langle e_v | \psi \rangle|^2$ ) applied to the entire vocabulary is computationally expensive ($O(NV)$). The authors apply the Born rule only as the scoring function for linear attention (a quadratic form $|\langle q | k \rangle|^2$ ), while retaining a standard tied-softmax head for the final vocabulary output.

Experimental Design

To isolate the effect of the substrate, the authors instantiated the complex substrate in two distinct backbones:

H-Mamba: A complex realization of the Mamba state-space model.
H-Transformer: A complex realization of the Transformer using linear attention with Born-rule scoring.

Controls:

Parameter Matching: Complex and real models were matched to within 0.02% of total parameters.
Protocol: A single, fixed training protocol (AdamW, cosine schedule, 750-step warmup) was applied to all models without per-model tuning.
Corpora: Three byte-level corpora spanning 150x in size: enwik8 (100 MB), OpenWebText (3 GB), and FineWeb (15 GB).
Metrics: The primary metric is the number of optimization steps to reach specific validation loss targets (bits per character), rather than just final loss.

3. Key Contributions

Definition of a Deployable Complex Substrate: The paper formalizes a substrate that retains the interference properties of complex phases (via unit-modulus Cayley transforms) while allowing for selective forgetting (via sigmoid gates) and efficient vocabulary readouts (via softmax).
The "Quantum Shortcut": Empirical evidence that this substrate reduces the optimization steps required to reach a target loss by a factor of approximately 3 for state-space models and 2 for attention models, compared to their real-valued counterparts of identical size.
Mechanism Analysis:
- Phase Transport: Theoretical analysis (Lemma 6.2) shows that relative phase is transported through the recurrence with unit sensitivity, regardless of the contraction gates, whereas real magnitudes decay.
- Interference: The quadratic Born score allows for destructive interference (cancellation) of hypotheses, a capability absent in linear real readouts.
- Decomposition: The paper argues that the state-space acceleration is primarily due to improved gradient conditioning (phase transport), while the attention acceleration is a mix of interference and conditioning, though the latter diminishes as the real baseline is already well-conditioned.

4. Results

Optimization Efficiency

Across all three corpora and three loss targets (2.5, 2.2, and 2.0 bpc):

H-Mamba reached targets in approximately 1/3 the steps of the real Mamba.
H-Transformer reached targets in approximately 1/2 the steps of the real Transformer.
This advantage was established early in training (within the first 200 steps) and remained stable across the descending loss curve.

Post-Warmup Behavior

The paper distinguishes between early training effects and persistent advantages:

State-Space (Mamba): The performance gap widened after the learning-rate warmup ended (e.g., from 0.321 to 0.354 bpc on OpenWebText), suggesting the advantage is not an artifact of the warmup schedule but a structural benefit of the recurrence.
Attention (Transformer): The performance gap decayed toward zero after the warmup, indicating the advantage is largely an early-training effect.

Terminal Performance and Data Limits

On the large web corpora (OpenWebText, FineWeb), the complex models achieved lower final validation losses than their real counterparts.
On the small enwik8 corpus (100 MB), the complex models reached the data limit earlier and began to overfit sooner than the real models, a consequence of their higher sample efficiency.

Computational Cost

Tokens: The reduction in steps translates directly to a reduction in tokens consumed (e.g., ~0.57B tokens saved for H-Mamba on FineWeb to reach 2.0 bpc).
Wall-Clock Time: For the attention pair, the current fused kernel overhead (running at half the throughput of the real counterpart) cancels the step advantage, resulting in parity in wall-clock time. For the state-space pair, a fair comparison is pending an optimized real baseline kernel, but the step advantage suggests potential time savings if such kernels are developed.

5. Significance and Claims

The paper claims that the substrate (the number system and readout form) is a design axis as critical as the backbone. The primary significance is the demonstration that replacing the real substrate with a complex one, even in a relaxed, deployable form, yields a substantial reduction in the optimization steps required for sequence modeling.

The authors explicitly state:

The "Quantum Shortcut" is a reduction in optimization steps, not necessarily a reduction in wall-clock time (due to current kernel inefficiencies).
The advantage is backbone-independent in the early phase but persistent only for the state-space backbone, supporting the hypothesis that the mechanism is rooted in the recurrence's ability to transport phase without attenuation.
The results are not attributed to parameter count differences or tuning artifacts, given the strict matching and identical protocols.
The paper does not claim to solve the rank bottleneck at the vocabulary level (as the softmax head remains), but rather demonstrates that the complex substrate improves the optimization trajectory and sample efficiency of existing backbones.

The work concludes that the field over which a model computes is a fundamental design decision, and that complex-valued dynamics offer a viable path to more efficient training of sequence models.