🤖 machine learning

Complex-Valued Phase-Coherent Transformer

This paper introduces the Phase-Coherent Transformer (PCT), a novel architecture that replaces standard softmax attention with a smooth, element-independent gating mechanism on L2-normalized complex similarities to preserve phase information, thereby achieving superior generalization and performance across diverse long-range and complex-valued tasks compared to both real-valued and existing complex-valued baselines.

Original authors: Leona Hioki

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Leona Hioki

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to organize a massive library where every book has a secret "phase" or hidden rhythm attached to it. In the world of artificial intelligence, this is like dealing with Complex-Valued Neural Networks. These networks are powerful because they can handle this extra "phase" information (like the timing of a sound wave or the direction of a signal), but they have been struggling to work well on general tasks like reading text or recognizing images.

The main problem, according to this paper, is that the standard way these networks organize information is like a competitive reality show.

The Problem: The "Token Competition" Reality Show

In standard AI models (Transformers), when the model looks at a sentence, it uses a mechanism called Softmax Attention. Think of this as a reality show where all the words (tokens) in a sentence are contestants. They all fight for a limited amount of "attention mass" (like a prize pool). The model forces them to compete so that the total attention adds up to 100%.

The Issue: In a complex-valued network, the "phase" (the hidden rhythm) is crucial. If you force these contestants to compete, you are essentially telling them, "If you win, you must make the others lose." This competition scrambles the delicate phase relationships. It's like trying to keep a group of dancers in perfect sync while forcing them to fight each other for the spotlight. The paper argues this "token competition" destroys the very information the complex network needs to work.

The Solution: The Phase-Coherent Transformer (PCT)

The authors introduce a new model called the Phase-Coherent Transformer (PCT). Instead of a reality show, imagine a collaborative orchestra.

No Fighting: In PCT, the words don't compete. Each word gets to speak its piece independently.
The Smooth Gate: Instead of a harsh competition, the model uses a "smooth gate" (a mathematical function called a sigmoid). Think of this as a volume knob for each word. If a word is relevant, the knob turns up; if not, it turns down. Crucially, turning one knob up doesn't force another to turn down.
Preserving the Rhythm: Because the words aren't fighting, the hidden "phase" information flows through the layers of the network without getting scrambled. It's like passing a baton in a relay race where the runner doesn't drop it because they are too busy shoving the person next to them.

The Experiments: Putting the Orchestra to the Test

The authors tested this new "orchestra" against the old "reality show" models across 9 different challenges, ranging from remembering long strings of numbers to classifying images and decoding radio signals.

Here is what they found, using simple analogies:

The "Needle in a Haystack" Test (NIAH): Imagine a library with a million books, and you need to find one specific sentence hidden in the middle.
- Old Complex Models: Failed completely (0% success). They got lost in the noise.
- Old Real Models: Failed completely.
- PCT: Found the needle every single time (100% success). It proved that complex networks can be better than real ones if you stop them from fighting.
The "Copy Memory" Test: Imagine asking the model to remember a long list of numbers and repeat them back later.
- Old Models: As the list got longer, they forgot everything.
- PCT: Remembered lists of 5,000 numbers perfectly. It showed that without the "competition," the network could hold onto information for much longer.
The "Depth" Test: Imagine stacking the network deeper and deeper (adding more layers).
- Old Complex Models: Usually, as you add more layers, the signal gets so noisy that the model breaks down (like a whisper getting lost after passing through 20 people).
- PCT: Worked perfectly even at 20 layers deep. It didn't break. The "orchestra" stayed in sync no matter how many musicians were added.
The "Radio Signal" Test: They tested the model on real-world radio data (complex signals).
- Result: PCT performed very well, beating the old complex models. Interestingly, a different "non-competitive" real-valued model (called real_screen) was slightly better on this specific task, suggesting that while PCT is a huge leap forward, there is still room for tuning in specific physical domains.

The "Why" Behind the Magic

The paper digs deep into why PCT works and identifies two golden rules:

No Token Competition: Don't force the data points to fight for attention.
Phase Preservation: Don't delete the "negative" or "anti-phase" parts of the signal.

They tested this by deliberately breaking the rules:

If they used a gate that deleted negative signals (like a "ReLU" gate), the model collapsed and failed completely.
If they used a gate that was too wild and unbounded, the model struggled.
But if they kept the gate smooth and non-competitive (like PCT), the model thrived.

The Bottom Line

This paper claims that the reason complex-valued AI hasn't been a "general-purpose" tool until now is that we were trying to force it to play by rules designed for simple, real-valued AI (the "competition" rules).

By switching to a Phase-Coherent Transformer, which lets information flow without forced competition, complex networks can finally generalize. They can handle long memories, deep reasoning, and complex signals just as well as, and sometimes better than, traditional models. The authors conclude that this "non-competitive" approach is the key to unlocking the true potential of complex AI.

Technical Summary: The Phase-Coherent Transformer (PCT)

1. Problem Statement

Complex-valued neural networks (CvNNs) have historically been confined to signal-processing domains (e.g., MRI, wireless communication) where phase information is intrinsic. Their adoption as general-purpose architectures for text, vision, and reasoning has been limited. The paper identifies a specific architectural bottleneck: existing complex-valued Transformers largely inherit the softmax attention mechanism from real-valued architectures.

The authors argue that softmax attention relies on row-normalized token competition, where attention weights within a row sum to one. This structure is fundamentally misaligned with complex-valued computation, which requires the preservation of phase information across layers. The "token competition" enforced by softmax introduces a coupling that disrupts phase superposition, leading to a "complex penalty" where vanilla complex Transformers underperform their real-valued counterparts, particularly on long-range dependency tasks.

2. Methodology: The Phase-Coherent Transformer (PCT)

To address this, the paper proposes the Phase-Coherent Transformer (PCT), a new attention primitive designed to preserve phase coherence across layers.

Core Mechanism

PCT replaces the row-normalized softmax with a token-non-competing attention mechanism:

Substrate: It utilizes native complex linear projections for Query ( $Q$ ), Key ( $K$ ), and Value ( $V$ ).
Normalization: Queries and Keys are L2-normalized ( $\bar{q}, \bar{k}$ ).
Scoring: The attention score is derived from the real part of the inner product of normalized vectors: $s_{ij} = \text{Re}\langle \bar{q}_i, \bar{k}_j \rangle \cdot \sqrt{d}$ .
Gating: Instead of softmax, PCT applies a real-valued, element-independent, smooth gate to the score. The primary instance uses a sigmoid function with a learnable bias initialized to $-\log N$ :
$\alpha_{ij} = \sigma(s_{ij} + b)$
Aggregation: The output is computed as a weighted sum of complex values: $\text{out}_i = W_o \sum_{j} \alpha_{ij} v_j$ .

Crucially, there is no row normalization. This allows tokens to attend independently, preserving the phase superposition of complex values without the destructive interference caused by softmax's competitive normalization.

Theoretical Framework: Two-Level Phase Coherence

The paper formalizes the success of PCT through a "two-level phase coherence" framework defined by four conditions on the gate function $f$ :

C1: The gate output is real-valued.
C2: The gate is bounded within the operating range.
C3: The gate is smooth with a non-zero gradient (preserving anti-phase components).
C4: The gate is element-independent (no row coupling).

Level 1 (L1) Coherence: Satisfied by C1 and C4, ensuring per-layer phase relationships are preserved (no token coupling).
Level 2 (L2) Coherence: Satisfied by L1, C2, and C3, ensuring that phase noise does not accumulate exponentially across a deep stack. The paper proves that violating C3 (e.g., using ReLU) causes catastrophic cascade failure, while violating C2 (e.g., unbounded gates) leads to partial degradation.

3. Experimental Setup

The authors conducted a systematic 6-cell comparison across 9 mid-scale benchmarks, ensuring parameter fairness ( $\text{real\_dim} = \text{complex\_dim} \times 1.41$ ).

The 6 Cells:

real_softmax: Standard Transformer baseline.
real_sigmoid: Real-valued sigmoid attention.
real_screen: Real-valued screening attention (Nakanishi 2026).
complex_softmax: Vanilla complex Transformer (softmax of magnitude).
complex_screen: A complex variant with a hard threshold gate (partial C3 violation).
PCT (complex_sigmoid): The proposed architecture.

Benchmarks:

Synthetic/Algorithmic: Copy Memory (short/long range), Needle In A Haystack (NIAH), LRA-ListOps, FFT-MNIST, Phase-Memory, Multi-Pitch.
Real-World: RadioML (modulation classification), MusicNet.
Depth Scaling: Tests from depth 2 to 20.

4. Key Results

Generalization and Performance

Superiority over Vanilla Complex: PCT consistently outperforms complex_softmax across all tasks. On long-range Copy tasks, complex_softmax collapses to chance levels, while PCT achieves 1.000 accuracy.
Superiority over Real Baselines: Under parameter-fair conditions, PCT outperforms real_softmax on every task tested, including long-range retrieval and algorithmic reasoning.
Competitiveness with Strongest Real Baselines: PCT matches or exceeds real_screen (the strongest non-softmax real baseline) on synthetic and algorithmic tasks. On the physical complex domain task (RadioML), real_screen slightly edges PCT, but PCT still significantly outperforms complex_softmax.

Specific Task Findings

NIAH (Needle In A Haystack): PCT achieves 1.000 accuracy on NIAH (L=2048), a task where complex Q/K offer no inductive advantage (purely positional). This contradicts the notion that complex networks are inherently weak on such tasks; the failure was due to the attention mechanism, not the substrate.
Long-Range Robustness: PCT is uniquely robust to hyperparameter variations (learning rate and batch size) on long-range Copy tasks, maintaining 1.000 accuracy across a wide window where other cells fail.
Depth Scaling: PCT shows no depth-related accuracy collapse across depths 2 to 20. The accuracy remains stable, addressing a long-standing concern about phase noise accumulation in deep complex networks.

Ablation and Isolation

Gate Violations: Experiments with gates that deliberately violate PCT conditions confirm the theoretical framework:
- C3 Violation (Anti-phase deletion): Gates like ReLU or Clamped ReLU cause immediate collapse to chance level on long-range tasks, regardless of depth.
- C2 Violation (Unboundedness): Gates like Cubic (unbounded in the operating range) cause significant performance degradation (e.g., dropping from 1.000 to 0.200 on Copy tasks) but do not cause total collapse if C3 is intact.
Substrate Robustness: The complex substrate is necessary, but no single component (Q, K, V, or Output) is irreducibly responsible; the architecture is robust to parameterization choices as long as the complex nature is preserved.

5. Significance and Claims

The paper claims that the introduction of token-non-competing attention and multi-layer phase preservation enables complex-valued neural networks to generalize beyond their traditional signal-processing domains.

Architectural Principle: The primary contribution is the identification that softmax is structurally incompatible with phase-preserving complex computation. The "token competition" inherent in softmax is an unnecessary constraint that breaks phase coherence.
Design Guideline: The paper establishes a "closeness-to-PCT" taxonomy. Architectures that satisfy the C1–C4 conditions (or have partial/bypassed violations) generalize well, while those with strict violations (like ReLU or Softmax) fail.
Modest Scope: The authors explicitly state that while PCT demonstrates strong generalization on the tested benchmarks, they do not claim a quantitative scaling law (e.g., Chinchilla-style) for depth. The claim is qualitative: PCT eliminates the depth-collapse failure mode previously observed in complex networks.
Open Question: The paper notes an anomaly where real_screen outperforms PCT on the RadioML task, flagging this as an open question regarding the specific mechanisms of physical complex-domain data, though it does not invalidate the core anti-softmax claim.

In summary, the paper argues that for complex-valued Transformers, the key to generalization is not merely porting real-valued components to complex space, but redesigning the attention primitive to respect the structural requirements of phase coherence.