⚡ electrical engineering

Beyond Sinusoids: A Morlet Wavelet Framework for Transformer Positional Encoding

This paper introduces Morlet Positional Encoding (MoPE), a novel framework that unifies sinusoidal and rotary positional encodings as limiting cases of a learnable wavelet-based approach, demonstrating improved performance in transformer models by simultaneously encoding position and locality while converging to the wavelet admissibility boundary.

Original authors: Athanasios Zeris

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Athanasios Zeris

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: "Where" vs. "How Far"

Imagine you are reading a book. Standard AI models (Transformers) have a built-in way of knowing where a word is in the sentence. They use a system called "Positional Encoding."

Currently, the two most popular systems (Sinusoidal and ROPE) act like a perfect, infinite ruler. They tell the AI, "This word is the 5th word," and "This word is the 100th word." But they treat every position exactly the same: they assume the influence of a word stretches forever.

The author argues this is wrong for language. In a story, the word "he" might refer to a character mentioned two sentences ago, but it probably doesn't refer to a character mentioned two chapters ago. Standard rulers don't know how far a word's influence should reach; they just say "it's here."

The Solution: The "Flashlight" (MOPE)

The author proposes a new system called Morlet Positional Encoding (MOPE).

Instead of a ruler that goes on forever, MOPE acts like a flashlight.

The Beam (Phase): Like the old systems, it tells the AI the exact angle or "phase" of the word (e.g., "You are at the 5th step of the dance"). This part is identical to the popular ROPE system.
The Glow (Amplitude): This is the new part. The flashlight beam gets dimmer the further you get from the center. In MOPE, the "brightness" of a position signal fades away as you move away from the start of the sequence.

The Analogy:

Old System (ROPE/Sin-cos): Imagine a stadium where every seat is lit up with a bright, unchanging light. A fan in seat 1 is just as "visible" to the announcer as a fan in seat 10,000.
New System (MOPE): Imagine a spotlight. The person in the center is super bright. As you move to the back rows, the light gets softer and eventually fades. This tells the AI: "Pay close attention to words nearby; words far away are less relevant right now."

The "Magic" Connection

The paper proves something fascinating: The old systems (Sin-cos and ROPE) are actually just broken versions of this new flashlight system.

If you make the flashlight beam infinitely wide (so it never gets dim), you get the old ROPE system.
If you make the beam infinitely wide and remove the rotation, you get the old Sin-cos system.

So, MOPE isn't just a new idea; it's the "parent" of the old ideas. It learns from the data how wide the flashlight beam should be for each part of the AI's brain.

What Happened in the Experiments?

The author tested this on a tiny dataset (a few thousand words from Shakespeare). Here is what they found:

The Combo is King: When they combined this new "Flashlight" (MOPE) with a different tool called "Energy-Gated Attention" (which acts like a bouncer deciding which words are important), the AI got significantly better at predicting the next word. It beat using either tool alone.
The "Sweet Spot" Discovery: The AI had to learn how wide to make the flashlight beams. Surprisingly, every single one of the 128 "flashlights" in the AI's brain adjusted itself to hit a specific mathematical limit. They all settled on a beam width that is neither too narrow nor too wide, but just right for the "character-to-word" scale.
- Think of it like tuning 128 radios: Instead of tuning them to different stations, they all locked onto the exact same frequency where the signal is clearest.
The "Start" Bias: The current version of MOPE has a quirk. It assumes the "center" of the flashlight is always at the very beginning of the text (Word #1). This means Word #5 is naturally "brighter" than Word #200, even if the story is about to start. The author admits this is a limitation and suggests future versions should let the AI learn where the "center" of the flashlight should be.

Summary

The paper suggests that instead of using a rigid, infinite ruler to track word positions, we should use a smart, fading spotlight. This spotlight learns how far its influence should reach.

Old way: "I am here, and I am important forever."
New way (MOPE): "I am here, and I am important for a short distance, then I fade out."

The experiments show that this "fading" approach, when paired with other smart attention tools, helps the AI understand the local structure of language (like sentences and phrases) much better than the old methods.

Technical Summary: Beyond Sinusoids: A Morlet Wavelet Framework for Transformer Positional Encoding

1. Problem Statement

Standard positional encodings for Transformers, specifically sinusoidal (sin/cos) and Rotary Position Embedding (ROPE), treat every position as equally local. While they encode where a token is, they fail to encode how far a token's positional influence should extend.

Sinusoidal PE: Uses fixed frequencies with uniform amplitude across all positions.
ROPE: Encodes relative position as a rotation in complex space but maintains a constant amplitude envelope (all positions are equally present).
Theoretical Gap: These methods lack a unifying mathematical framework. Furthermore, they assume signal stationarity (infinite position uncertainty), which is suboptimal for language, a non-stationary signal where the distribution of characters and syntactic structures varies with position.

2. Methodology: Morlet Positional Encoding (MOPE)

The authors propose Morlet Positional Encoding (MOPE), which utilizes the complex Morlet wavelet as the natural basis for positional encoding. The Morlet wavelet simultaneously minimizes uncertainty in both position and frequency, adhering to the Heisenberg uncertainty principle.

2.1 Definition

For a token at position $b$ and embedding dimension $i$ , MOPE is defined as:
$\text{MOPE}(b, i) = e^{i\omega_i b} \cdot e^{-b^2/2\sigma_i^2}$
This formulation introduces two learnable parameters per dimension:

Center Frequency ( $\omega_i$ ): Determines the oscillation rate (phase).
Bandwidth ( $\sigma_i$ ): Determines the Gaussian locality envelope (amplitude decay).

2.2 Geometric Interpretation

Phase: The phase component $\omega_i b$ is identical to the rotation angle in ROPE. It encodes the relative position within the oscillation cycle.
Amplitude: The term $e^{-b^2/2\sigma_i^2}$ acts as a Gaussian envelope centered at the sequence origin ( $b=0$ ). Unlike standard encodings where amplitude is fixed at 1 (implying $\sigma_i \to \infty$ ), MOPE learns a finite $\sigma_i$ , causing the positional signal to decay with distance from the start of the sequence.
Uncertainty Trade-off: MOPE learns the optimal trade-off between position uncertainty ( $\Delta b_i = \sigma_i$ ) and frequency uncertainty ( $\Delta \omega_i = 1/\sigma_i$ ), satisfying the bound $\Delta b_i \cdot \Delta \omega_i \geq 1/2$ .

2.3 Unification of Existing Methods

The paper proves that standard encodings are limiting cases of MOPE:

Sinusoidal PE: Emerges when $\sigma_i \to \infty$ (infinite locality, no decay).
ROPE: Emerges when $\sigma_i \to \infty$ . The MOPE cross-correlation kernel recovers the ROPE attention score exactly in this limit, modulated by a learned Gaussian locality kernel in the general case.
ALiBi: Conceptually analogous as a zero-frequency locality limit (exponential decay without oscillation).

3. Key Contributions

Theoretical Unification: Demonstrated that sin/cos PE and ROPE are degenerate cases of MOPE (where locality is switched off).
Phase Equivalence: Established that the MOPE phase difference is mathematically identical to the ROPE rotation angle. The primary difference is the addition of a learned Gaussian locality kernel in MOPE.
Empirical Combination: Combined MOPE with Energy-Gated Attention (EGA). The resulting model (EGA-MORLET) achieved a +0.119 improvement in validation perplexity over standard attention on the TinyShakespeare dataset, outperforming either component used in isolation.
Parameter Analysis: Analyzed learned parameters and found that all 128 frequency-bandwidth pairs converged exactly to the wavelet admissibility boundary ( $\omega_i \sigma_i = 5$ ).

4. Experimental Results

Experiments were conducted on a character-level TinyShakespeare dataset with a context length of $T=256$ using a 6-layer, 8-head Transformer architecture.

Performance:
- EGA-MORLET: 1.3550 Val Perplexity (+0.119 over baseline).
- EGA-1 (Baseline): 1.3821 (+0.092).
- ROPE: 1.4637 (+0.011).
- Sin/Cos: 1.5863 (-0.112).
- MOPE Alone: 1.5060 (-0.032).
- Note: MOPE alone underperformed the learned embedding baseline, likely due to the limited capacity of the small model (512 parameters for PE vs. 65,536 for a lookup table) and the structural bias of the origin-centered Gaussian. However, when combined with EGA, the synergy was significant.
Learned Parameters:
- Admissibility Boundary: All 128 dimensions converged to $\omega_i \sigma_i = 5.000$ . The optimizer consistently pushed parameters toward this boundary, suggesting a preference for stronger locality and lower frequencies than the constraint allows.
- Scale Concentration: Learned bandwidths ( $\sigma_i$ ) ranged from 1.49 to 4.50 tokens. This indicates the model concentrated all positional capacity on the character-to-word scale, rather than spanning broader scales (clauses, sentences) which would be expected in longer contexts.
- Frequency Compression: Learned frequencies were compressed into a narrow band ( $\omega_i \in [1.11, 3.35]$ ), far above the initial dyadic range.

5. Significance and Claims

The paper claims to provide the first unifying mathematical framework for oscillatory positional encodings, revealing them as special cases of a time-frequency localized wavelet basis.

Theoretical Insight: The work posits that language is non-stationary, and thus positional encodings should not assume infinite locality ( $\sigma = \infty$ ). MOPE allows the model to learn the appropriate "temporal range of influence" for each dimension.
Complementarity: The results suggest that spectral salience (handled by EGA, determining what to attend to) and time-frequency locality (handled by MOPE, determining where influence extends) are complementary mechanisms. Their combination yields super-additive benefits.
Reproducibility: The convergence to the admissibility boundary ( $\omega \sigma = 5$ ) was observed in both the positional encoding experiments and a separate Energy Gate experiment (Paper 1), suggesting a reproducible empirical property of character-level language signals.

6. Limitations and Future Work

The authors maintain a modest tone regarding the current findings:

Scale: Experiments are limited to character-level, short context ( $T=256$ ), and a single seed.
Origin Bias: The current formulation uses a Gaussian centered at $b=0$ , creating an "absolute origin prior" where tokens near the start have stronger positional amplitude than those at the end. This breaks translation invariance. The authors propose a future generalization with a learnable center $b_{0,i}$ .
Admissibility Constraint: The convergence to the boundary $\omega \sigma = 5$ may be an artifact of the projection-based implementation (clamping) rather than a true unconstrained optimum. Relaxing this constraint is a priority for future investigation.
Next Steps: Future work should validate MOPE at word-level tokenization, longer contexts, and explore Morlet-ROPE (applying the Gaussian envelope to the ROPE rotation mechanism).