🤖 machine learning

PsiLogic: Chaos-Aware Active Cancellation for Adam with a Fair Cross-Domain Benchmark

The paper introduces PsiLogic, a chaos-aware optimizer that dynamically augments Adam with an active cancellation term to stabilize early training without manual warmup schedules, demonstrating statistically significant performance improvements over Adam, AdamW, and Lion across NLP and vision benchmarks in a rigorous, reproducible cross-domain evaluation.

Original authors: Ali Sultonov

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ali Sultonov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Great Balancing Act of Learning Machines

Imagine you are trying to teach a robot to recognize cats by showing it thousands of pictures. To do this, the robot uses a "brain" made of math, and it learns by making tiny adjustments to its internal settings every time it sees a new picture. This process is called training, and the tool that decides how big or small those adjustments should be is called an optimizer. Think of the optimizer as the robot's teacher. If the teacher is too strict, the robot learns slowly; if the teacher is too lenient, the robot gets confused and forgets what it just learned.

For years, the most popular teacher in the world of AI has been a method called Adam. It's like a very experienced coach who knows how to adjust the robot's steps based on how fast it's currently running. However, there's a catch: this coach uses the same rules whether the robot is just starting out (where everything is chaotic, messy, and loud) or whether it's almost finished (where things are quiet and steady). In the real world, we know that starting a new task is usually a chaotic mess, while finishing it is calm. The paper you are about to read asks a simple question: What if our robot's teacher could sense when things are chaotic and change its behavior accordingly? This is the heart of a new study exploring a smarter way to train AI models, aiming to make them learn faster and more reliably without getting lost in the noise of the beginning.

Meet PsiLogic: The Chaos-Sensing Coach

The researchers behind this study, led by independent researcher Ali Sultonov, have introduced a new optimizer called PsiLogic (or $\Psi$ Logic). You can think of it as an upgrade to the famous Adam optimizer, but with a special superpower: a "chaos detector."

Imagine you are driving a car. When you first start the engine on a cold morning, the car might sputter, shake, and make weird noises. If you floor the gas pedal immediately, you might stall the engine. But once the engine is warm and running smoothly, you can drive normally. Standard optimizers like Adam treat the cold, sputtering engine the same way they treat the smooth, warm one—they just keep pressing the gas. PsiLogic, however, is like a smart driver who feels the engine shaking. When the car is sputtering (the "chaotic" early phase of training), PsiLogic gently eases off the gas and applies the brakes to stop the car from spinning out of control. As the engine warms up and the ride becomes smooth (the "stable" phase near the end), the brakes fade away, and the car drives just like it normally would.

How does it work?
The paper explains that during the early stages of training, the math behind the AI's learning is very noisy and unpredictable. PsiLogic watches the "volume" of this noise. It uses two different "thermometers" (mathematical averages) to measure how wild the data is. If the short-term thermometer spikes way above the long-term one, the system knows chaos is happening. It then activates a "cancellation term"—a fancy way of saying it adds a damping force to shrink the learning steps. This happens automatically, without the human trainer needing to set a timer or a schedule. It's an implicit warmup: the system knows when to be careful and when to be bold, all on its own.

The Big Test: FairBench
To see if this new idea actually works, the researchers didn't just guess; they built a rigorous testing ground they call FairBench. They wanted to make sure they weren't cheating by giving one optimizer a better learning rate than another. So, they gave every optimizer (Adam, AdamW, Lion, and PsiLogic) the exact same starting point and let them find their own best speed. They tested them on four different "arenas" (tasks):

NLP (Language): Teaching a model to predict the next word in a story.
ViT (Vision): Teaching a model to recognize images.
ResNet (Vision): Another image recognition task.
Diffusion (Art): Teaching a model to generate images from noise.

They ran these tests on a powerful NVIDIA H100 computer, repeating each experiment three times to ensure the results weren't just luck.

What Did They Find?
The results were quite exciting, but the authors were very honest about where PsiLogic shines and where it doesn't.

The Wins: In three out of the four arenas, PsiLogic came out on top.
- In the Language (NLP) task, it achieved a "perplexity" (a score where lower is better) of 7.79, beating AdamW's 8.17. The researchers say this difference is statistically significant, meaning it's likely a real improvement and not a fluke.
- In the ViT (Vision) task, PsiLogic reached an accuracy of 0.244, significantly higher than AdamW's 0.223.
- In the ResNet task, it hit 0.222 accuracy, which was better than the standard Adam optimizer's 0.172.
The Ties:
- In the Diffusion (image generation) task, PsiLogic performed about the same as the standard methods. The difference was so small it was statistically "tied."
- In the ResNet task, when compared specifically to AdamW, the results were a "numerical tie" (0.222 vs 0.219) that wasn't statistically significant.

The Catch: Speed
The paper is very transparent about a downside. While PsiLogic learns better in some cases, it is slower. On the ViT task, it took 1.79 times longer to complete a single step than AdamW. The authors explain that this isn't because the math is too hard, but because their current computer code isn't perfectly optimized yet. They believe this speed penalty can be fixed in the future by writing better code, but for now, it's a trade-off: you might get better results, but it will take a bit more time.

The Bottom Line
This paper suggests that by adding a "chaos detector" to our AI teachers, we can help them navigate the messy early days of learning without getting confused. PsiLogic doesn't claim to be the perfect solution for every single problem (it didn't win the image generation contest, for example), but it shows that being sensitive to the "mood" of the training process can lead to better performance in language and image recognition tasks. The researchers have made all their code and data public, inviting others to check their work and see if they can get the same results. It's a promising step toward AI that knows when to be gentle and when to be bold, all by listening to the noise of its own learning.

Technical Summary: PsiLogic – Chaos-Aware Active Cancellation for Adam

1. Problem Statement

Adaptive optimizers like Adam and AdamW dominate deep learning practice but apply a static update rule regardless of the training phase. They fail to distinguish between the chaotic early phase (characterized by large, noisy gradients) and the stable convergence phase (small, stable gradients). Standard Adam treats these regimes with structurally similar updates, potentially leading to instability during initialization or suboptimal damping near convergence. While manual learning rate (LR) warmup is a common heuristic, it relies on hand-tuned schedules rather than the model's actual state of confusion.

2. Methodology: PsiLogic

PsiLogic ( $\Psi$ Logic) is proposed as a drop-in replacement for torch.optim.Adam. It augments the standard Adam update with a dynamic Active Cancellation Term gated by a "chaos detector."

Core Mechanism

The optimizer introduces a scalar chaos warmup weight ( $w_t$ ) and a chaos signal ( $chaos_t$ ) that modulate a unified decay coefficient ( $\delta_t$ ).

Chaos Detector:
- Computes a scale-normalized gradient norm: $g_{nt} = \|g_t\|_2 / \sqrt{\text{numel}}$ .
- Maintains two Exponential Moving Averages (EMAs):
  - fast: Short horizon (~10 steps, $\alpha=0.1$ ).
  - slow: Long horizon (~100 steps, $\alpha=0.01$ ).
- Calculates a ratio $r_t = \text{fast}_t / (\text{slow}_t + \epsilon)$ .
- The chaos signal is defined as:
  $chaos_t = \tanh(\text{slow}_t) \cdot (1 + 0.5 \cdot \tanh(\text{relu}(r_t - 1)))$
- Logic: When fast significantly exceeds slow (indicating a spike in gradient volatility), the signal activates. As training stabilizes ( $\text{slow}_t \to 0$ ), the signal vanishes.
Unified Decay & Update Rule:
Instead of applying weight decay and cancellation separately, PsiLogic computes a single combined decay coefficient $\delta_t$ to avoid over-damping cross-terms:
$\delta_t = \eta\lambda + w_t \cdot \min(\tilde{c}_t, c_{max})$
Where $\tilde{c}_t$ is the raw cancellation fraction based on the chaos signal, group-specific amplification factors ( $P_k$ ), and a gain parameter ( $\gamma$ ). $c_{max}$ caps the shrinkage during volatile initialization.

The parameter update becomes:
$\theta_{t+1} = \theta_t \cdot (1 - \delta_t) - \eta \cdot \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$
This structure provides implicit warmup: damping is strong when gradients are unstable and fades automatically as the model converges, without an external schedule.
Optional Features:
- Gradient Centralization (GC) & Adaptive Gradient Clipping (AGC): Integrated to further improve stability.
- Quantum Decay: An optional, gradient-dependent regularization term active only when the chaos gate is off ( $s_t=0$ ).

3. Key Contributions

The paper outlines three primary contributions:

PsiLogic Optimizer: A chaos-gated Active Cancellation mechanism built on Adam, featuring unified decay, optional GC/AGC, and adaptive group-specific amplification.
FairBench: A reproducible, bias-mitigated evaluation protocol. Unlike previous comparisons that often use a single shared learning rate, FairBench performs per-optimizer learning rate sweeps (7 log-spaced rates) and uses identical initialization per seed to ensure fair comparison. It employs Welch t-tests for statistical significance.
Reference Benchmark: A comprehensive evaluation on an NVIDIA H100 80GB across four domains (NLP, ViT, ResNet, Diffusion) with full open-source release of code, raw CSVs, and learning curves.

4. Experimental Results

Evaluated over 3 seeds with 2000 steps per arena (bf16 AMP, batch 64):

NLP (TinyStories): PsiLogic achieved a perplexity of 7.79 ± 0.18, significantly outperforming AdamW (8.17 ± 0.08, $p=0.049$ ).
ViT (CIFAR-100): PsiLogic reached 0.244 ± 0.006 top-1 accuracy, significantly better than AdamW (0.223 ± 0.002, $p=0.015$ ).
ResNet (Tiny ImageNet): PsiLogic achieved 0.222 ± 0.001, a numerical improvement over AdamW (0.219 ± 0.005) but not statistically significant ( $p=0.44$ ). It significantly outperformed standard Adam (0.172 ± 0.004, $p=0.001$ ).
Diffusion (CelebA 64²): Validation MSE was statistically tied with Adam/AdamW ( $p=0.49$ ).
Overhead: Peak GPU memory is comparable. However, wall-clock time overhead is 1.2–1.8× on transformer-heavy tasks (e.g., 1.79× on ViT). The authors attribute this to implementation limitations (sequential PyTorch ops, lack of fused CUDA kernels) rather than theoretical cost.

5. Significance and Claims

The paper positions PsiLogic as a method that implicitly adapts to training stability without manual warmup schedules.

Implicit Warmup: The chaos term acts as a dynamic dampener, suppressing destructive updates during the volatile early phase and fading as the model stabilizes.
Modest Claims: The authors explicitly state they do not claim universal dominance. They report:
- No significant gain over AdamW on ResNet or Diffusion tasks.
- Significant step-time overhead in the current implementation.
- Limitations regarding the small seed count (3) and short training budget (2000 steps), noting that results have not yet been replicated by external groups.
Reproducibility: The paper emphasizes transparency by releasing all raw data and admitting non-significant or negative results, aiming to set a standard for fair optimizer evaluation via FairBench.

In conclusion, PsiLogic demonstrates that a chaos-aware damping term can improve convergence and stability in specific domains (NLP, ViT) by dynamically adjusting the effective step size based on gradient volatility, offering a complementary approach to existing parameter-free or hypergradient methods.