🤖 machine learning

Activation-Free Backbones for Image Recognition: Polynomial Alternatives within MetaFormer-Style Vision Models

This paper introduces PolyNeXt, a family of activation-free vision backbones that replace standard nonlinearities with polynomial alternatives using Hadamard products within MetaFormer-style architectures, achieving state-of-the-art performance in image recognition and segmentation tasks while reducing computational costs.

Original authors: Jeffrey Wang, Jonathan Gregory, Grigorios G. Chrysos

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jeffrey Wang, Jonathan Gregory, Grigorios G. Chrysos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a super-smart robot that can look at a picture and tell you exactly what it sees. For years, the engineers building these robots have relied on a specific set of "magic switches" called activation functions (like ReLU or GELU). These switches act like traffic lights or gates; they decide which information gets to pass through the robot's brain and which gets blocked. The industry standard has been that without these specific gates, the robot can't learn complex patterns.

This paper, titled "Activation-Free Backbones for Image Recognition," challenges that long-held belief. The authors, Jeffrey Wang and colleagues, say: "Actually, you don't need those specific traffic lights. You can build a smarter, more efficient robot using a different kind of math entirely."

Here is a simple breakdown of what they did and why it matters, using everyday analogies.

1. The Problem: The "Traffic Light" Bottleneck

In modern AI, the robot's brain is made of layers. Information flows through these layers, and at every step, it hits a "traffic light" (an activation function) that turns the signal on or off or changes its shape.

The Old Way: The robot calculates a number, hits a traffic light, and then moves on.
The Issue: The authors argue that these traffic lights aren't the only way to create complexity. In fact, they can be inefficient and sometimes get in the way.

2. The Solution: The "Recipe" Instead of the "Switch"

Instead of using traffic lights, the authors replaced them with polynomials.

The Analogy: Imagine you are baking a cake.
- The Old Way (Activation): You mix ingredients, then stop to check a specific rule (a traffic light) to see if you should add more sugar.
- The New Way (Polynomials): You just mix the ingredients together in a specific mathematical recipe. The complexity comes from how you mix them, not from stopping to check a rule.

Specifically, they replaced the "stop-and-check" steps with Hadamard products.

What is that? Imagine you have two lists of numbers. Instead of adding them up, you multiply them together, element by element.
The Magic: When you multiply two lists of numbers, you automatically create a complex, curved relationship (a polynomial) without needing any special "traffic light" switches. It's like how mixing red and blue paint creates purple naturally, without needing a "purple switch."

3. The Three New Tools

The team built three new tools to replace the old ones in the robot's brain:

PolyMLP: Replaces the standard "feedforward" layer (the part that processes information). Instead of a gate, it multiplies two streams of data together.
PolyConv: Replaces the "convolution" layer (the part that looks for shapes like edges or textures). It mixes different views of the image using multiplication.
PolyAttn: Replaces the "attention" mechanism (the part that decides what parts of the image are important). Instead of using a complex exponential math function (like a steep hill), it uses a smooth polynomial curve.

4. The Challenge: The "Snowball Effect"

There was a big problem with this new approach.

The Analogy: If you multiply two big numbers, you get a huge number. If you do that again in the next layer, you get an enormous number. If you keep doing this for 100 layers, the numbers get so big they explode (like a snowball rolling down a hill getting too heavy to control). This caused the robot's training to crash.

The Fix: The authors invented a "stabilization recipe" to keep the snowball from exploding:

Sigmoid-Scale: They added a tiny "dimmer switch" that automatically turns down the volume if the numbers get too loud.
Multi-Input Skips: They built "bypass roads" that let information from two steps ago skip a layer, ensuring the signal doesn't get lost or distorted.
Deep and Narrow: They found that making the robot's brain very deep (many layers) but narrow (less wide) worked better than the usual wide-and-shallow design.

5. The Results: Faster, Stronger, and More Robust

They tested their new "PolyNeXt" robots on a massive dataset of images (ImageNet).

Performance: Their polynomial robots performed just as well as, or better than, the best robots using the old "traffic light" switches.
Efficiency: They achieved these results with fewer parameters (less brain matter) and less computing power.
Robustness: When they showed the robots blurry, noisy, or weirdly drawn images (out-of-distribution data), the polynomial robots were surprisingly better at guessing correctly than the old ones.
Privacy Potential: Because they removed the complex "exponential" math and division steps, these robots are much closer to being compatible with Fully Homomorphic Encryption (FHE).
- What is FHE? It's a way to do math on encrypted data without ever decrypting it. This is huge for privacy. The old "traffic light" switches were hard to use with encryption; these new polynomial switches are much friendlier to it.

6. The Bottom Line

The paper proves that activation functions are not a fundamental necessity for building powerful image-recognition AI. They are just one solution to an engineering problem. By switching to polynomial math (multiplication instead of gating) and adding a few safety mechanisms to keep the math stable, the authors created a new family of AI models that are:

Just as smart (or smarter) than current models.
More efficient (cheaper to run).
More robust (better at handling weird images).
More privacy-friendly (easier to encrypt).

In short: They took the "traffic lights" out of the robot's brain, replaced them with a clever mixing recipe, and found that the robot actually drove better.

Technical Summary: Activation-Free Backbones for Image Recognition

Problem Statement

Modern vision architectures, including MetaFormer-style backbones, rely heavily on pointwise activation functions (e.g., ReLU, GELU, SiLU) and exponential operations within softmax attention to introduce nonlinearity. While these components enable neural networks to approximate complex functions, they are not the sole source of nonlinearity. The paper posits that these activations are not strictly necessary for competitive performance. Furthermore, the reliance on non-polynomial operations (like exponentials and pointwise activations) creates significant overhead for Fully Homomorphic Encryption (FHE), a privacy-preserving computation paradigm where standard networks incur substantial costs due to their non-polynomial nature. Existing polynomial networks (e.g., MONet, DTTN) have demonstrated potential but often require custom, tightly coupled architectures that limit their practical utility and fail to match the performance of activation-based counterparts at comparable computational costs.

Methodology

The authors propose a framework to replace standard nonlinearities with polynomial alternatives while preserving the input-output interfaces of existing modular vision backbones. This approach allows the integration of polynomial modules into established frameworks like MetaFormer without redesigning the entire architecture.

Core Primitives

The paper introduces three activation-free modules that substitute standard nonlinearities with Hadamard products (elementwise multiplication) of learned linear projections:

PolyMLP (Channel Mixing): Replaces the pointwise activation in feedforward networks with the Hadamard product of two parallel linear projections: $PolyMLP(x) = W_o ((W_a x) * (W_b x))$ . This yields a second-degree polynomial in the input features.
PolyConv (Spatial Mixing): Replaces the activation in separable convolutions by fusing parallel convolutional branches with different receptive fields via elementwise multiplication. Specifically, it combines a "coarse" branch (dilated depthwise convolution) and a "fine" branch (standard depthwise convolution), applying a channel-flip to one branch before fusion to decorrelate features.
PolyAttn (Global Spatial Mixing): Replaces the exponential function in softmax attention with a polynomial kernel: $A = (s \cdot QK^\top + 1)^p$ , followed by $\ell_1$ normalization. This creates an activation-free attention mechanism that remains compatible with structural variants like windowed attention.

Stabilization Techniques

The authors identify that Hadamard products can amplify large values, leading to training instability in deep networks. To enable stable training of networks up to ~200 layers, they introduce:

Sigmoid-Scale: A learnable scalar bounded by a sigmoid function ( $\sigma(\lambda)$ ) that scales residual contributions ( $y = x + \sigma(\lambda)f(x)$ ), preventing unbounded growth.
Multi-Input Skip Connections: Inspired by NASNet, each cell receives inputs from the two preceding cells ( $x_{t-1}$ and $x_{t-2}$ ), improving gradient flow.
Depth-over-Width Design: The authors advocate for stacking narrower layers deeper rather than using wider, shallower configurations. This design leverages the fact that polynomial degree grows exponentially with depth ( $2^L$ for depth $L$ ), maximizing representational capacity without a linear increase in parameters.

Architecture: PolyNeXt

These modules are instantiated within a MetaFormer-style four-stage hierarchical architecture. Two variants are proposed:

CPolyNeXt: Uses PolyConv for spatial mixing in all stages.
APolyNeXt: A hybrid design using PolyConv in high-resolution stages and PolyAttn in low-resolution stages.

Key Results

The models were evaluated on ImageNet-1K classification, out-of-distribution (OOD) robustness, and ADE20K semantic segmentation.

ImageNet-1K Classification: PolyNeXt models match or exceed activation-based MetaFormer instantiations (ConvFormer, CAFormer) across all model scales (Tiny to Large). For instance, CPolyNeXt-S (26M params) achieves 83.9% Top-1 accuracy, outperforming ConvFormer-S18 (83.0%) and prior polynomial networks like DTTN-S (79.4%) by significant margins (2–3 percentage points) with lower computational cost.
Robustness: The polynomial backbones demonstrate superior robustness on ImageNet-C (corruptions), ImageNet-A (adversarial), ImageNet-R (renditions), and ImageNet-Sketch. CPolyNeXt-B (40M params) surpasses ConvFormer-B36 (100M params) on several OOD benchmarks.
Dense Prediction: On ADE20K semantic segmentation, CPolyNeXt-S outperforms both ConvFormer-S18 and CAFormer-S18 by +2.0 and +1.7 mIoU respectively at matched parameter counts.
Fully Polynomial Variants: By replacing LayerNorm with a polynomial-compatible BatchNorm (using running statistics to avoid division), the authors created fully polynomial models. CPolyNeXt-S (BN) achieves 82.7% on ImageNet, a 5.0-point improvement over the best prior fully polynomial model (MONet-T BN) and surpassing the activation-based ConvNeXt-T (82.1%).

Ablation Insights

Activations are Detrimental: Reintroducing activations (e.g., GELU) into the polynomial modules consistently degrades performance. The authors attribute this to the disruption of "mutual gradient coupling" inherent in the Hadamard product, where each branch's gradient is scaled by the other's output.
Stabilization is Critical: Removing Sigmoid-Scale or multi-input skip connections leads to significant performance drops or training collapse.
Recipe Sensitivity: The performance gains are attributed to the module design and stabilization techniques rather than a unique training recipe, as the modules transfer well when swapped into other architectures.

Significance and Claims

The paper claims that activation functions are not a representational necessity for modern vision backbones but rather a solution to optimization challenges. By addressing training stability through specific mechanisms (Sigmoid-Scale, depth-over-width), polynomial interactions can serve as effective sources of nonlinearity.

The primary contributions are:

Polynomial Alternatives: The introduction of PolyMLP, PolyConv, and PolyAttn, which substitute standard nonlinearities with Hadamard products while maintaining compatibility with architectural improvements.
Empirical Evidence: Demonstration that activation-free backbones (PolyNeXt) can match or exceed activation-based counterparts in accuracy and robustness across scales.
Stabilization Recipe: A lightweight set of techniques enabling the training of deep (up to 200 layers) polynomial networks.
FHE Progress: A step toward Fully Homomorphic Encryption-compatible inference by removing non-polynomial operations, including a fully polynomial variant that outperforms prior fully polynomial networks and approaches activation-based baselines.

The authors conclude that while their work does not imply immediate safety or fairness benefits, it opens a path toward privacy-preserving inference and suggests that the field should re-evaluate the necessity of traditional activation functions in deep vision models.