🤖 machine learning

Approximate Equivariance via Projection-based Regularisation

This paper introduces a projection-based regularisation method that penalises non-equivariance at the operator level across the full group orbit, offering a more efficient and effective alternative to existing sample-based approaches for training approximately equivariant neural networks.

Original authors: Torben Berndt, Jan Stühmer

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Torben Berndt, Jan Stühmer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to recognize objects. You want the robot to understand that a coffee cup is still a coffee cup whether it's sitting upright, tilted, or lying on its side. In the world of machine learning, this ability to recognize patterns regardless of how they are rotated or shifted is called equivariance.

For a long time, scientists built robots with "hard-coded" rules to ensure this symmetry. It was like building a car with a steering wheel that only turns left or right, never allowing the driver to drift. This made the car very safe and efficient at following the rules, but sometimes it was too rigid. If the road had a slight curve that didn't fit the rules perfectly, the car struggled.

Recently, engineers started building cars without these hard rules, letting the driver (the AI) figure it out. These cars are faster and more flexible, but they sometimes get confused by rotations they haven't seen before.

This paper introduces a new way to drive: Approximate Equivariance via Projection-Based Regularisation.

Here is the simple breakdown of what they did:

The Problem: The "Sample-Based" Approach

Previously, if you wanted a flexible car that still respected some symmetry, you used a method called "sampling." Imagine you are trying to teach the robot to be rotationally aware. You would take a picture of a cup, rotate it 10 different random angles, show it to the robot, and say, "Hey, this is still a cup!"

The problem? This is slow. It's like asking a student to take a quiz 10 times just to learn one concept. The computer has to do a lot of extra work (forward passes) for every single image, which slows everything down.

The Solution: The "Projection" Approach

The authors propose a smarter way. Instead of asking the robot to guess the answer for 10 different angles, they give the robot a mathematical mirror.

Think of the robot's brain (the neural network) as a giant, messy room full of furniture (data and weights).

The Old Way: You walk around the room, pick up a chair, rotate it, put it down, pick it up again, rotate it differently, and check if it looks right. This takes forever.
The New Way: You have a magical projector. You shine a light on the whole room at once. The projector instantly separates the room into two piles:
- Pile A: Everything that is perfectly symmetrical (the "good" furniture).
- Pile B: Everything that is messy and asymmetrical (the "bad" furniture).

The paper's method calculates exactly what "Pile B" looks like and gently pushes the robot to get rid of it. It doesn't need to check 10 different angles; it does a single, precise mathematical calculation (a "projection") to see how far the robot is from being perfectly symmetrical and nudges it back on track.

Why is this a big deal?

Speed: Because they aren't checking random angles one by one, the robot learns much faster. The paper shows that in tasks like removing metal artifacts from CT scans (think of cleaning up a blurry X-ray caused by a metal implant), their method is 50% to 60% faster than the old sampling methods.
Flexibility: The robot isn't forced to be perfectly symmetrical. Sometimes, real-world data isn't perfect (maybe a kidney is slightly different from its twin). The method allows the robot to be "mostly" symmetrical but break the rules just enough to fit the weird data. It's like a dance instructor who tells you to keep your rhythm but lets you improvise a step if the music changes.
Universal: This works for simple rotations (like turning a square) and complex, continuous rotations (like spinning a sphere in 3D space).

The "Magic" Math

The paper uses a concept called Fourier Space (which is like translating a song from lyrics into sheet music). In this "sheet music" version of the data, the math becomes very simple. The authors show that to make the robot symmetrical, you just need to zero out certain notes (the messy parts) and average others. It's like editing a song by muting the off-key notes and smoothing out the rest, rather than re-recording the whole song 10 times.

Real-World Tests

The team tested this on three main things:

Toy Problems: They made a robot learn to recognize shapes that were slightly wobbly. The robot learned to ignore the wobble when it should, but pay attention to it when the wobble was the actual clue.
Smoke Simulations: They predicted how smoke moves in a room. Even when the smoke didn't move perfectly symmetrically (due to wind or obstacles), their method predicted the future better than rigid models.
Medical Imaging: They cleaned up CT scans with metal implants. Their method produced clearer images than the previous "sampling" methods and did it much faster, allowing for larger batches of images to be processed at once.

The Bottom Line

This paper gives us a new tool to build AI that is smart enough to know the rules, but flexible enough to break them when necessary, all while running much faster than before. It's like upgrading from a robot that has to check a map 10 times to find a route, to a robot that can instantly see the whole map and pick the best path in one glance.

Technical Summary: Approximate Equivariance via Projection-Based Regularisation

Problem Statement

Equivariance serves as a powerful inductive bias in machine learning, enhancing generalization and physical consistency in domains such as chemistry, physics, and robotics. However, strict equivariance imposes significant constraints and computational costs. Recent trends, exemplified by models like AlphaFold-3, have shifted toward non-equivariant architectures to leverage more flexible feature parameterizations and broader search spaces, which can sometimes yield better minima. Furthermore, real-world data often exhibits imperfect or broken symmetries that strict equivariant models cannot capture.

Existing approaches to "approximate equivariance" generally fall into two categories:

Sample-based methods: These penalize deviations from equivariance using randomly sampled group transformations (e.g., data augmentation or pointwise penalties). While flexible, they incur high sample complexity and computational overhead due to the need for extra forward passes per sample.
Explicit parameterization: These models explicitly structure the network as a sum of equivariant and non-equivariant components (e.g., residual pathways). This often leads to double parameterization of the equivariant part and increased model complexity.

There is a need for a method that strikes a middle ground: retaining the benefits of equivariance without incurring its full constraints or the runtime costs of sample-based regularizers, while effectively handling imperfect symmetries.

Methodology

The authors propose Projection-Based Regularisation, a framework that regularizes neural networks toward equivariance at the operator level rather than the sample level. The core idea relies on the orthogonal decomposition of a network's linear layers into equivariant and non-equivariant components.

Theoretical Framework

Let $H$ be a hypothesis class of linear maps between $G$ -modules. The authors utilize the G-smoothing operator (Reynolds operator), denoted as $P(T)$ , which projects a linear map $T$ onto the subspace of $G$ -equivariant maps:
$P(T) = \int_G \pi'(g)^* T \pi(g) \, d\lambda(g)$
where $\lambda$ is the Haar measure of the compact group $G$ .

According to Lemma 2.1, any map $T$ can be uniquely decomposed into an equivariant component $P(T)$ and a non-equivariant (anti-symmetric) component $T - P(T)$ . The authors define the equivariance defect $E(T)$ as the worst-case deviation from equivariance. They prove (Lemma 3.2) that the distance to the equivariant subspace, $\|T - P(T)\|_\mu$ , is quantitatively equivalent to the equivariance defect $E(T)$ , bounded by constants.

The Regularization Objective

The learning objective for a task-specific loss $L_{task}$ is modified to:
$L = L_{task}(T) + \lambda_G \|P(T)\|_\mu^2 + \lambda_\perp \|T - P(T)\|_\mu^2$
Here, $\lambda_\perp$ penalizes the non-equivariant component, encouraging the model to align with the equivariant subspace. $\lambda_G$ controls the regularization of the equivariant part.

Efficient Computation via Fourier Space

A key challenge is computing $P(T)$ efficiently, especially for continuous groups (e.g., $SO(n)$) where the integral in the spatial domain is intractable. The authors leverage the Peter-Weyl theorem to perform this projection in the spectral (Fourier) domain:

Decomposition: Linear operators are decomposed into frequency blocks corresponding to irreducible representations (irreps) of the group.
Block-Diagonalization: By Schur's Lemma, equivariant maps are block-diagonal in the frequency domain (mixing only equivalent irreps).
Projection: The projection $P(T)$ $P (T)$ is computed by:
- Zeroing out all off-diagonal blocks (where $\pi \neq \sigma$ ).
- Averaging within the diagonal blocks to project onto the intertwiner space.
Reconstruction: The inverse Fourier transform yields the projected weights in the spatial domain.

This approach allows for exact, closed-form projection without sampling, applicable to both finite groups (via direct averaging) and continuous groups (via spectral masking).

Key Contributions

Theoretical Regularization: A theoretically grounded approach to regularize general machine learning models toward exact equivariance by penalizing the non-equivariant component of linear operators.
Operator-Level Penalty: Unlike sample-based methods that penalize pointwise deviations, this method penalizes non-equivariance over the entire group orbit at the operator level.
Spectral Efficiency: A framework for computing the projection exactly and efficiently in the Fourier domain, enabling application to continuous groups like $SO(n)$ without the overhead of sampling.
Empirical Validation: Demonstration that the method achieves competitive task performance with existing approximate-equivariance approaches while offering substantial runtime gains.

Experimental Results

The authors evaluated the method across three distinct experimental settings:

Learned SO(2) Invariance (Toy Problem):
- On a binary classification task with point clouds, tuning $\lambda_\perp$ successfully controlled the degree of learned invariance.
- When the target function included angular perturbations (breaking symmetry), the regularized model adapted to fit the data while maintaining invariance in regions where symmetry held, demonstrating a tunable bias.
Imperfectly Symmetric Dynamical Systems:
- Applied to smoke advection-diffusion simulations with broken translation, rotation, and scaling symmetries.
- The projection-based regularizer (+Reg) consistently improved generalization (Future and Domain settings) compared to baselines (Standard CNN, Equivariant CNN, RPP, Lift) in rotation and scaling regimes, while preserving performance where symmetries were already well-aligned.
CT-Scan Metal Artifact Reduction (MAR):
- Compared against sample-based regularizers (Bai et al., 2025) and Residual Pathway Priors (RPP) on the AAPM CT-MAR dataset.
- Performance: The projection-based method achieved competitive or superior reconstruction quality (PSNR/SSIM) across all tested architectures (ACDNet, DICDNet, OSCNet).
- Efficiency: The method significantly outperformed sample-based regularizers in throughput. Because it avoids extra forward passes for group sampling, it allowed for larger batch sizes (12 vs. 4 for sample-based) and reduced epoch times by 42–61%. It also required fewer parameters and less memory than RPPs.
MedMNIST (3D Biomedical Classification):
- Evaluated on 3D tasks (Nodule, Synapse, Organ) with varying degrees of rotational symmetry.
- The method achieved high accuracy, often matching or exceeding Partially Equivariant SCNNs (P-SCNN) and RPPs.
- Efficiency: It demonstrated the highest training and inference throughput and the lowest epoch time among all compared models, with significantly fewer parameters than RPPs.

Significance and Claims

The paper claims that projection-based regularisation offers a favorable accuracy-efficiency trade-off. By shifting the regularization from the sample level to the operator level via spectral projection, the method:

Eliminates Estimator Variance: It computes the penalty exactly without sampling noise.
Reduces Computational Overhead: It avoids the extra forward passes required by sample-based methods, enabling larger batch sizes and faster training/inference.
Remains Architecture-Agnostic: The general recipe can be applied to any model with linear layers, provided the projection can be derived (which is straightforward for linear operators).
Handles Imperfect Symmetries: It allows models to learn the appropriate degree of equivariance, relaxing strict constraints when data distribution demands it, without sacrificing the inductive bias where it is beneficial.

The authors conclude that this approach provides a robust, scalable alternative to both strict equivariant architectures and computationally expensive sample-based regularization, particularly for continuous symmetry groups.