Tunable Soft Equivariance with Guarantees

1. Problem Statement

Equivariance is a desirable property in computer vision models where applying a transformation (e.g., rotation, translation) to the input results in a predictable transformation of the output. While strict equivariance is theoretically sound, it often limits model expressiveness because real-world data rarely satisfies strict symmetry perfectly. Enforcing strict equivariance can degrade performance on tasks where the underlying symmetry is only approximate.

Existing approaches to "soft" equivariance (approximate equivariance) face two main limitations:

1. Lack of Guarantees: Methods like data augmentation or regularization do not provide theoretical bounds on the resulting equivariance error.
2. Architectural Rigidity: Methods that trade off expressiveness and equivariance (e.g., residual mixing) often require specialized architectures that cannot be easily adapted to off-the-shelf, pre-trained foundation models (like ViT or ResNet).

The authors aim to create a general, tunable framework that allows pre-trained models to transition smoothly between strictly equivariant and fully non-equivariant behaviors while providing theoretical guarantees on the equivariance error.

2. Methodology

The core contribution is a framework that constructs soft equivariant layers by projecting model weights into a designed subspace. This is inspired by signal processing concepts where low-pass (blurring) filters induce shift invariance.

A. Theoretical Foundation

The authors define $\eta$-Soft Equivariance as a relative measure of error:
$$\frac{\|F(\rho_X(g)x) - \rho_Y(g)F(x)\|}{\|J_F(x)\|_F \|x\|} \leq \eta$$
where $J_F$ is the Jacobian. This normalization ensures the error metric is scale-invariant and interpretable.

B. Projection Operators for Continuous Groups (Lie Groups)

For continuous groups (e.g., $SO(2)$, $SO(3)$), the method utilizes the Lie Algebra representation ($d\rho$) to quantify deviation from equivariance.

1. Taylor Expansion: The equivariance error is approximated using a first-order Taylor expansion around the identity element of the group.
2. Singular Value Decomposition (SVD):
   • For Invariant layers: The Lie algebra representation $d\rho(A)$ is decomposed. A projection operator $B_{inv}$ is constructed by retaining singular vectors corresponding to small singular values (below a cutoff $b$). This effectively "blurs" the weights in directions highly sensitive to the group action.
   • For Equivariant layers: A matrix $L$ is constructed using Kronecker products of input and output Lie algebra representations. The projection operator $B_{eq}$ filters out components where the singular values exceed the cutoff $b$.
3. Schur Decomposition (Efficiency): To avoid the high computational cost of SVD on large matrices ($O((dd')^3)$), the authors propose using Schur decomposition for groups with normal Lie algebra representations (like rotations). This reduces complexity to $O(\max(d, d')^3)$ and allows for efficient block-wise projection.

C. Projection Operators for Discrete Groups

For discrete groups, the Lie algebra approach is inapplicable. The authors introduce a forward-difference operator ($\Delta_s$) as a discrete analogue to the Lie algebra. They derive a first-order Taylor approximation for discrete groups and apply similar projection logic based on the magnitude of these difference operators.

D. Tunability

The degree of "softness" is controlled by a single hyperparameter: the cutoff value $b$.

• Small $b$: Aggressive projection $\rightarrow$ Stronger equivariance bias, lower error, potentially lower expressiveness.
• Large $b$: Weak projection $\rightarrow$ Weaker equivariance bias, higher expressiveness, higher error.
  This allows users to systematically tune the trade-off based on validation performance.

3. Key Contributions

1. General Framework: A novel, architecture-agnostic method to construct soft equivariant layers by projecting weights of any pre-trained model without adding learnable parameters.
2. Theoretical Guarantees: Derivation of explicit upper bounds ($\eta_b$) on the equivariance error, linking the cutoff value $b$ to the group's properties (injective radius, number of generators).
3. Efficient Implementation: Introduction of a Schur decomposition-based projection method that is significantly faster than SVD for large layers, making it feasible for modern foundation models.
4. Empirical Validation: Extensive experiments across classification, segmentation, and trajectory prediction demonstrating that soft equivariance can improve performance while reducing error.

4. Experimental Results

The method was evaluated on three tasks using various backbones (ViT, DINOv2, ResNet, SegFormer).

• Image Classification (CIFAR-10/100, ImageNet):
  • On ImageNet, the proposed method improved top-1 accuracy (e.g., +0.61% on ViT, +1.04% on DINOv2) while simultaneously reducing equivariance error compared to the baseline.
  • It outperformed the "Canonicalizer" baseline (which attempts to rotate images to a standard orientation) which often suffered significant accuracy drops.
• Semantic Segmentation (PASCAL VOC):
  • The method improved combined mIoU and reduced equivariance error across ViT, DINOv2, and SegFormer backbones.
• Human Trajectory Prediction (ETH/UCY):
  • Applied to an autoregressive transformer, the method achieved better combined ADE/FDE metrics than both standard non-equivariant and strict-equivariant baselines.
• Synthetic O(5) Invariant Regression:
  • Demonstrated superior performance in learning invariant functions with varying degrees of noise/deviation from strict invariance.
• Ablation Studies:
  • Confirmed that Schur decomposition is orders of magnitude faster than SVD for computing projection operators.
  • Showed that smooth cut-off (soft thresholding) performs better than hard thresholding.

5. Significance

This paper bridges the gap between the theoretical benefits of equivariance and the practical realities of deep learning on real-world data.

• Practicality: Unlike previous methods requiring architectural changes, this approach acts as a "plug-and-play" layer for existing pre-trained models.
• Control: It provides a principled, mathematically grounded knob (the cutoff $b$) to control the trade-off between symmetry and expressiveness.
• Performance: Crucially, it demonstrates that soft equivariance is not just a compromise; in many cases (especially ImageNet), it yields better performance than both strict equivariant and non-equivariant models, suggesting that real-world data benefits from approximate rather than exact symmetry.

The work establishes a new standard for incorporating symmetry priors into foundation models, offering a path to more robust and generalizable vision systems.