Technical Summary: Generator-Aligned Representation Interfaces for Diagnostic Soft Equivariance

1. Problem Statement

Existing equivariant architectures (e.g., Group CNNs, Steerable Networks, SE(3)-Transformers) enforce exact equivariance by encoding specific group actions directly into their operators. While this provides strong theoretical guarantees, it creates a "design rigidity" where adapting to a new group, a generic backbone, or a different data modality requires redesigning the operator and representation structure simultaneously. Conversely, modern generic sequence backbones (e.g., Transformers, State-Space Models) offer high portability across modalities (images, genomics, point clouds) but lack inherent mechanisms to ensure that input transformations induce corresponding transformations in the internal representation.

This paper identifies an integration gap: How can group structure be exposed to a generic backbone without redesigning every layer as a hard-equivariant operator? The authors argue that enforcing exact equivariance in every layer is often unnecessary and restrictive, proposing instead a diagnostic approach to learn "soft equivariance."

2. Methodology: Generator-Aligned Representation Interface (GARI)

The paper introduces GARI, a representation-level design principle that exposes selected transformation generators to a generic sequence backbone through aligned views. Rather than hard-coding the group law into operators, GARI constructs a trainable interface that makes generator structure visible, learnable, and measurable.

Core Components

The framework is instantiated as GARI-Net, a sequence-based model with the following architectural features:

1. Generator-Indexed Streams: For a set of exposed generators $S = \{s_1, \dots, s_q\}$, the model constructs $q+1$ streams: one canonical stream ($P_\theta(x)$) and $q$ generator-induced streams ($\rho_{s_j}P_\theta(x)$). These streams represent the input in different transformation-aligned frames.
2. Shared Backbone Processing: All streams are processed by a shared sequence backbone (e.g., Mamba2, Transformer) with shared parameters. This ensures that the model learns a consistent function family across views rather than an ensemble of independent networks.
3. Frame Conversion and Context Repair:
   • ODBC (One-dimensional Backward Convolution): Applied to each stream to repair local context mismatches caused by reindexing (e.g., sequence reversal), particularly for directional or autoregressive operators.
   • Directional Frame Conversion: Streams are temporarily converted from the canonical interaction frame to generator-specific processing frames and back, ensuring alignment before cross-stream interaction.
4. Cross-Stream Interaction: A "leave-one-out" attention mechanism allows each stream to attend to the others, enabling the exchange of evidence between the canonical and generator-aligned views without treating them as independent networks.
5. Discrepancy-Aware Fusion: The terminal fusion block avoids blind averaging. Instead, it uses a pairwise-discrepancy signal (measuring disagreement between streams) and a stream-sum signal to weight the aggregation. This allows the model to react to inter-stream disagreement, preserving informative variations.

Diagnostic Framework: Soft Equivariance and DEE

The paper redefines the goal from exact equivariance to soft equivariance, measured by the residual error of the transformation relation under specific data and transformation distributions.

• Soft Equivariance Residual: Defined as the expected squared distance between the representation of a transformed input and the prescribed transformation of the canonical representation: $E_{\mu,\nu} = E[\Delta(F(\alpha_g x), \tau_g F(x))^2]$.
• Direct Equivariance Error (DEE): A finite-probe diagnostic metric computed on frozen checkpoints. It compares the representation of a transformed input against the prescribed transformation of the canonical representation using known token or voxel reindexing. DEE is not a training loss but a post-hoc diagnostic to measure representation-level consistency.
• Residual Localization: The framework decomposes the total error into three localized sources:
  1. Interface Residual: Mismatch in converting transformed data to indexed representations.
  2. Stream Residual: Incompatibility in shared backbone processing.
  3. Fusion Residual: Mismatch in aggregating multiple views.

3. Key Contributions

1. Generator-Aligned Representation Interfaces (GARI): A modality-agnostic design primitive that shifts the symmetry-design unit from a group-specialized operator to a reusable interface. It exposes generator structure to generic backbones via aligned canonical and transformed views.
2. Residual-Based Soft Equivariance and Finite Measurement: The authors define soft equivariance via the residual of the transformation relation and instantiate its finite-probe estimate as Direct Equivariance Error (DEE). They distinguish this from task robustness and exact certification.
3. GARI-Net Realization: A concrete sequence-model implementation featuring generator-indexed streams, shared processing, ordering-aware context repair, cross-stream interaction, and discrepancy-aware fusion.
4. Cross-Domain Evidence: The interface principle is validated across three distinct domains: genomic sequences (reversal), 2D images (planar rotations), and 3D point clouds (axial SO(3) transfer).

4. Experimental Results

The experiments are designed as a diagnostic chain to test the interface principle under controlled held-out transformations, rather than to establish a universal SOTA leaderboard.

• Genomic Sequences (GenomicBenchmarks):
  • Task: Sequence reversal ($Z_2$).
  • Result: GARI-Net achieved the highest average accuracy on reversed inputs (0.627) compared to Caduceus-PH (0.612) and Caduceus-PS (0.610), despite having slightly lower accuracy on normal inputs. This demonstrates that the generator-aligned interface better preserves task-relevant information under sequence reversal without training on reversed samples.
• Planar Images (ImageNet-1K & MNIST):
  • Task: Cyclic rotations ($C_2$ and $C_4$).
  • Result: On ImageNet-1K, GARI-Net-C4 improved pure-generalization rotation accuracy (angles outside training augmentation) by 1.34 percentage points over a matched Vision Mamba baseline. It reduced the C4 DEE from 0.983 to 0.831. On MNIST, broader $C_4$ generator coverage outperformed $C_2$ configurations on sampled-orbit generalization, improving orbit-average accuracy by 2.61 points.
• 3D Point Clouds (ModelNet40):
  • Task: Controlled axial transfer ($SO(3)$).
  • Result: Strengthening the $x/y$-axis interface from $C_2$ to $C_4$ increased held-out $z$-axis accuracy by 8.97 points with minimal change in clean accuracy. An explicit $x/y/z$ reference added only 0.51 points, localizing the primary gain to the transfer through the strengthened $x/y$ interface rather than explicit $z$-axis exposure.

5. Significance and Claims

The paper positions GARI as a portable, diagnostic complement to hard-equivariant architectures, not a replacement.

• Modesty of Claims: The authors explicitly state that their findings are finite and distribution-dependent. They do not certify exact equivariance over a continuous group, nor do they claim transformation robustness is equivalent to equivariance.
• Core Value: The significance lies in making generator structure accessible, learnable, and measurable within generic neural models. It offers a route to expose Lie-group generator structure to general sequence models without the computational and architectural overhead of redesigning operators for every new group or modality.
• Diagnostic Nature: The framework provides a "falsifiable" contract. If the interface fails to improve held-out accuracy or DEE on declared probes, the claim of soft equivariance is weakened. The paper demonstrates that this interface supports task-relevant transformation consistency and generalization to held-out probes without requiring group-specific redesign of the backbone.

In summary, GARI provides a flexible architectural pattern that bridges the gap between the rigidity of hard-equivariant operators and the portability of generic sequence models, validated through a rigorous diagnostic protocol of representation residuals and held-out task performance.