Measuring the Symmetry--Data Exchange Rate

Problem Statement

Equivariance theory predicts that incorporating an architectural symmetry prior (invariance or equivariance under a group $G$) reduces sample complexity by a factor of $|G|$. While widely cited, this prediction is rarely measured as a scaling law with controls that isolate the structural prior from confounding factors such as reduced effective capacity, implicit data augmentation, or regularization. Most empirical studies report accuracy gaps at fixed sample sizes or efficiency ratios at single operating points, failing to trace the sample-complexity scaling law or separate "alignment" (the correct symmetry) from "constraint" (any constraint of similar strength). The paper addresses this methodological gap: how to convert the qualitative claim "structure helps" into a calibrated, falsifiable exchange rate between symmetry and data.

Methodology

The authors construct a controlled, synthetic 2-D task where the symmetry group is known exactly and its order $n$ is a free parameter.

• The Task: Inputs are sampled from an annulus and labeled based on an alternating angular pattern $1[\cos(n\theta) > 0]$. The symmetry group is the cyclic group $C_n$ (order $n$), with $n \in \{1, 2, 3, 4, 6, 8, 12\}$. The task is designed to be non-trivial, with no coordinate-based shortcuts.
• Model Families: Five model families share an identical fully-connected architecture (1,185 parameters) to ensure matched capacity:
  1. Equivariant (Treatment): Uses orbit pooling (averaging outputs of $n$ rotated copies) to enforce $C_n$ invariance by construction.
  2. Wrong-Group (Control): Uses the same orbit pooling architecture but with rotation angles misaligned by an irrational factor ($0.7\times$ the correct period). This isolates "alignment" from "constraint" by keeping orbit size and compute identical while destroying the symmetry match.
  3. Augmented: An unconstrained network trained on all $n$ rotated copies (averaging logits in the loss) but using a single-input forward pass at test time.
  4. Vanilla: A standard unconstrained network.
  5. Regularized: A vanilla network with L2 weight decay matched to the degrees of freedom removed by orbit pooling.
• Metric: The study measures $N_{target}$, the minimum sample size required for 3 out of 5 seeds to reach 80% validation accuracy.
• The Estimator: To address the confound that task difficulty increases with $n$ (making absolute slopes positive for all models), the authors introduce a relative exchange rate $\beta_{diff}$. This is the slope of $\log_2(N_{vanilla}^{target} / N_{treatment}^{target})$ against $\log_2(n)$. Theoretically, if the symmetry reduces sample complexity by $|G|$, this slope should be $+1$.
• Inference: The study employs a non-parametric pairs bootstrap (10,000 resamples) and a joint pairwise bootstrap for treatment-vs-control comparisons. A pre-specified failure taxonomy classifies outcomes (e.g., no advantage, regularization collapse, augmentation explanation, or genuine signal).

Key Contributions

1. A Confound-Free Measurement Methodology: The introduction of the relative exchange-rate estimator, which cancels the shared task-difficulty scaling, allowing the theoretical $|G|$ prediction to be tested. This is paired with a "wrong-group" control to distinguish alignment from generic constraint and a joint pairwise bootstrap to handle overlapping confidence intervals.
2. The Estimated Rate: On the controlled task, the equivariant model yields a point estimate of $\beta_{diff} = 1.28$. This is consistent with the theoretical prediction of $1.0$ in sign and order of magnitude. However, the authors emphasize that the precise numerical value is exploratory; the qualitative direction is the robust finding.
3. Isolating Alignment from Constraint: The "wrong-group" control (misaligned orbit of identical size) is found to be worse than no constraint at all ($\beta_{diff} \approx -0.77$), demonstrating that misaligned structural priors are actively harmful, not merely unhelpful.
4. Architecture vs. Augmentation: The study finds that training-time-only augmentation (without test-time orbit averaging) fails to reach the target accuracy where the equivariant model succeeds. However, a CPU replication shows that adding test-time orbit averaging to the augmented model makes it bit-identical to the equivariant model. Thus, the gap is conditional on asymmetric test-time computation, not an unconditional superiority of architecture.
5. Reproducibility: The full experiment (5,600 runs) is released with code, configuration hashes, and all per-run records, running in ~90 minutes on a single GPU.

Results

• Exchange Rate: The primary estimator yields $\beta_{diff} = 1.28$.
  • The single-level percentile bootstrap CI is $[+0.92, +2.05]$ (excluding zero).
  • The more conservative two-level bootstrap (resampling seeds and group sizes) yields a CI of $[-0.63, +1.72]$, which includes zero.
  • A finer-$N$ replication on a $\sqrt{2}$-spaced grid produced an inconclusive result ($\beta_{diff} = -0.82$, wide CI), highlighting sensitivity to grid discretization.
• Controls:
  • The Wrong-Group control is significantly worse than the Vanilla baseline in joint pairwise comparisons (CI $[+0.79, +3.26]$ for the difference), confirming that the benefit comes from correct alignment, not just the presence of a constraint.
  • The Regularized baseline shows no advantage over Vanilla ($\beta_{diff} \approx -0.04$), ruling out effective degrees-of-freedom reduction as the primary mechanism.
  • The Augmented model (train-time only) fails to reach the target accuracy for $n \ge 3$, while the equivariant model succeeds.
• Robustness: The relative rate remains stable (retaining 88–97% of its clean value) when up to 30% of training labels are corrupted with asymmetric noise.

Significance and Claims

The paper explicitly frames itself as an exploratory study, not a confirmatory measurement. The authors are modest about their claims:

• No Universal Constant: The reported rate is an operational estimate conditional on a specific task, target accuracy, and scaling model, not a universal information-theoretic constant.
• Qualitative Direction: The primary contribution is the qualitative direction (positive, order-of-magnitude agreement with theory) and the wrong-group finding (misaligned constraints are harmful). The precise numerical value ($1.28$) is not claimed as a definitive measurement due to the exploratory nature of the estimator adoption (post-hoc) and the inclusion of zero in the conservative confidence interval.
• Methodological Transfer: The framework (relative-rate estimator, wrong-group control, joint pairwise bootstrap, failure taxonomy) is presented as a transferable tool for measuring the strength of any parameterizable inductive bias, not just symmetry.
• Limitations: The study acknowledges that the task is synthetic and 2-D, the symmetry is exact (not approximate or latent), and the comparison to theory involves a mismatch between kernel/random-feature theory and finite-width ReLU MLPs. The "FLOP-neutral" claim is also nuanced: while total training FLOPs are roughly matched, the equivariant model incurs higher per-sample compute and memory overhead.

The authors conclude that the "wrong-group" result is the cleanest and most confident finding, while the headline number requires a future, externally registered replication on fresh seeds to be considered confirmatory.