Scaling Laws and Symmetry, Evidence from Neural Force Fields

This paper demonstrates that incorporating task symmetries through equivariant architectures, particularly those with higher-order representations, significantly improves scaling laws for interatomic potentials, suggesting that fundamental inductive biases should be explicitly engineered rather than left for models to discover as they scale.

The Gist

Imagine you are trying to teach a robot how to predict how atoms in a molecule will move and interact. This is a bit like teaching a child to understand how a complex Lego structure holds together. You can give the robot two different types of instruction manuals:

1. The "Blind" Manual: You just show the robot millions of pictures of Lego structures and say, "Figure out the rules yourself." The robot has to learn everything from scratch, including the fact that if you rotate the whole structure, the physics don't change.
2. The "Symmetry" Manual: You give the robot a manual that explicitly says, "Hey, remember, if you spin this structure, it's still the same structure. If you flip it, the rules stay the same." You bake the laws of physics (symmetry) directly into the robot's brain.

For a long time, many researchers believed in the "Blind" approach. They thought that if you just gave the robot enough data and enough computing power (a "bigger brain"), it would eventually figure out the symmetry rules on its own. They believed that explicitly teaching the rules was unnecessary and that a simple, flexible model would eventually catch up.

This paper says: "Actually, no. The 'Symmetry' manual is much better, and the gap gets wider as you get bigger."

Here is the breakdown of their findings using simple analogies:

1. The Race: Speed vs. Efficiency

The researchers ran a race between different types of robot brains (architectures) to see how fast they could learn to predict atomic forces.

• The "Blind" Robots (Unconstrained): These are flexible but inefficient. They have to "re-learn" the fact that a rotated molecule is the same molecule every single time they see it.
• The "Symmetry" Robots (Equivariant): These have the rules of rotation and translation built-in. They don't waste energy re-learning basic physics.

The Finding: When the robots were small, the difference wasn't huge. But as the researchers made the robots massive (scaling up the data and computing power), the "Symmetry" robots didn't just stay ahead; they pulled away dramatically. The "Blind" robots hit a wall where adding more data didn't help them much, while the "Symmetry" robots kept getting smarter and smarter.

2. The "Degree" of Symmetry Matters

Not all "Symmetry" robots are created equal. Some only understand simple rotations (like a flat coin), while others understand complex 3D rotations (like a spinning globe).

• Low-Order Symmetry: Understands basic rules.
• High-Order Symmetry: Understands very complex, detailed rules about how shapes interact in 3D space.

The Finding: The more complex the symmetry rules baked into the robot, the faster it learned. A robot with "High-Order" symmetry learned so much faster that the gap between it and the "Blind" robot became a canyon. It's like comparing a student who knows the alphabet to a student who already knows the grammar and vocabulary of the language; as the book gets thicker, the second student leaves the first one in the dust.

3. The "Bitter Lesson" vs. Reality

There is a famous idea in AI called the "Bitter Lesson," which suggests that we should stop trying to hard-code human knowledge (like symmetry) into AI and just let the AI learn it from raw data because it's cheaper and scales better.

• This paper argues: In the world of atoms and molecules, the "Bitter Lesson" is wrong. If you try to let a model discover symmetry on its own, it's like asking a student to rediscover gravity. It's possible, but it's incredibly inefficient. By the time the student figures it out, the student who was taught gravity is already flying.

4. The "Goldilocks" Balance

The paper also looked at how to spend money (computing power) most efficiently.

• The Old Way: Maybe you should buy a bigger brain (more parameters) or get more textbooks (more data).
• The New Finding: It turns out you need to buy both at the same time. If you double your data, you should also double your model size. This "tandem scaling" works best for all types of robots, but the "Symmetry" robots are just much more efficient at using that combined power.

5. What About "Cheating" with Loss Functions?

Some researchers tried to trick the "Blind" robots by adding a penalty score if they made a mistake about symmetry (e.g., "If you say a rotated molecule is different, you get a bad grade").

• The Finding: This didn't work well. It's like telling a student, "Don't forget the rules," but not actually teaching them the rules. The robot still had to struggle to learn the pattern. It was much better to just build the rule into the robot's brain from the start.

The Bottom Line

If you want to build a super-smart AI to understand molecules, don't just throw more data at a simple, flexible model and hope it figures out the laws of physics. Build the laws of physics directly into the model's design.

As you scale up to massive sizes, the models that respect the fundamental symmetries of the universe (rotation, translation) will not just be slightly better; they will be exponentially more powerful than those that try to learn these rules from scratch. The "Symmetry" approach changes the very nature of the learning curve, making the task easier and the results better.

Technical Summary

Problem Statement
The paper addresses the scaling behavior of Neural Network Interatomic Potentials (NNIPs), which are deep learning models designed to predict quantum mechanical properties (specifically potential energy and atomic forces) of atomistic systems. While recent literature in natural language and vision suggests that scaling laws (power-law relationships between performance and data/parameters/compute) are largely architecture-independent—implying that models can learn necessary inductive biases like symmetry on their own as they scale—this view is contested in geometric domains. The authors investigate whether explicit architectural equivariance (enforcing rotational and permutation symmetries) provides a distinct advantage in scaling laws for NNIPs, or if simpler, non-equivariant models can achieve comparable performance given sufficient compute.

Methodology
The authors conduct a comprehensive empirical study on the OpenMol neutral-molecule dataset (approx. 34M training samples, ~9.2 × 10⁸ tokens). They compare four distinct architectural families representing varying degrees of symmetry constraints:

1. Unconstrained MPNN: A vanilla Message Passing Neural Network processing geometric features (relative positions) without symmetry constraints.
2. Invariant Scalars (GemNet-OC): Uses invariant features (distances, angles, dihedrals) but approximates equivariant functions via edge-based message passing; classified as a 4-body, tensor order $\ell=0$ architecture.
3. Cartesian Vectors (EGNN): An $E(n)$-equivariant GNN using vector channels (tensor order $\ell=1$).
4. High-Order Spherical Tensors (eSEN): An equivariant network utilizing higher-order irreducible representations of the rotation group ($\ell \ge 2$), employing frame alignment to sparsify tensor products.

The study employs a single-epoch training regime to align with theoretical scaling law literature, utilizing scheduler-free AdamW optimizers to mitigate learning rate schedule artifacts. Scaling laws are fitted against three metrics:

• Compute: Both theoretical FLOPs ($C$) and wall-clock training time (GPU-hours, $H$).
• Data: Number of training tokens ($D$).
• Parameters: Model size ($N$).

The authors also investigate the effects of symmetry loss regularization (penalizing deviations from equivariance in non-equivariant models), multi-epoch training with data augmentation, and test-time group averaging.

Key Contributions

1. Architecture-Dependent Scaling Exponents: The paper demonstrates that scaling exponents are not constant across architectures. As the "degree" of equivariance increases (from unconstrained to low-order to high-order), the power-law exponents for data ($\beta$) and parameters ($\alpha$) increase significantly.
2. Superior Scaling of Equivariant Models: Equivariant architectures, particularly those with higher-order tensor representations (eSEN), exhibit steeper scaling curves. This implies that the performance gap between equivariant and non-equivariant models widens as compute and data scale, contradicting the notion that models can simply "learn" symmetry later.
3. Compute-Optimal Allocation: The study finds that for compute-optimal training, model size ($N$) and dataset size ($D$) should scale in tandem ($N \propto D$) across all architectures, mirroring findings in language modeling (Chinchilla scaling). However, the constant of proportionality and the resulting loss reduction differ based on the architecture's symmetry bias.
4. Inefficacy of Symmetry Loss: Enforcing symmetry through a loss term (regularization) in unconstrained models does not yield the same scaling benefits as building equivariance into the architecture. While it improves data efficiency slightly, it fails to match the scaling exponents of native equivariant models.
5. Multi-Epoch and Augmentation Insights: In low-data, multi-epoch settings, data augmentation is required for unconstrained models to prevent overfitting and recover power-law scaling. However, even with augmentation, unconstrained models do not match the scaling exponents of equivariant models.

Results

• Scaling Laws: The validation loss follows a power law $L \propto C^{-\gamma}$. The exponent $\gamma$ increases with architectural complexity:
  • Unconstrained MPNN: $\gamma \approx 0.14$
  • EGNN: $\gamma \approx 0.17$
  • GemNet-OC: $\gamma \approx 0.25$
  • eSEN (High-order): $\gamma \approx 0.40$
• Data and Parameter Scaling:
  • Data scaling exponents ($\beta$) range from 0.31 (Unconstrained) to 0.75 (eSEN).
  • Parameter scaling exponents ($\alpha$) range from 0.28 (Unconstrained) to 0.82 (eSEN).
• Symmetry Loss: Adding a symmetry loss term to an unconstrained model increases the data exponent ($\beta$) slightly but decreases the parameter exponent ($\alpha$), resulting in no net gain in the compute-optimal frontier slope compared to the unconstrained baseline.
• Depth: For equivariant models, optimal network depth increases with the order of rotation representation, whereas unconstrained models suffer from over-smoothing at higher depths.

Significance and Claims
The paper argues that, contrary to the "bitter lesson" hypothesis (which suggests models should learn inductive biases from data), explicit architectural symmetry is critical for scaling in geometric tasks. The authors claim that symmetry is not merely a data-reduction technique but fundamentally alters the inherent difficulty of the task and its scaling laws.

The primary significance lies in the finding that higher-order equivariant representations translate to better scaling exponents. This suggests that for large-scale NNIPs, investing in complex, symmetry-aware architectures (like eSEN) is more effective than scaling up simpler, non-equivariant models. The authors conclude that fundamental inductive biases like symmetry should be encoded in the architecture rather than left for the model to discover, as they change the scaling trajectory itself.

The paper remains modest regarding its scope, noting limitations such as the focus on single-epoch training, the specific dataset used (neutral molecules), and the exclusion of denoising pretraining strategies used in other recent works. It calls for future theoretical work to explain why symmetry changes scaling exponents and suggests extending these studies to more diverse molecular types and multi-epoch regimes.