📊 statistics

There Will Be a Scientific Theory of Deep Learning

This paper argues that a scientific theory of deep learning, termed "learning mechanics," is emerging through five converging research strands that focus on the training dynamics, aggregate statistics, and universal behaviors of neural networks to provide falsifiable quantitative predictions.

Original authors: Jamie Simon, Daniel Kunin, Alexander Atanasov, Enric Boix-Adserà, Blake Bordelon, Jeremy Cohen, Nikhil Ghosh, Florentin Guth, Arthur Jacot, Mason Kamb, Dhruva Karkada, Eric J. Michaud, Berkan Ottlik

Published 2026-04-24

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jamie Simon, Daniel Kunin, Alexander Atanasov, Enric Boix-Adserà, Blake Bordelon, Jeremy Cohen, Nikhil Ghosh, Florentin Guth, Arthur Jacot, Mason Kamb, Dhruva Karkada, Eric J. Michaud, Berkan Ottlik, Joseph Turnbull

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master chef who has created the most delicious, complex dish in the world. It feeds millions, solves problems, and changes how we live. But here's the catch: you don't actually know the recipe.

You know that if you mix ingredients A, B, and C in a specific pot and stir it for 10 hours, you get a masterpiece. But if you ask why it works, or what happens if you change the heat by one degree, you can only guess. You've been cooking by trial and error, not by science.

This is the current state of Deep Learning (the technology behind AI). It works incredibly well, but it's a "black box." We know that it learns, but we don't have a unified scientific theory explaining how or why.

This paper, written by a large team of researchers, argues that we are finally on the verge of discovering the "Physics of AI." They call this new field "Learning Mechanics."

Here is the breakdown of their argument, using simple analogies.

1. The Goal: From Alchemy to Physics

Right now, building AI is like alchemy. Practitioners mix ingredients (data, code, settings) hoping for gold (a smart model). If it works, they celebrate; if it fails, they tweak the recipe and try again.

The authors want to turn this into physics.

Physics doesn't just say "this bridge holds up." It explains why using laws of gravity, tension, and material strength. It allows engineers to build bridges they've never seen before with confidence.
Learning Mechanics wants to do the same for AI. It wants to find the "laws of motion" for how a neural network learns, so we can predict exactly what will happen before we even start training.

2. The Five Clues That a Theory Exists

The authors say, "Don't just take our word for it. Look at the evidence." They found five signs that a scientific theory is emerging, similar to how physicists found clues before inventing the theory of gravity:

Clue 1: The "Toy Models" Work.
Just as physicists study a simple "frictionless block" to understand motion, AI researchers have found simplified versions of neural networks (like "linear networks") where the math actually works out perfectly. These simple models reveal the core mechanics that also happen in the complex, real-world AI.
Clue 2: The "Infinite" Trick.
Real AI models are huge (billions of parameters). But physicists often solve problems by imagining a system is infinitely big to see the underlying pattern. Researchers found that if you imagine an AI network is infinitely wide or deep, the chaotic math simplifies into beautiful, predictable laws.
Clue 3: Simple Laws Rule the Chaos.
Despite the complexity, big AI models follow simple rules. For example, there are "Scaling Laws" that predict exactly how much better an AI will get if you give it more data or more computer power. It's like finding that the speed of a car is always directly related to how hard you press the gas pedal, regardless of the car's color.
Clue 4: Untangling the Knobs.
AI has hundreds of "knobs" (settings) to turn. The authors found that these knobs aren't all independent. They can be grouped and understood. It's like realizing that on a car, the gas pedal and the engine size are linked; you don't need to tune them separately. This helps us predict how to set them up without guessing.
Clue 5: Universal Patterns.
Whether you train an AI to recognize cats, write poetry, or play chess, they all seem to learn in surprisingly similar ways. They develop similar internal structures. This suggests there is a single, universal "language of learning" that all these models speak.

3. The Big Picture: Mechanics vs. Biology

The paper makes a fascinating comparison between two ways of studying AI:

Mechanistic Interpretability (The Biology): This is like a biologist dissecting a frog to see its organs. Researchers look inside a trained AI to find specific "circuits" or "neurons" that do specific things (like a neuron that only lights up when it sees a face). This is great for understanding what the AI has learned.
Learning Mechanics (The Physics): This is like a physicist studying the laws of motion. It doesn't care about the specific "face neuron"; it cares about the forces that caused the network to learn in the first place. It asks: "What laws of motion made the face neuron appear?"

The authors argue these two fields need each other. Biology needs Physics to explain how the organs formed, and Physics needs Biology to give it real-world examples to explain.

4. Why Does This Matter?

Why should a regular person care about this "Learning Mechanics"?

Safety: If we don't understand how AI works, we can't control it. If we have a "physics manual" for AI, we can predict when it might go wrong or behave strangely before it happens.
Efficiency: Right now, training AI is expensive and wasteful. If we understand the laws, we can design better models that learn faster and use less energy.
Understanding Ourselves: Since AI learns from human data (like language), understanding the laws of AI learning might actually teach us new things about how the human brain learns.

5. The Road Ahead

The authors admit this is hard. We are at the beginning, like Newton before he invented calculus. But they are optimistic. They are calling on young scientists, mathematicians, and engineers to join the party.

They offer advice to beginners:

Don't just read; do experiments. AI is cheap to test.
Simplicity is king. A simple, clear idea is better than a complex, confusing math proof.
Collaborate. Don't try to solve this alone.

The Bottom Line

This paper is a manifesto. It says: "Stop treating AI like magic. It's a complex system, but it follows rules. We are starting to find those rules. Let's build the science of Learning Mechanics together."

It's the moment where AI stops being a mysterious black box and starts becoming a predictable, understandable machine.

Based on the paper "There Will Be a Scientific Theory of Deep Learning" by Jamie Simon et al., here is a detailed technical summary covering the problem, methodology, key contributions, results, and significance.

1. Problem Statement

Deep learning has achieved superhuman performance across diverse tasks, yet it remains a "black box" lacking a unified scientific framework to explain why or how it works. Current practices rely heavily on trial-and-error engineering rather than first-principles understanding.

The Gap: Classical learning theory (PAC, statistical learning) and optimization theory were designed for simple, convex, and parsimonious models. They fail to explain the behavior of modern deep neural networks, which are non-convex, overparameterized, and exhibit complex dynamics (e.g., feature learning, double descent, scaling laws).
The Challenge: The field needs to transition from a mathematical study of "what is possible" to a scientific effort to "describe, explain, and predict" the behavior of complex empirical systems. The central challenge is not opacity (as the equations are known) but complexity (nonlinear, coupled, high-dimensional interactions).

2. Proposed Methodology: "Learning Mechanics"

The authors propose that a scientific theory of deep learning is emerging, which they term Learning Mechanics. This approach draws a direct analogy to physics:

Analogy: Just as classical mechanics studies how forces determine movement in space/time, learning mechanics studies how gradients (forces) determine movement through parameter space.
Core Philosophy: The theory should be fundamental (first-principles), mathematical (quantitative), predictive (falsifiable), comprehensive (covering training dynamics, representations, and weights), intuitive, useful, and humble (explicit about regimes of applicability).
Approach: Instead of seeking a single "Theory of Everything," the methodology involves identifying coarse aggregate statistics and universal behaviors across systems, similar to how statistical mechanics describes gases without tracking individual molecules.

3. Key Contributions & Evidence

The paper identifies five growing bodies of work that serve as evidence for an emerging mechanics of learning, drawing parallels to tools used in physics (e.g., solvable models, limits, empirical laws).

A. Analytically Solvable Settings (Section 2.1)

Concept: Studying simplified, representative models to gain intuition for complex systems (analogous to the harmonic oscillator in physics).
Examples:
- Deep Linear Networks: Despite linearity in data, they exhibit deep learning behaviors like saddle-point landscapes, phase transitions, and sequential learning of singular modes.
- Linearized Networks (NTK): Truncating Taylor expansions around initialization reduces training to kernel ridge regression. This explains inductive bias and generalization in the "lazy" regime but fails to capture feature learning.
- Beyond Linearization: Recent work on multi-index models and quadratic activations is beginning to solve genuinely nonlinear dynamics.

B. Insightful Limits (Section 2.2)

Concept: Taking asymptotic limits (infinite width/depth) to simplify complex systems while retaining fundamental insights.
Key Insight: The Lazy vs. Rich Dichotomy.
- Lazy (NTK) Regime: Weights change negligibly; no feature learning occurs.
- Rich (Feature Learning) Regime: By scaling initialization differently (e.g., Maximal Update Parameterization, $\mu$ P), infinite-width networks do learn features, adapting internal representations to data structure.
Discretization Hypothesis: Finite neural networks are viewed as noisy, finite approximations of infinite continuum systems (analogous to discretizing differential equations).

C. Simple Empirical Laws (Section 2.3)

Concept: Identifying macroscopic statistics that obey simple mathematical laws, similar to Kepler's or Ohm's laws.
Examples:
- Neural Scaling Laws: Test loss follows predictable power laws based on compute, dataset size, and parameter count.
- Edge of Stability: During training, loss landscape sharpness (Hessian eigenvalue) rises and stabilizes near $2/\eta$ (where $\eta$ is the learning rate).
- Neural Collapse: Final layer representations cluster tightly around class means, forming a regular simplex.
- Conservation Laws: Certain quantities (e.g., differences in layer covariances) are conserved under gradient flow due to symmetries (Noether's principle).

D. Disentangling Hyperparameters (Section 2.4)

Concept: Treating hyperparameters (learning rate, batch size, width) as system parameters that can be decoupled to reveal simpler effective dynamics.
Key Findings:
- Linear Scaling Rule: SGD dynamics are invariant under simultaneous rescaling of learning rate and batch size (interpreted via Stochastic Differential Equations).
- Implicit Regularization: Optimizer hyperparameters (like learning rate) implicitly regularize loss curvature.
- $\mu$ P (Maximal Update Parameterization): A parameterization scheme that allows hyperparameters tuned on small models to transfer effectively to large models, decoupling architecture scaling from optimization tuning.

E. Universal Phenomena (Section 2.5)

Concept: Different architectures, datasets, and tasks exhibit shared behaviors, suggesting a simple underlying explanation.
Examples:
- Inductive Biases: CNNs and Transformers achieve similar performance on vision tasks; UNets and Transformers generate identical images in diffusion models.
- Data Structure: Universal statistical properties in data (e.g., power-law spectra, Zipf's law) drive learning.
- Representation Universality: Models trained on different tasks/modalities converge to similar internal representations as they scale up.

4. Results & Synthesis

Synthesis of Fields: The paper argues that the "Physics of Deep Learning" community is effectively developing this mechanics. It also highlights a symbiotic relationship with Mechanistic Interpretability:
- Learning Mechanics provides the "physics" (dynamics, first principles).
- Mechanistic Interpretability provides the "biology" (identifying circuits, features, and semantic meaning).
- Together, they form a complete scientific picture.
Refutation of Skepticism: The authors address common counterarguments (e.g., "theory is impossible," "microscopic theories are useless") by arguing that:
- Theory is emerging now due to new empirical phenomena (scaling laws, universality).
- "Local theories" (e.g., scaling laws, hyperparameter transfer) are already practically useful even without a full theory.
- Understanding the "physics" is a prerequisite for understanding the "psychology" (capabilities/goals) of AI.

5. Significance and Future Directions

Scientific Impact: Establishes deep learning as a rigorous scientific discipline, moving beyond engineering heuristics to predictive theory.
Practical Impact:
- Reduces the need for trial-and-error hyperparameter tuning.
- Provides tools for dataset design and model scaling.
- Offers a foundation for AI safety by identifying relevant variables and mechanisms.
Open Directions (Section 5): The paper outlines ten critical open problems for the next decade, including:
1. Finding solvable models for genuinely nonlinear deep learning.
2. Developing a theory of natural data structure.
3. Defining "functional complexity" and implicit minimization.
4. Formally defining "features" and "circuits."
5. Validating the Discretization Hypothesis.
6. Predicting scaling law exponents a priori.
7. Understanding the role of loss curvature and optimizers.

Conclusion

The paper posits that a Learning Mechanics is not only possible but is actively emerging. By focusing on coarse-grained statistics, universal behaviors, and solvable limits, the field is transitioning from a collection of empirical observations to a predictive science grounded in first principles. This framework promises to demystify deep learning, improve AI safety, and guide the development of future AI systems.