🤖 machine learning

The Propagation Field: A Geometric Substrate Theory of Deep Learning

This paper proposes a "Propagation Field" framework that redefines neural networks through their internal geometric trajectories and Jacobian structures rather than just input-output mappings, demonstrating that explicitly optimizing these field properties enhances generalization, robustness, and continual learning performance beyond what endpoint losses alone can achieve.

Original authors: Xingrui Gu

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xingrui Gu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: It's Not Just About the Destination

Imagine you are teaching a student to drive from Point A to Point B.

The Old Way (Standard Deep Learning): You only care if the student arrives at the correct destination. If they get there, you give them an A. You don't care if they took a smooth highway, a bumpy dirt road, or if they drove in circles before finally stopping. As long as the "endpoint" is correct, the training is considered successful.
The New Way (This Paper): The authors argue that we should also care about the journey itself. They propose that inside a neural network, information doesn't just jump from input to output; it travels along a specific "path" or "field." This path has a shape, a speed, and a direction.

The paper suggests that two networks can get the exact same answer (the same destination) but take completely different, and potentially very different quality, journeys to get there.

The Core Concept: The "Propagation Field"

Think of a neural network not as a black box that spits out answers, but as a landscape or a river system.

The Hidden States: As data moves through the layers of the network, it's like a boat moving down a river. The "hidden states" are the position of the boat at every moment.
The Jacobians: These are like the current or the slope of the river at any given point. They tell you how much the water (data) speeds up, slows down, or gets squeezed as it flows.
The Field: The combination of the boat's path and the river's currents creates a "Propagation Field."

The authors claim that standard training only looks at where the boat ends up. It ignores whether the river was smooth, whether the boat spun in circles, or if the current was chaotic.

Key Findings (The "Experiments")

1. Same Destination, Different Journeys

The paper proves that you can have two models that are perfect at their job (getting the right answer) but have totally different internal "fields."

Analogy: Imagine two hikers reaching the summit of a mountain. Hiker A took a direct, smooth trail. Hiker B took a path that zig-zagged wildly, slid down a cliff, and climbed back up. Both reached the top (same "endpoint accuracy"), but their experiences (the "field") were worlds apart.
The Result: The paper shows that standard training doesn't force the network to take the "smooth trail." It just finds any path that works.

2. The "Teacher-Flow" Test

To prove this, the researchers created a controlled environment (like a physics simulation) where they knew the "true" path the data should take.

They trained one model to just get the answer right.
They trained another model to get the answer right and follow the true path.
The Surprise: Both models got the answer right. But the first model's internal path was messy and wrong, while the second model's path was perfect. This proves that getting the right answer doesn't mean the network learned the right "rules of the road."

3. Why Does the Journey Matter? (Generalization)

The paper asks: Does a smooth journey help the network handle new situations?

The Good News: In some tasks (like looking at an image where parts are revealed in different orders), forcing the network to follow a consistent, smooth path helped it handle new, unseen variations better. It made the network more robust.
The Bad News (The Collapse): If you force the network to be too consistent, it can break.
- Analogy: Imagine a teacher telling a student, "No matter what question you get, you must walk in a perfectly straight line to the answer." The student might just stop thinking and give the same answer to everything to keep their line straight.
- The Result: The paper found that if you over-constrain the "field," the network might become very consistent internally but stop learning the actual task, leading to terrible performance.

4. Forgetting (Continual Learning)

When a network learns a new task, it often "forgets" the old one.

The Old View: Forgetting means the network can no longer give the right answer for the old task.
The New View: The paper suggests forgetting also happens at the "field" level. Even if the network remembers the answer, the internal "river" might have been rerouted or dried up.
The Solution: They found that if you add a rule to "preserve the shape of the river" (the internal path) while learning new tasks, the network remembers the old tasks better. It acts like a supplement to existing memory techniques, helping the network keep its internal structure intact.

The Takeaway

The paper introduces a new way to look at AI. Instead of just asking, "Is the answer right?" we should also ask, "Is the path the data took to get there healthy and consistent?"

Endpoint Supervision: "Did you get the right answer?" (Current standard)
Propagation Field Theory: "Did you get the right answer via a stable, logical journey?" (The new proposal)

The authors conclude that the "quality of the journey" is a measurable and trainable property. By paying attention to the internal geometry of the network, we can build models that are more robust and less likely to forget, but we must be careful not to force the journey so strictly that the network stops learning the task entirely.

Technical Summary: The Propagation Field: A Geometric Substrate Theory of Deep Learning

1. Problem Statement

Modern deep learning primarily treats neural networks as endpoint functions mapping inputs to outputs, optimized via task-specific losses (LeCun et al., 2015). While this approach has driven significant success, the internal mechanisms of information propagation remain poorly understood. Standard training constrains the boundary behavior (input-output mapping) but leaves the geometry of layerwise representations and internal state trajectories underdetermined. This gap manifests in issues ranging from adversarial robustness (Szegedy et al., 2013) and in-context learning (Brown et al., 2020) to catastrophic forgetting in continual learning (Kirkpatrick et al., 2017).

The central problem addressed is that endpoint supervision does not uniquely identify the internal propagation field. Two models can be "endpoint-equivalent" (achieving nearly identical task loss) while possessing vastly different internal geometric structures, including hidden-state trajectories and local Jacobian operators. Consequently, standard scaling laws characterize endpoint loss scaling but not necessarily the scaling of internal propagation quality.

2. Methodology: The Propagation Field Framework

The authors propose a "Propagation Field" perspective, inspired by the shift from force-based mechanics to geometric field theories in physics (e.g., General Relativity and Yang-Mills theory). Instead of viewing layers as discrete computational steps, the network is viewed as a field defined by the collection of hidden-state trajectories and local Jacobian operators across depth.

2.1 Core Definitions

Propagation Field ( $\Phi_\theta$ ): Defined as the set of hidden states $\{h_0, \dots, h_L\}$ and local Jacobians $\{J_0, \dots, J_{L-1}\}$ , where $J_\ell = \partial h_{\ell+1} / \partial h_\ell$ . For continuous-flow models, this includes the trajectory $h(t)$ and the vector field $f_\theta(h, t)$ .
Endpoint Equivalence: Two models are endpoint-equivalent if their task losses differ by less than a tolerance $\epsilon$ . The paper proves (Proposition 2.3) that endpoint equivalence does not imply field equivalence; models can have identical accuracy but differ by orders of magnitude in trajectory and Jacobian structure.
Field Distance: A metric $d_{field}$ quantifies the discrepancy between the hidden-state trajectories and Jacobian structures of two models.

2.2 Observable Metrics

To make the propagation field measurable, the authors introduce discrete geometry metrics computable in standard networks:

Path Length: Total representation displacement across depth.
Discrete Curvature: A proxy for the bending of the hidden trajectory.
Velocity Alignment: The consistency of propagation direction across layers.
Path Sensitivity: The variance in hidden states or outputs when the same input is presented via different observation paths (e.g., different patch orders or temporal windows).
Solver Consistency: The stability of terminal states when the same underlying field is discretized with different step sizes or numerical solvers.
Retention Scores (FRS/JRS): In continual learning, metrics measuring the drift of hidden trajectories (FRS) and local Jacobians (JRS) between old and new models.

2.3 Optimization Objectives

The paper introduces Field-Aware Objectives to regularize the internal geometry:

Multi-path Consistency ( $L_{reveal}$ ): Minimizes differences in hidden states/logits across different observation paths.
Solver Consistency ( $L_{solver}$ ): Ensures terminal states remain consistent under different numerical discretizations.
Jacobian Smoothness ( $L_{jac}$ ): Regularizes the smoothness of local propagation operators.
Field Preservation Regularization (FPR): In continual learning, FPR adds constraints to preserve the hidden trajectories and Jacobian fields of previous tasks, complementing standard replay or distillation.

3. Key Contributions and Experimental Results

The paper provides empirical evidence across controlled systems, real-world tasks, and continual learning settings.

3.1 Controlled Systems (Teacher-Flow & PDEs)

Underdetermination: In teacher-flow experiments, models with identical endpoint accuracy (0.9975) exhibited trajectory recovery errors differing by 58.7x (3.52 vs. 0.06) when field supervision was applied.
Law Recovery: In PDE extrapolation tasks, standard endpoint fitting failed to recover the underlying propagation law (MSE = 1.217 at $T=2.0$ ), whereas generator-style field models achieved significantly lower error (MSE = 0.195) and preserved energy evolution.
Conclusion: Endpoint fitting identifies a behavioral equivalence class but fails to recover the underlying propagation law.

3.2 Real Multi-Path Tasks

Task Alignment: Field-aware objectives improve generalization (OOD accuracy, unseen-path accuracy) and calibration only when aligned with the task structure.
- Success: On Tiny-ImageNet, the full field-aware objective improved accuracy (0.097 $\to$ 0.118), OOD accuracy, and calibration while reducing path sensitivity.
- Failure (Collapse): On CIFAR-100, enforcing field consistency reduced path sensitivity but caused field collapse, degrading accuracy from 0.0405 to 0.0093. This demonstrates that low path sensitivity is not sufficient for discriminative performance; the field must retain sufficient directional richness.
Optimization Trade-offs: Gradient analysis reveals conflicts between task and field objectives (up to 36% negative gradient fractions). Standard backpropagation does not automatically find the optimal task-field trade-off; specialized algorithms (e.g., LocalFieldMatch) are required to navigate the Pareto frontier.

3.3 Continual Learning

Field vs. Function Retention: In Split CIFAR-100, standalone Field Preservation Regularization (FPR) successfully preserved internal field metrics (JRS = 0.033) but failed to maintain endpoint accuracy (AA = 0.037). Conversely, standard replay methods (DER++) preserved accuracy (AA = 0.135) but allowed field drift (JRS = 0.177).
Complementarity: Combining FPR with DER++ (DER++ + FPR-Full) improved both dimensions simultaneously, increasing Average Accuracy (0.125 $\to$ 0.136) and Backward Transfer while reducing field drift. This suggests field retention and function retention are complementary, not redundant, objectives.

4. Significance and Claims

The paper claims to establish propagation-field quality as a measurable and trainable property of neural networks that exists beyond endpoint performance.

Theoretical Shift: It reframes deep learning from optimizing a static input-output map to managing a dynamic internal geometric substrate.
Identifiability: It demonstrates that endpoint supervision is insufficient to uniquely determine the internal propagation field, necessitating explicit field-aware objectives for tasks involving path dependence or long-horizon processes.
Practical Utility: Field-aware objectives can enhance robustness and generalization in specific regimes (e.g., multi-path tasks) but require careful balancing to avoid "field collapse," where the network optimizes for geometric consistency at the expense of discriminative power.
Continual Learning: It proposes that forgetting occurs at the field level (rewriting of internal propagation laws) and that preserving this geometry complements traditional function-level retention strategies.

The authors conclude with a "Kakeya-like directional principle": useful propagation fields must reduce spurious path sensitivity while preserving sufficient directional richness for discrimination. This work suggests that future deep learning research should explicitly measure and optimize the geometry of internal propagation, not just the final output.