🤖 machine learning

Representation Learning for Spatiotemporal Physical Systems

This paper argues that evaluating spatiotemporal physical models based on their ability to estimate governing physical parameters, rather than next-frame prediction, reveals that latent-space learning methods like JEPAs often outperform both pixel-level prediction models and generic self-supervised approaches in learning physically relevant representations.

Original authors: Helen Qu, Rudy Morel, Michael McCabe, Alberto Bietti, François Lanusse, Shirley Ho, Yann LeCun

Published 2026-03-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Helen Qu, Rudy Morel, Michael McCabe, Alberto Bietti, François Lanusse, Shirley Ho, Yann LeCun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand how a complex machine works, like a car engine or a weather system. For a long time, scientists and AI researchers have tried to teach computers to predict the future of these systems. They say, "Here is a video of the engine running; please guess what the next frame looks like."

This paper argues that while guessing the next frame is impressive, it's not the best way to truly understand the physics. Instead, the authors propose a different approach: Don't just guess the picture; guess the "rules" behind the picture.

Here is a breakdown of their findings using simple analogies:

1. The Old Way: The "Pixel Painter" vs. The "Rule Learner"

The paper compares two types of AI learning methods:

The Pixel Painter (Masked Autoencoders/VideoMAE):
Imagine a student trying to learn how a river flows. The "Pixel Painter" student is given a video of the river with some parts covered up. Their job is to fill in the missing water droplets and ripples perfectly. They become amazing at drawing the water, but they might not actually understand why the water flows that way. They are memorizing the visual details (the pixels) rather than the physics.
The Rule Learner (JEPA - Joint Embedding Predictive Architectures):
This student is also given a video of the river, but they aren't asked to redraw the water. Instead, they are asked to predict the next step in the story using a secret code (a "latent space"). They learn to say, "The water is moving fast because the slope is steep," without needing to draw every single drop. They are learning the underlying logic of the system.

The Big Surprise: The authors found that the "Rule Learner" (JEPA) was actually better at understanding the physics than the "Pixel Painter," even though the Pixel Painter was trained to be a better artist.

2. The Test: Can You Guess the Settings?

To see who really understood the systems, the researchers didn't ask the AI to draw a pretty picture. They asked a harder question: "What are the settings of this simulation?"

They used three different physical systems as test subjects:

Active Matter: Like a swarm of tiny robots that move on their own.
Shear Flow: Like layers of honey sliding past each other.
Rayleigh-Bénard Convection: Like a pot of soup heating up, creating swirling cells.

In each case, the AI had to guess the hidden "knobs" (parameters) that controlled the simulation, such as how sticky the fluid is or how strong the heat is.

The Result:
The "Rule Learner" (JEPA) guessed the settings much more accurately than the "Pixel Painter." In fact, the Rule Learner was so good that it almost matched the performance of specialized physics models that were built specifically for this job.

3. The "Data Diet" Analogy

The paper also tested how much data each AI needed to learn.

The Pixel Painter is like a student who needs to read the entire encyclopedia to understand a single concept. If you only give them 50% of the data, they get confused and make big mistakes.
The Rule Learner is like a student who understands the principles. Even if you only give them 10% of the data, they can still figure out the answer because they grasped the core logic.

4. Why This Matters

The authors conclude that for scientific discovery, we don't need AI that can perfectly recreate a video frame-by-frame. We need AI that learns the essence of the physical laws.

Autoregressive models (predicting the next pixel) are like a parrot repeating a song. It sounds good, but it doesn't know the lyrics' meaning.
Latent space models (predicting the next rule) are like a musician who understands music theory. They can improvise and explain why the song works.

The Takeaway

If you want an AI to help you solve real-world physics problems (like designing better engines or predicting climate change), don't just train it to be a good video editor. Train it to be a theorist that understands the hidden rules of the universe. This paper shows that "Rule Learners" are the future of scientific AI.

1. Problem Statement

Current machine learning approaches for spatiotemporal physical systems predominantly focus on next-frame prediction (autoregressive surrogate modeling). The goal is typically to learn an accurate emulator that predicts pixel-level evolution over time. However, this paradigm faces two critical limitations:

Computational Cost: Training high-fidelity pixel-level emulators is expensive.
Downstream Utility: These models often suffer from compounding errors during autoregressive rollout and may not effectively capture the high-level physical principles required for downstream scientific tasks, such as parameter estimation (e.g., determining the Reynolds number or Rayleigh number from observed dynamics).

The authors argue that optimizing for pixel-level reconstruction does not necessarily yield representations that are physically meaningful or efficient for scientific inference. They propose shifting the focus from generative fidelity to representation learning that prioritizes the extraction of governing physical parameters.

2. Methodology

The study evaluates and compares different self-supervised learning paradigms on three distinct Partial Differential Equation (PDE) governed systems: Active Matter, Shear Flow, and Rayleigh-Bénard Convection.

A. Representation Learning Frameworks

The authors compare two primary self-supervised approaches:

Joint Embedding Predictive Architectures (JEPA):
- Concept: Instead of predicting raw pixel values, JEPAs predict the evolution of the system in a latent representation space.
- Mechanism: The model learns an encoder $f$ and a predictor $g$ . Given a context sequence of $k$ frames, the predictor estimates the representation of the next $k$ frames.
- Loss Function: The model minimizes a VICReg-style loss (Variance-Invariance-Covariance Regularization) in the latent space to prevent mode collapse and ensure the embeddings capture high-level dynamics rather than low-level noise.
- Architecture: A downsampling CNN (ConvNeXt-based) for the encoder and a CNN with an inverse bottleneck for the predictor.
Masked Autoencoding (MAE):
- Concept: A standard approach where the model learns to reconstruct masked regions of the input data.
- Implementation: Implemented as VideoMAE (ViT-tiny/16) with temporal tube masking (masking the same spatial region across all frames).
- Objective: Minimizes pixel-level reconstruction error ( $L_2$ loss between predicted and actual masked pixels).

B. Physical Modeling Baselines

To establish a performance ceiling, the authors compare their self-supervised methods against specialized physical modeling baselines:

DISCO: An in-context operator learning framework that infers trajectory-specific evolution rules.
MPP (Multiple Physics Pretraining): An autoregressive foundation model trained on diverse physics datasets.

C. Evaluation Protocol

The core evaluation metric is Physical Parameter Estimation.

Task: Fine-tune a simple regression head on top of the frozen pretrained encoders to predict the governing physical parameters of the system (e.g., $\alpha, \zeta$ for active matter; Rayleigh/Prandtl numbers for convection; Reynolds/Schmidt numbers for shear flow).
Rationale: Low error in parameter estimation serves as a quantifiable proxy for how well the model has learned the underlying physical laws, independent of generative pixel fidelity.

3. Key Contributions

Latent Space vs. Pixel Space: The paper demonstrates that latent-space prediction (JEPA) consistently outperforms pixel-level reconstruction (MAE) for scientific tasks. Learning dynamics in the latent space forces the model to abstract away irrelevant visual details and focus on the underlying physical evolution.
Superiority over Autoregressive Models: The study finds that generic self-supervised latent predictors (JEPA) can outperform or match specialized autoregressive foundation models (like MPP) on parameter estimation tasks, challenging the notion that autoregressive modeling is the only path for physical systems.
Data Efficiency: JEPA exhibits superior sample efficiency during fine-tuning. It achieves near-optimal performance with significantly less labeled data compared to pixel-based models.
Disentanglement of Fidelity and Physics: The work highlights that high generative fidelity (predicting the next frame perfectly) does not correlate with high physical relevance (estimating parameters accurately).

4. Experimental Results

The experiments were conducted on datasets from "The Well" (a large-scale collection of physics simulations).

Table 1 Summary: Parameter Estimation MSE (Lower is Better)

Method	Active Matter	Shear Flow	Rayleigh-Bénard
JEPA	0.079	0.38	0.13
VideoMAE (MAE)	0.160	0.67	0.18
DISCO (Baseline)	0.057	0.13	0.01
MPP (Baseline)	0.230	0.59	0.08

JEPA vs. VideoMAE: JEPA showed substantial improvements over VideoMAE across all systems:
- Active Matter: 51% relative improvement.
- Shear Flow: 43% relative improvement.
- Rayleigh-Bénard: 28% relative improvement.
JEPA vs. Specialized Baselines:
- JEPA approached the performance of DISCO (the best-performing specialized model) on Active Matter and Shear Flow.
- Notably, MPP (autoregressive) struggled significantly despite end-to-end fine-tuning, performing worse than the frozen encoder approaches. This aligns with findings in NLP where encoder-only models often outperform autoregressive models on non-generative tasks.
Scaling Behavior (Table 2): On the Shear Flow task, JEPA achieved 95% of its best performance using only 50% of the fine-tuning data. In contrast, VideoMAE required 100% of the data to reach comparable performance, and even with 10% of the data, JEPA outperformed VideoMAE's full-data performance.

5. Significance and Conclusion

This paper provides a paradigm shift for applying AI to scientific discovery in physical systems:

Scientific Relevance: It establishes that the primary goal of learning representations for physical systems should be physical interpretability (parameter estimation) rather than generative fidelity (pixel prediction).
Efficiency: Latent prediction models (JEPA) are more data-efficient and computationally effective for downstream scientific inference than traditional autoregressive or pixel-reconstruction models.
Future Direction: The results suggest that latent-space predictive learning is a promising foundation for scientific machine learning, potentially offering a more robust alternative to autoregressive surrogate modeling for tasks involving parameter discovery, control, and qualitative prediction.

The authors conclude that disentangling the physical relevance of learned representations from generative fidelity is crucial for advancing AI in scientific domains.