Physics Priors Offer Useful Accuracy-Carbon Trade-Offs in Spatio-Temporal Forecasting

Technical Summary: Physics Priors Offer Useful Accuracy-Carbon Trade-Offs in Spatio-Temporal Forecasting

Problem Statement
The rapid advancement of machine learning (ML) has been driven primarily by the pursuit of improved model efficacy (accuracy metrics). This singular focus has resulted in the development of large-scale models requiring massive computational resources, leading to significant energy consumption and carbon footprints across the model lifecycle. While resource-aware metrics exist, there is a lack of empirical understanding regarding how incorporating domain knowledge—specifically physics inductive biases—affects the trade-off between predictive accuracy and environmental cost (compute, energy, and carbon). This paper investigates whether embedding physical priors into models can offer a more sustainable balance between efficacy and efficiency in spatio-temporal forecasting.

Methodology
The study focuses on the spatio-temporal forecasting of incompressible shear flow, a task governed by the Navier-Stokes equations. The authors frame this as an autoregressive prediction task on a 2D mesh, where models predict the next time step based on a history of previous states.

• Dataset: The authors utilize a subset of the "Well" collection, containing 240 simulations of coupled non-linear evolution of four physical fields (horizontal/vertical velocity, pressure, and a passive tracer) on a periodic domain. The data is split into training, validation, and testing sets based on initial conditions.
• Model Spectrum: Seven models are evaluated, representing a spectrum of physics inductive biases:
  • No Bias: Standard U-Net and ConvNeXt-based U-Net (UNet-CN).
  • Weak Bias: U-Net variants incorporating circular padding (UNet-CP, UNet-CN-CP) to respect periodic boundaries.
  • Strong Bias: Three Fourier Neural Operator (FNO) variants (FNO, Tucker-factorized TFNO, and U-shaped UFNO) that utilize spectral representations to encode global and periodic structures.
• Training & Evaluation: All models are trained for one-step prediction using MSE loss. Performance is assessed via Pearson correlation over 20-step autoregressive rollouts.
• Carbon Measurement: The study employs Carbontracker to measure energy consumption and convert it to CO2 equivalent (CO2eq) emissions using a fixed global average carbon intensity (445 gCO2/kWh). Crucially, the authors evaluate costs across the full lifecycle, including both training and inference (one-step and rollout).

Key Contributions

1. Carbon-Aware Evaluation: The paper advocates for evaluation practices that explicitly account for training and inference costs alongside predictive accuracy, moving beyond accuracy as the sole metric.
2. Empirical Characterization of Trade-offs: The authors provide an empirical analysis of how varying degrees of physics priors influence the accuracy-carbon trade-off in a complex, high-resolution spatio-temporal forecasting task.
3. Lifecycle Perspective: The work highlights that advantages in one stage (e.g., training) do not necessarily translate to another (e.g., inference), emphasizing the need for holistic lifecycle assessment.

Results

• Predictive Performance: All models achieved high accuracy for the first few rollout steps. Weak biases (circular padding) consistently improved performance for U-Net variants. Among strong-bias models, the U-shaped FNO (UFNO) performed significantly better than standard FNO and TFNO, suggesting that multi-scale architecture is critical for this task, not just spectral bias.
• Training Efficiency: Models with stronger physics priors (FNO variants) achieved substantially lower training carbon footprints (2.4 to 7.4 times lower than U-Net variants). UFNO was the most carbon-efficient model during training.
• Inference Efficiency: The trend reversed during inference. FNO variants were, on average, more carbon-intensive than U-Net variants for one-step inference. However, during long-term rollouts, the cost differences narrowed, likely due to the fixed overhead of model loading becoming a larger fraction of the total cost.
• The Trade-off: The results demonstrate that no single model dominates across all metrics. For instance, UFNO offers a favorable profile combining strong accuracy with low training costs, but its inference cost remains nearly double that of the best-performing U-Net variant (UNet-CN-CP).

Significance and Claims
The paper argues that model efficiency must become a core consideration alongside efficacy in ML development and deployment. The authors claim that:

• Physics Priors are Not a Panacea: While strong physics priors can drastically reduce training footprints, they do not automatically guarantee lower inference costs or overall efficiency. The "best" model depends on the specific deployment context (e.g., one-shot vs. repeated inference).
• Shift in Mindset: The community should move away from an "abundance mindset" that treats resources as unlimited and instead exploit existing domain knowledge to constrain learning tasks. This approach offers a sustainable alternative to scaling models solely for marginal accuracy gains.
• Holistic Evaluation: Identifying a single "best" model is less straightforward when both training and inference costs are considered. A model with low training costs may be inefficient in deployment, and vice versa. Therefore, carbon-aware evaluation across the full model lifecycle is essential for making informed decisions about model selection and deployment.

The authors conclude that while their study is limited to spatio-temporal forecasting and specific hyperparameter settings, the central message—that appropriate inductive biases can shape the efficacy-efficiency trade-off—applies broadly to machine learning tasks.