Technical Summary: Aligning AI-driven Discovery with Human Intuition

Problem Statement

As data-driven modeling of physical dynamical systems advances, a critical bottleneck remains: the identification of appropriate state variables. While automated scientific discovery generally involves identifying state variables and deriving governing equations, the initial step of variable discovery is mathematically under-defined and challenging. Existing methods often produce variables that lack physical significance or are difficult for human scientists to interpret.

Current approaches face two primary limitations:

1. Non-uniqueness: State variables are not unique; a system can be described by infinite combinations of variables (e.g., a pendulum can be described by angle/velocity or energy). Without constraints, models tend to favor overly complex representations.
2. Reliance on Priors: Many methods (e.g., sparse regression, physics-informed neural networks) require prior knowledge of the system, specific basis functions, or assumptions about physical laws. This limits their applicability to novel or unknown environments where such priors are absent.

Previous attempts to learn variables without priors, such as the two-stage autoencoder approach by Chen et al. (2022), struggle to produce variables with physical significance or explicit analytical expressions because they lack interpretability criteria to guide the discovery process.

Methodology: The Temporally-Informed Dynamics Encoder (TIDE)

The authors propose a general principle for distilling representations aligned with human intuition without relying on prior physical knowledge. The core of this approach is the Temporally-Informed Dynamics Encoder (TIDE), a data-driven framework that incorporates key mathematical properties typical of physical variables into its training objective.

Model Architecture

TIDE employs a two-step process using a Variational Autoencoder (VAE) backbone, a dynamics module, and a regularization term:

1. VAE Backbone: The model compresses high-dimensional input data (e.g., video frames) into a low-dimensional latent representation $z$. The encoder $h_\phi$ approximates the posterior $q_\phi(z|x)$, and the decoder $p_\theta$ reconstructs the input. The training objective includes the Evidence Lower Bound (ELBO).
2. Dynamics Module: A feed-forward neural network ($h_{dyn}$) is integrated to model the time evolution operator $g$. It predicts the next latent state $\hat{z}_{t+1}$ based on the current state. The module is trained to maximize the likelihood of the next observation, jointly optimizing the dynamics with the representation learning.
3. Time-Derivative Regularization: To address the non-uniqueness of representations and encourage variables that align with human intuition (specifically smoothness and continuity), the model applies a penalty term based on the magnitudes of discrete time derivatives.
   • The latent vectors are normalized to prevent global contraction.
   • The regularization loss $L_{reg}$ aggregates the first $n$ discrete derivatives (set to $n=4$ in experiments).
   • The final objective function is a weighted combination: $L_{TIDE} = -L_{ELBO} - L_{dyn} + \lambda_2 L_{reg}$.

Intrinsic Dimension (ID) Estimation

To determine the correct number of state variables, the authors use a manifold learning algorithm (DANCo) on a high-dimensional latent space generated by a preliminary TIDE network. The estimated ID then defines the latent dimension for a second TIDE network that learns the final state variables.

Symbolic Regression

For systems where standard human state variables are known, the authors use PySR (a symbolic regression library) to derive explicit analytical expressions for the learned latent variables. This step validates whether the learned variables can be mapped to simple mathematical functions of physical quantities.

Key Contributions

1. A New Principle for Alignment: The paper proposes a general principle for distilling representations that naturally align with human intuition by enforcing smoothness and differentiability, without requiring prior physical knowledge.
2. TIDE Architecture: The introduction of the TIDE network, which jointly learns dynamics and state variables while incorporating time-derivative regularization to tackle the state variable discovery problem.
3. Interpretable and Analytical Results: The method learns variables that not only correlate more closely with human-defined quantities but also yield explicit analytical expressions, a significant step forward from "black box" latent representations.

Results

The model was evaluated on nine benchmark datasets, including five simulated systems (e.g., reaction-diffusion, single/double pendulum) and four real-world systems (e.g., lava lamp, fire flame).

• Intrinsic Dimension Estimation: TIDE provided more precise estimates of the intrinsic dimension compared to the baseline Neural State Variables (NSV) approach, particularly for systems with known ground truth IDs. The authors attribute this to the time-derivative regularization encouraging a smooth latent space.
• Variable Smoothness and Continuity: Visualizations of the latent space (e.g., for pendulum systems) showed that TIDE variables evolve smoothly over time, closely resembling human-interpretable quantities like angle and angular velocity. In contrast, the baseline exhibited discontinuous jumps.
• Interpretability Metrics:
  • Mutual Information (MI): TIDE demonstrated higher MI between model variables and human variables, indicating stronger correlation.
  • Analytical Mean Squared Error (AMSE): TIDE achieved significantly lower AMSE when fitting symbolic expressions to the learned variables, confirming their analytical recoverability.
• Analytical Expressions: The authors successfully derived explicit formulas for the latent variables of the single and double pendulum (e.g., $z_0$ expressed as a combination of sine functions of the angle $\theta$). No such symbolic combination could be found for the baseline variables.

Significance and Claims

The paper claims that TIDE represents a significant step toward making human-AI collaboration in scientific discovery more fruitful. By aligning AI-generated representations with human intuition through mathematical properties (smoothness, differentiability) rather than explicit physical priors, the method bridges the gap between data-driven modeling and human understanding.

The authors emphasize that this is, to their knowledge, the first result to derive explicit formulas for state variables learned without prior knowledge of the system. They suggest this principle can help researchers gain insights into unknown phenomena, particularly when theoretical predictions diverge from empirical observations.

Limitations: The authors modestly note that the method relies on the assumption of a stable state variable representation, which may not hold in scenarios with inefficient data collection. They also acknowledge that while the representation aligns with human quantities, it may not perfectly match them due to the inherent non-uniqueness of state variable configurations. Future work is suggested to explore semi-supervised learning and theoretical conditions for identifiability.