Technical Summary: The Perception–Physics Paradox: Probing Scientific Alignment with TC-BENCH

1. Problem Statement

Vision Foundation Models (VFMs) have demonstrated strong zero-shot performance on visual benchmarks, leading to their adoption in scientific domains. However, the authors identify a critical limitation: high predictive accuracy on satellite imagery often stems from visual correlations rather than an understanding of underlying structural invariants. This discrepancy is termed the Perception–Physics Paradox.

Specifically, models may appear to generalize well perceptually (e.g., across different geographic regions or agencies) while failing to preserve the physical degrees of freedom necessary for scientific reasoning. In extreme physical regimes, such as intense tropical cyclones, visual cues (like cloud structure) can saturate, making physically distinct states (e.g., 905 hPa vs. 915 hPa pressure) visually indistinguishable. VFMs may map these distinct states to similar latent representations, collapsing the physical variation required for accurate scientific inference. Current evaluation protocols, which rely on average-case performance or perception-based out-of-distribution (OOD) tests, fail to detect this "latent collapse."

2. Methodology

2.1 Scientific Alignment and Structural Isomorphism

To address this gap, the paper introduces Scientific Alignment as an implicit objective for representation learning. The authors formalize a testable aspect of this alignment called Structural Isomorphism.

• Definition: A representation $z = g(x)$ is structurally isomorphic to a physical system $S$ if there exists an injective linear map $A$ such that $z = Ay + \epsilon_\mu(y)$, where $y$ represents the physical state variables and $\epsilon_\mu(y)$ is a uniformly bounded residual.
• Implication: This definition does not require coordinate-wise disentanglement but ensures that the latent space identifies the physical system up to a linear reparameterization. It implies that the latent geometry preserves the structure of the physical state manifold.

2.2 The Structural Alignment Probe Framework

Based on the implications of structural isomorphism, the authors derive a hierarchy of necessary conditions, operationalized as Structural Alignment Probes. These probes test whether a learned representation supports specific physical properties:

1. Static Fidelity ($Q_{stat}$): Can the physical state $y$ be linearly recovered from the latent representation $z$? (i.e., $\|y - Lz\| \le \epsilon_{stat}$).
2. Dynamic Coherence ($Q_{dyn}$): Do the latent derivatives align with physical derivatives? (i.e., $\|\dot{y} - L\dot{z}\| \le \epsilon_{dyn}$).
3. Manifold Consistency ($Q_{con}$): Do recovered variables satisfy known physical constraints (e.g., pressure-wind coupling)?

Theoretical analysis (Theorem 2.1) establishes that if static fidelity and dynamic coherence hold with uniform bounds, the representation supports interventional consistency, allowing for physically plausible rollouts under perturbations.

2.3 TC-BENCH Dataset

To operationalize these probes, the authors introduce TC-BENCH, a global, versioned, and reproducible benchmark dataset for tropical cyclone research.

• Data Sources: Integrates the International Best Track Archive for Climate Stewardship (IBTrACS v4r01) for metadata (minimum central pressure $P_c$, maximum sustained wind $V_m$) and GridSat-B1 for geostationary infrared brightness temperature imagery.
• Scope: Covers 1980–2024 across seven meteorological agencies and all major basins.
• Design: Unlike previous benchmarks focused on average-case performance, TC-BENCH explicitly includes intense regimes ($P_c < 980$ hPa) and life-cycle transitions to enable "physically targeted stress tests."

2.4 Experimental Protocol

The authors evaluate a diverse suite of frozen VFMs (including DINOv2/3, CLIP, SigLIP, and MAE) using a restricted linear probe ($L \in \mathcal{H}_{lin}$). The evaluation is stratified by intensity regimes:

• Moderate Regime: $P_c \ge 980$ hPa.
• Intense Regime: $P_c < 980$ hPa.
  The protocol controls for confounding factors such as sample imbalance, probe capacity (via nonlinear probes), and representation aggregation (via spatial token means).

3. Key Results

3.1 The Perception–Physics Paradox in Action

The experiments reveal a systematic failure of current VFMs in intense regimes:

• Static Fidelity Degradation: While models perform well in moderate regimes, the error in linearly recovering pressure ($P_c$) increases sharply in the intense regime. This "Visual Saturation" hypothesis suggests that features saturate, making fine-grained intensity differences indistinguishable in the latent space.
• Dynamic Coherence Collapse: The alignment between latent and physical temporal evolution degrades significantly in intense regimes, with error spikes in catastrophic regimes ($P_c < 920$ hPa).
• Manifold Inconsistency: Recovered pressure and wind variables fail to respect physical couplings (e.g., the gradient wind balance constraint regarding latitude) in intense regimes, with constraint violations rising from ~20% in moderate regimes to ~55% in intense regimes.

3.2 Latent Geometry Analysis

Geometric diagnostics on DINOv3 representations (the strongest performer) confirm the mechanism of failure:

• Dimensionality Collapse: In intense regimes, the effective dimensionality of the feature covariance drops by approximately 60%.
• Feature Compression: Physically distinct storms are mapped to a narrower region of the latent space, reducing the feature spread.
• Resolution Loss: The first principal component (PC1), which correlates with pressure in moderate regimes, loses its systematic variation with pressure in intense regimes.

3.3 Robustness of Findings

The failure is not an artifact of the linear probe or static image pretraining:

• Nonlinear Probes: Increasing probe capacity (MLP, Transformer) does not resolve the performance gap.
• Supervised Baselines: A supervised model trained from scratch on pixel data achieves lower error in intense regimes, proving that the signal is statistically present in the imagery but not captured by frozen VFM representations.
• Video Models: Temporal pretraining (VideoMAE, V-JEPA2) improves average performance but fails to eliminate the regime-dependent degradation.

4. Contributions

1. Scientific Alignment as a Diagnostic Lens: The paper articulates the Perception–Physics Paradox and formalizes Structural Isomorphism as a set of falsifiable, necessary conditions for scientific usability. It shifts the evaluation focus from perceptual robustness to structural preservation of physical invariants.
2. TC-BENCH: A global, reproducible benchmark dataset with an automated construction pipeline, designed specifically to stress-test representations in physically extreme regimes where visual cues saturate.
3. Unified Probing Framework: A systematic protocol (Static Fidelity, Dynamic Coherence, Manifold Consistency) that exposes a failure mode—regime-dependent latent collapse—missed by conventional average-case or OOD evaluations.

5. Significance and Claims

The authors position this work as diagnostic rather than prescriptive. They do not claim that VFMs are inherently incapable of learning physically aligned representations, nor that task-specific training cannot recover physical information. Instead, the paper demonstrates that:

• Current evaluations are insufficient: Standard metrics and perception-driven OOD tests cannot certify alignment with respect to specific scientific queries, particularly in extreme regimes.
• Scaling is not a silver bullet: Scientific alignment does not arise as a natural byproduct of scaling alone; models can look correct without reasoning correctly.
• Necessary conditions for science: The proposed framework provides a minimal set of testable conditions to determine if a representation can serve as a valid structural surrogate for physical systems.

The authors note that the Perception–Physics Paradox is likely generalizable to other high-impact remote sensing settings (e.g., wildfire monitoring) where observations become locally insensitive to physically meaningful variation in extreme regimes. The work emphasizes the need for query-aware, physically grounded probes to ensure that learned representations support coherent scientific reasoning.