HPC-Enabled Video-based Coastal Wave Parameter Estimation Using V-JEPA and Deep Spatiotemporal Learning

Technical Summary: HPC-Enabled Video-based Coastal Wave Parameter Estimation Using V-JEPA and Deep Spatiotemporal Learning

Problem Statement
Traditional in-situ methods for coastal wave monitoring, such as wave buoys and pressure sensors, face significant limitations including high deployment costs, poor spatial coverage, and susceptibility to damage during storm conditions. While vision-based methods offer a cost-effective alternative, existing deep learning approaches often struggle with data scarcity, inadequate multi-scale temporal modeling, poor generalization across diverse environments, and a lack of physical interpretability. Furthermore, capturing the complex temporal dynamics of ocean waves across different scales (from short-term breaking events to long-period swells) remains a challenge for standard supervised models.

Methodology
The authors propose a novel, High-Performance Computing (HPC) enabled deep learning framework designed for the sensor-free, joint estimation of five coastal wave parameters: significant wave height ($H_s$), maximum wave height ($H_{max}$), peak period ($T_p$), zero upcrossing period ($T_z$), and wave direction ($\theta$). The system processes monocular coastal video through a modular pipeline:

1. Backbone (V-JEPA): The framework utilizes a Vision Transformer (ViT-Small) backbone pre-trained with V-JEPA (Video Joint Embedding Predictive Architecture). This self-supervised approach enables robust spatiotemporal feature extraction from unlabeled video, effectively handling visual challenges such as foam, spray, and illumination changes without relying heavily on large labeled datasets.
2. Temporal Encoding (Dual-Stream SlowFast): To capture the broadband temporal spectrum of coastal waves, a dual-stream SlowFast encoder is employed. The "fast" pathway processes the full temporal sequence (16 frames) to capture short-term breaking events, while the "slow" pathway operates on a temporally down-sampled sequence (4 frames) to model longer-period swells.
3. Saliency-Guided Augmentation: An optical flow stream based on the Farneback algorithm generates motion saliency maps. This guides the cropping process to bias the model's attention toward hydrodynamically active regions, such as active surf and breaking zones.
4. Regression and Physics Constraints: A multi-task regression head predicts the five parameters. Training is supervised by a composite loss function combining Mean Squared Error (MSE) with a physics-guided regularization term. This term enforces consistency with the Airy wave deep-water dispersion relation ($\lambda_p = 0.1$), ensuring the model adheres to physical wave dynamics.
5. Computational Infrastructure: The entire framework was trained on an NVIDIA DGX A100 cluster (8× A100 GPUs, 320 GB aggregate memory) using PyTorch. The authors note that this HPC infrastructure was essential for training the full spatiotemporal pipeline and achieving a theoretical 600x–900x speedup compared to consumer-grade hardware.

Key Contributions

• Sensor-Free Multi-Parameter Estimation: The paper presents a unified architecture capable of simultaneously estimating five distinct wave parameters from a single camera view, eliminating the need for in-situ instrumentation.
• Data-Efficient Architecture: By leveraging V-JEPA's self-supervised pre-training, the framework demonstrates feasibility in a data-limited regime, utilizing only six annotated training scenes while maintaining generalization to geographically diverse held-out test sites.
• Physics-Informed Deep Learning: The integration of Airy wave dispersion constraints into the loss function provides a mechanism to regularize the regression head, reducing the solution space and stabilizing gradient flow.
• HPC-Enabled Scalability: The study demonstrates the critical role of HPC resources in training complex video-based spatiotemporal models, achieving rapid convergence (early stopping at epoch 31) and high throughput.

Experimental Results
The model was evaluated on a dataset comprising six annotated training scenes and a geographically diverse held-out test set. Key performance metrics include:

• Correlation: The framework achieved statistically significant Pearson Correlation Coefficients (PCC) across all parameters: $H_s$ (0.451), $H_{max}$ (0.578), $T_p$ (0.643), $T_z$ (0.680), and Wave Direction $\theta$ (0.832).
• Generalization: Performance on the held-out test set remained consistent with the full-set metrics, indicating the model's ability to generalize across different coastal locations, camera angles, and lighting conditions without overfitting.
• Limitations: The coefficient of determination ($R^2$) values were relatively low (maximum 0.246 for $T_p$), which the authors attribute to the small size of the annotated training dataset (6 scenes) rather than architectural deficiencies. The model exhibited systematic biases, including an underestimation of wave heights and a positive bias in wave direction ($+17.9^\circ$), the latter attributed to a lack of camera bearing calibration and orthorectification.

Significance and Claims
The paper positions this work as a proof-of-concept demonstrating the feasibility of using HPC-enabled, self-supervised deep learning for coastal wave monitoring in data-scarce environments. The authors claim that while the current $R^2$ values indicate that variance capture requires larger annotated datasets, the high temporal correlations (PCC) confirm that the framework successfully extracts statistically valid wave parameter signals from raw video.

The significance of the work lies in its demonstration that self-supervised learning (V-JEPA) combined with physics-informed constraints can overcome the limitations of traditional supervised models that rely on massive labeled datasets. The authors conclude that the proposed method serves as a foundation for future improvements, specifically suggesting that scaling to larger, diverse datasets and implementing geometric calibration could significantly enhance performance, particularly for directional estimation and variance capture.