GNSS-FM: A Self-Supervised Foundation Model for Daily GNSS Displacement Time Series

Technical Summary of GNSS-FM: A Self-Supervised Foundation Model for Daily GNSS Displacement Time Series

Problem Statement
Global Navigation Satellite System (GNSS) networks provide continuous, high-precision measurements of Earth-surface displacement, essential for monitoring tectonic deformations and earthquake cycles. While datasets now encompass tens of thousands of stations spanning decades, most existing machine learning approaches for GNSS time series are fully supervised and task-specific (e.g., discontinuity detection or forecasting). This creates a bottleneck because labeled data for geophysically relevant tasks (such as slow slip events or specific seismic offsets) are scarce, incomplete, or dependent on external catalogs. Conversely, vast amounts of unlabeled GNSS data are freely available. The challenge lies in adapting self-supervised learning—successful in speech and vision—to GNSS data, which is non-stationary over multi-year windows, combines slow tectonic drift with abrupt seismic steps and seasonal loading, and contains common data gaps and measurement noise.

Methodology
The authors present GNSS-FM, a self-supervised foundation model adapted from the wav2vec 2.0 framework and the SeisLM architecture, specifically redesigned for daily geodetic time series.

• Data and Preprocessing: The model is pretrained on daily displacement time series (East, North, Up components) from 17,652 globally distributed stations provided by the Nevada Geodetic Laboratory. Preprocessing involves outlier removal, masking equipment-related offsets, and filling gaps using linear interpolation (short gaps) or bootstrap-based synthesis (long gaps). Each day is assigned a reliability label (observation, interpolation, synthesis, or padding) to weight training loss and condition the model.
• Dual-Stream Input: The model processes two parallel streams:
  1. Displacement Stream: Maintains absolute displacement levels and long-range context (tectonic drift, seasonal patterns).
  2. Velocity Stream: Derived from first-order differences of displacement, highlighting local dynamics and reducing non-stationarity.
     Both streams are normalized and transformed via inverse hyperbolic sine to compress large values.
• Architecture:
  • Encoders: Separate strided 1D convolutional encoders process each stream, downsampling the 512-day input windows to 120 feature steps.
  • Transformer: A dual-stream transformer encoder (12 layers, 768 hidden dimension) processes the features. It includes bidirectional cross-attention blocks to allow information exchange between streams.
  • Conditioning: Station metadata (latitude, longitude, height) and reliability labels are injected into the transformer layers via FiLM (Feature-wise Linear Modulation) to modulate the representations.
  • Vector Quantization: Targets are generated by discretizing convolutional features using a vector quantizer with 4 groups of 320 codevectors. The velocity stream feeds the first two groups, and displacement feeds the last two.
• Pretraining Objective: The model employs a masked latent prediction objective. Continuous sub-windows of features are masked, and the model predicts the vector-quantized targets for these positions using an InfoNCE contrastive loss. The training curriculum gradually increases masking probability and span length. Additional regularization terms prevent codebook collapse and encourage diverse usage.
• Downstream Adaptation: After pretraining, the encoder is fine-tuned for two tasks:
  1. 90-day Displacement Forecasting: Uses a cross-attention decoder to predict future trajectories, with a second stage adding Patch-TST-style residual branches for uncertainty estimation.
  2. Seismic Step Localization: Uses a temporal DETR (Detection Transformer) head to predict event probabilities and localize step dates within the window.

Key Contributions

1. Dual-Stream Representation: A novel input format combining displacement (for long-range context) and velocity-like increments (for local dynamics), addressing the trade-off between learning local changes and maintaining deformation context.
2. Adapted Pretraining Framework: A masked latent prediction framework featuring reliability-aware masking and loss weighting, station metadata conditioning, and a collapse-resistant grouped spherical quantizer tailored for non-stationary geodetic data.
3. Foundation Model Performance: Demonstration that a single pretrained backbone can be fine-tuned for distinct downstream tasks (forecasting and event localization), outperforming strong task-specific baselines.

Results

• Pretraining Analysis: Validation probes show steady improvement in reconstruction quality, with cosine similarity between predicted and target representations increasing from ~0.85 to ~0.90. Codebook analysis reveals that learned representations specialize in distinct signal types, including seismic offsets, tectonic drift, and seasonal patterns. Ablation studies confirm that the dual-stream approach is superior to single-stream models, with the velocity stream being critical for extrapolation.
• Displacement Forecasting: On the test set, GNSS-FM Stage 1 achieved a Root Mean Squared Error (RMSE) of 6.78 mm, significantly outperforming the supervised Patch-TST baseline (58.77 mm) and the Calibrated Base Forecast (77.05 mm). While Mean Absolute Error (MAE) was slightly higher for the foundation model, the lower RMSE indicates superior handling of rare, large errors caused by seismic events. The model also demonstrated better generalization stability between validation and test splits compared to supervised baselines.
• Seismic Step Localization: The GNSS-FM model achieved an end-to-end localization F1 score of 0.429, a substantial improvement over the hybrid baseline (0.035). The model's primary advantage lies in accurate date localization rather than just window-level event detection. The conditional localization success rate (given a positive detection) was 0.769, with a median absolute error of two days for correctly identified events.

Significance and Claims
The paper claims that self-supervised pretraining is a promising approach for GNSS time series analysis, effectively bridging the gap between the abundance of unlabeled data and the scarcity of labels. The results suggest that the model learns a meaningful latent vocabulary that implicitly captures the classical decomposition of geodetic signals (trend, periodic, offset, noise). By outperforming task-specific supervised baselines in both forecasting and localization, GNSS-FM demonstrates that a single pretrained backbone can support multiple downstream applications, reducing dependence on labeled data and generalizing well to stations unseen during training. The authors note that while the model currently processes stations independently, the foundation model architecture offers a pathway for future integration of spatial context and broader geodetic data sources.