Multi-Year Geospatial Reasoning using Interannually-Consistent Historical Predictions as a Free Input Modality

Technical Summary: Multi-Year Geospatial Reasoning using Interannually-Consistent Historical Predictions

1. Problem Statement

Operational Earth Observation (EO) systems, such as the Copernicus Land Monitoring Service (CLMS) High Resolution Layer (HRL) Croplands product, typically generate annual land-cover maps using deep learning models that process satellite time series (Sentinel-1/2, meteorology) on a year-by-year basis. Despite accumulating a dense, multi-year archive of their own predictions and utilizing ancillary data (like the Base Vegetation Layer, BVL) during post-processing, these systems suffer from two critical limitations:

1. Lack of Temporal Feedback: The model for a given year (e.g., 2024) has no access to its own past predictions (e.g., 2017–2023). Consequently, it cannot leverage the temporal stability of perennial crops (e.g., olives) or the logic of crop rotations, treating each year as an isolated classification task.
2. Inconsistent Signal Integration: Ancillary signals, such as the BVL mask which identifies non-crop areas, are applied as rule-based post-processing steps outside the neural network. This creates a representation gap where the model cannot "read" masked pixels in its own historical data, treating them as undefined rather than as a specific signal (masked vs. missing data).

The authors posit that recurring EO models should treat their own prediction history and external ancillary layers as free input modalities to enable multi-year geospatial reasoning directly within the model, rather than relying on brittle, rule-based post-processing.

2. Methodology

The proposed approach modifies an existing operational pixel-based Transformer architecture to ingest historical predictions and consistent mask representations as new input channels.

A. Input Modalities and Architecture

• Baseline: A multi-source Transformer encoder processes 24-step (dekadal) time series of Sentinel-2 optical bands, Sentinel-1 SAR backscatter, derived indices, and AgERA5 meteorology. Static terrain scalars are processed via a dense encoder.
• Historical Prediction Input (CTY History): The model ingests two aligned sequences for each pixel: the predicted class codes and their associated confidence scores (winning-class probability) from previous years (2017–2029).
  • CTY Embedding Encoder: Instead of treating class codes as raw integers or simple scalars, the authors introduce a learnable embedding encoder. Each past class code $c_t$ is mapped to a learnable vector and scaled by its confidence $p_t$ ($\mathbf{e}_t = \text{Embed}(c_t) \cdot p_t$).
  • Temporal Positioning: Learnable sinusoidal position embeddings are added to preserve the order of years.
  • Masked Token: A specific learnable token (index 1) is reserved for "externally masked" (BVL) observations, distinguishing them from "no-data" (padding) and valid crop classes.
• Auxiliary Scalars: Two cheap, hand-crafted scalars derived from history are added: "crop stability" (fraction of past years matching the current crop) and "years since change."
• Fusion: The output of the CTY Embedding Encoder is concatenated with the optical/SAR/meteo embeddings before the final MLP classification head.

B. Training Strategy

• End-to-End Training: Unlike approaches that fine-tune large geospatial foundation models, the authors train a compact, end-to-end model (2.6M parameters) because their dataset is large (5.4M labeled pixels). This allows the model to jointly learn the weighting of phenological signals against historical predictions.
• Loss Function: The primary loss is focal cross-entropy. A soft self-supervised consistency loss is added as an auxiliary term, where a small linear head predicts class logits from the history embedding alone, trained to agree with the main target at a low weight ($\lambda=0.1$). This encourages the model to use history as a soft prior without over-relying on it.
• Handling Imbalance: Class imbalance is managed via balanced batch sampling (equalizing class frequencies per batch). The authors explicitly found that adding inverse-frequency loss weighting on top of this sampling caused performance collapse due to double-compensation.

C. Representation of Masked Observations
The study investigates how the model should interpret pixels masked by the BVL in both the target year and the history. The optimal strategy involves:

1. Using the dedicated "masked" token in the history sequence.
2. Either folding masked observations into the "no-crop" class or assigning them a dedicated class in the target year, provided the history is consistently remapped.

3. Key Contributions

1. Free Input Modality: Demonstrating that a model's own interannual prediction history, when encoded as confidence-scaled, time-ordered tokens, serves as a powerful, label-free input that significantly improves classification.
2. CTY Embedding Encoder: A novel architecture component that transforms categorical prediction history into a learnable, confidence-weighted representation, allowing the model to attend over the year axis.
3. Consistent Signal Representation: Showing that representing external masks (BVL) consistently within the model's input space (distinguishing masked from missing data) improves crop classification performance without altering the external mask itself.
4. Error Profile Correction: Moving beyond simple accuracy metrics to demonstrate that historical reasoning corrects a "recall-skewed" error profile, reducing over-prediction while maintaining high recall.

4. Results

Evaluated on a pan-European dataset of ~5.4M labeled pixels (18 crop classes + no-crop):

• Overall Performance: Adding the prediction history increased the crop-only F1 score by 1.6 percentage points (pp) (from 0.787 to 0.803).
• Perennial and Tree Crops: The gains were most pronounced for crops with stable, multi-year histories:
  • Olives: +4.6 pp
  • Fruits: +3.7 pp
  • Nuts: +3.2 pp
  • Dry pulses: +2.9 pp
• Consistent Mask Representation: Representing the BVL mask consistently in both history and target years added approximately 2.5 pp to the crop-only F1 score.
• Precision-Recall Balance: The baseline model was recall-skewed (Precision 0.769 vs. Recall 0.810), indicating over-prediction. The history-aware model rebalanced this (Precision 0.806 vs. Recall 0.803), moving 13 of 18 crop classes closer to the precision-recall diagonal.
• Ablation Findings:
  • Raw scalar or time-series encodings of history provided negligible gains.
  • Increasing model size to 12M parameters led to overfitting.
  • Inverse-frequency loss weighting combined with balanced sampling caused a catastrophic drop in performance (F1 dropped to 0.64).

5. Significance and Claims

The paper claims that this approach offers a low-cost recipe for improving any recurring geospatial or foundation model that emits class maps. By feeding the model its own "interannual-fixed" history, the system effectively distills the logic of legacy rule-based post-processors (which correct anomalies and enforce consistency) and learns to improve upon them directly.

The authors argue that this shifts the paradigm from "single-year classification + rule-based correction" to "multi-year geospatial reasoning." The model learns the "temporal grammar" of land use (rotations, persistence, gradual change) directly from its outputs. The approach is presented as modular and applicable to any operational EO product that updates annually, potentially enhancing geospatial foundation models without requiring new labeled data. The results suggest that the model can achieve cleaner maps with fewer spurious year-to-year flips and smoother field interiors, reducing the need for downstream sieving and interannual overrides.