🤖 machine learning

Martingale-Consistent Self-Supervised Learning

This paper introduces a martingale-consistent self-supervised learning framework that enforces coherence between coarse and refined predictions under partial observation, thereby improving model robustness and calibration compared to standard invariance-based objectives.

Original authors: Moritz Gögl, Hanwen Xing, Christopher Yau

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Moritz Gögl, Hanwen Xing, Christopher Yau

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Shaky Crystal Ball"

Imagine you are trying to predict the weather.

Scenario A: You look out the window and see dark clouds. You guess, "It might rain."
Scenario B: Five minutes later, you check a radar app and see a massive storm front moving in. You update your guess to, "It will definitely rain."

This is normal. As you get more information, your prediction should change.

However, the paper points out a weird flaw in how current AI models (specifically "Self-Supervised Learning" models) are trained. These models are often asked to make predictions based on different amounts of information: sometimes they see the whole picture, sometimes only half, and sometimes just a tiny slice.

The problem is that these models often act like a shaky crystal ball.

When they see only the dark clouds (little info), they might say, "50% chance of rain."
When they see the radar (lots of info), they might say, "90% chance of rain."
The Flaw: If you asked the model before it saw the radar, "What do you think your prediction will be once you see the radar?" a smart model should say, "I expect it to stay around 50%." It shouldn't secretly expect to jump to 90% or drop to 10%. If it does, it means the model is "drifting" or being inconsistent. It's like a student who guesses a random answer on a quiz, then later changes it to the right answer, but claims they knew the right answer all along.

The Solution: The "Martingale" Rule

The authors introduce a concept called Martingale Consistency. In plain English, this is a rule that says: "Your current guess should be the average of all your future guesses."

Think of it like a treasure hunt:

The Coarse View (Little Info): You are standing at the starting line. You have a map that shows the general area. You guess the treasure is in the "North Field."
The Refined View (More Info): You walk closer. Now you see a specific tree. You update your guess: "The treasure is under that specific oak tree."
The Martingale Rule: Before you walked to the tree, your "North Field" guess should have been the average of all the specific spots you might eventually find yourself looking at. If your starting guess was "North Field," but you secretly expected to end up looking at the "South Mountain" once you got closer, your starting guess was wrong.

The paper argues that standard AI training doesn't care about this rule. It just teaches the AI to be right when it has all the info, and right when it has some info, without checking if those two answers make sense together.

How They Fixed It

The researchers built a new training method that forces the AI to follow this "Martingale" rule. Here is how they did it, using a Chef Analogy:

The Setup: Imagine a chef (the AI) trying to guess the ingredients in a soup.
The Partial View: The chef is blindfolded and only allowed to smell the soup (coarse view). They guess, "It's probably tomato soup."
The Refinement: The chef takes off the blindfold and sees the whole pot (refined view). They guess, "It's definitely tomato soup with basil."
The Trick (The Imputer): To train the chef, the researchers use a "magic helper" (an imputer). This helper takes the blindfolded chef's guess and imagines two different versions of what the full soup might look like.
- Imagination A: A soup with lots of basil.
- Imagination B: A soup with a little basil.
The Test: The researchers ask the chef: "If you had to guess the full soup based on only the smell, would your guess match the average of Imagination A and Imagination B?"
- If the chef's "smell guess" was wildly different from the average of the two "full soup" guesses, the AI gets a penalty.
- This forces the AI to learn that its early guesses must be a stable "center point" for all its future, more detailed guesses.

What They Found

The team tested this on many types of data:

Time Series: Like stock prices or heart rate monitors (where you might miss some seconds of data).
Tabular Data: Like medical records or bank forms (where some columns might be missing).
Images: Like looking at a picture where half of it is covered by a black box.

The Results:

Better Stability: When the AI was forced to follow the Martingale rule, it became much more stable. When information was missing, it didn't panic or make wild swings in its predictions.
Better Calibration: The AI became more honest about its confidence. If it said "50% chance," it was actually right about 50% of the time, even when it didn't have all the data.
Works Everywhere: This worked whether the AI was learning with labels (supervised) or without labels (self-supervised), and whether it was looking at numbers or pictures.

The Takeaway

The paper claims that by adding this "consistency check" (making sure the AI's early guesses are the logical average of its later guesses), we can build AI models that are more robust and reliable when they are forced to work with incomplete information. It's like teaching a student not just to memorize the final answer, but to understand how their thinking evolves as they learn more facts.

Technical Summary: Martingale-Consistent Self-Supervised Learning

Problem Statement

Self-supervised learning (SSL) is widely deployed in scenarios where information availability is dynamic, such as when dealing with shorter histories, missing features, or partially observed images. In these settings, a model is often queried under multiple "information sets" (e.g., coarse vs. refined views). A fundamental requirement for such models is coherence: the prediction made under a coarse information set should match the expected prediction after refinement.

Mathematically, if $F_1 \subseteq F_2$ represent nested information sets (where $F_2$ contains more information than $F_1$ ), and $g_\phi(x_F)$ is a predictor, the martingale property dictates:
$E[g_\phi(x_{F_2}) \mid x_{F_1}] = g_\phi(x_{F_1})$
This implies that before observing additional information, the model should not systematically expect its future predictions to drift upward or downward. Standard SSL objectives (e.g., contrastive learning, masked reconstruction) typically train predictors for individual views independently or enforce invariance (making views look similar). They do not enforce this directional consistency, potentially leading to models that perform well on specific views but lack global coherence across nested information sets.

Methodology

The authors propose a Martingale-Consistent SSL Framework that enforces the martingale property as a training objective.

1. Objective Function

The total loss function combines a standard predictive loss with a martingale consistency penalty:
$\mathcal{L}_{total} = \mathcal{L}_{pred} + \lambda_{imp}\mathcal{L}_{imp} + \lambda_{mart}\mathcal{L}_{mart}$

Predictive Term ( $\mathcal{L}_{pred}$ ): A standard SSL loss (e.g., classification cross-entropy or reconstruction MSE) applied to the refined (complete) view.
Martingale Term ( $\mathcal{L}_{mart}$ ): This term penalizes deviations from the martingale identity. It requires sampling nested information sets $F_1 \subseteq F_2$ . The loss is defined as:
$\mathcal{L}_{mart} = \mathbb{E}_{F_1 \subseteq F_2} \left[ \| g_\phi(x_{F_1}) - \mathbb{E}[g_\phi(x_{F_2}) \mid x_{F_1}] \|^2 \right]$
Imputation Loss ( $\mathcal{L}_{imp}$ ): An auxiliary loss used to train an imputer network $q_\psi$ that approximates the conditional distribution of the complete input given the partial view.

2. Unbiased Two-Sample Monte Carlo Estimator

Directly computing the conditional expectation $\mathbb{E}[g_\phi(x_{F_2}) \mid x_{F_1}]$ is intractable. A naive Monte Carlo average introduces bias due to variance. The authors propose an unbiased two-sample estimator:

Given a coarse view $x_{F_1}$ , sample two conditionally independent refinements $x^{(a)}_{F_2}$ and $x^{(b)}_{F_2}$ using the learned imputer $q_\psi$ .
Compute the coarse prediction $u = g_\phi(x_{F_1})$ and refined predictions $v_a = g_\phi(x^{(a)}_{F_2})$ , $v_b = g_\phi(x^{(b)}_{F_2})$ .
The estimator is:
$\hat{\mathcal{L}}^{2MC}_{mart} = (u - v_a)^\top (u - v_b)$
Theoretical analysis (Appendix B.1) proves this estimator is unbiased for the squared error of the martingale identity.

3. Implementation Variants

The framework is instantiated in two primary spaces:

Prediction-Space: The martingale penalty is applied directly to the model's output predictions.
Latent-Space: The penalty is applied to the encoder's latent representations ( $z_{F_1}$ vs. $z_{F_2}$ ). This is particularly useful for high-dimensional data (images, long sequences) where enforcing consistency in observation space is impractical.
EMA Targets: To stabilize training, the refined targets ( $v_a, v_b$ or $z_a, z_b$ ) can be computed using an Exponential Moving Average (EMA) copy of the network, preventing moving-target instability.

Key Contributions

Identification of a Structural Gap: The paper identifies that standard SSL treats predictors as a family of conditional expectations indexed by information sets but fails to enforce the necessary consistency relations (martingale property) between them.
Practical Training Recipe: The authors introduce a method to enforce martingale consistency without modality-specific redesign. This includes prediction- and latent-space penalties and the unbiased two-sample Monte Carlo estimator based on stochastic refinement.
Empirical Validation: The framework is evaluated across temporal, tabular, and image benchmarks in both semi-self-supervised (with labels) and fully self-supervised (label-free) settings.

Experimental Results

The framework was tested on synthetic and real-world datasets including time-series (UCI HAR, Traffic, Stock), tabular data (Adult, Bank Marketing), and images (CIFAR-10, STL-10).

Robustness under Partial Observation: Enforcing martingale consistency consistently improved downstream probe accuracy when the amount of observed information was reduced (low completeness levels $c \in \{0.05, 0.2, \dots, 0.8\}$ $c \in {0.05, 0.2, \dots, 0.8}$ ).
- Time-Series: In the semi-self-supervised setting, martingale variants achieved the best mean accuracy on all 7 datasets, with significant relative gains (e.g., +155.6% on Character Trajectories, +30.9% on T-SIM-RC).
- Tabular: Martingale variants outperformed baselines on all 5 static tabular datasets, with gains up to +34.1% on synthetic data.
- Images: On CIFAR-10, the latent-martingale variant improved mean accuracy from 0.311 to 0.383 (+23.2%) over the base model.
Calibration: Models trained with martingale consistency showed significantly better calibration (lower Expected Calibration Error and Anytime Regret) under partial observation compared to standard SSL.
Violation Reduction: The training objective successfully reduced martingale violations (the discrepancy between coarse predictions and expected refined predictions), particularly in low-completeness regimes.
Comparison to Invariance: Unlike standard consistency objectives that force views to be identical (invariance), martingale consistency allows predictions to update as information is revealed while preventing systematic drift.

Significance and Claims

The paper claims that martingale consistency provides a probabilistic perspective on SSL models deployed under varying masking patterns or context lengths. By enforcing agreement in expectation under information refinement, the method allows predictions to update naturally as new data arrives while preventing systematic drift.

The authors position this as a structural inductive bias that captures information left unconstrained by standard objectives like SimCLR or BYOL. They suggest this is particularly relevant for foundation models, which are routinely queried under varying context lengths and incomplete observations, yet existing objectives do not explicitly enforce the coherence required for such dynamic deployment. The work demonstrates that closing this gap improves robustness, calibration, and representation stability, offering a principled way to handle partial information regimes in self-supervised learning.