💻 computer science

Auditing Frozen Time-Series Foundation Models Under Informative Context Missingness

This paper presents a pre-registered audit revealing that while mechanism-balanced adaptation is less harmful than average-risk adaptation for frozen time-series foundation models under informative missingness, all learned adaptation objectives perform significantly worse than simply leaving the models unadapted, rendering comparisons between adaptation strategies uninterpretable without an explicit no-adaptation anchor.

Original authors: Muhammetalp Erdem

Published 2026-09-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Muhammetalp Erdem

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of data science, there is a growing belief that large, pre-trained computer models can predict the future of almost anything that changes over time, from electricity usage to stock prices. These models are like vast libraries of patterns, trained on massive amounts of history so they can recognize trends without needing to be taught from scratch for every new job. However, in the real world, history is rarely perfect. Sensors fail, reports arrive late, and data gets lost. When a model is asked to make a prediction based on a broken or incomplete history, the missing pieces are often not random; they happen for a reason. A sensor might stop working because a machine is overheating, or a report might be delayed because a shipment is stuck. This means the pattern of what is missing actually contains clues about what is happening. The challenge for scientists is to figure out how to teach these powerful, frozen models to pay attention to those clues without breaking their ability to forecast.

A researcher set out to test a specific idea: could they improve these models by teaching them to treat different types of missing data with equal importance? Imagine a student who usually studies by averaging all their notes. The researcher wondered if that student would do better if they forced themselves to study every type of difficult note equally, rather than just focusing on the most common ones. To test this, they took two of the most advanced forecasting models available and kept their core brains completely frozen, meaning they could not learn anything new. Instead, they attached a tiny, adjustable layer around them—a small adapter—and trained this layer to handle missing data. They tested this setup across twelve different real-world datasets, ranging from energy grids to weather patterns, and introduced missing data in four distinct ways: some random, some based on past values, and some clustered in bursts.

The researcher first compared the "equal attention" training method against the standard "average" method. In this specific head-to-head test, the equal attention approach did indeed produce slightly better results for one of the models and showed a promising trend for the other. It seemed, at first glance, that the new training strategy was a success. However, the study was designed to look deeper than just comparing two variations of the same idea. The researcher had also registered a third, crucial comparison: what happens if you simply use the original, frozen model with no adapter at all? This was the control group, the baseline of doing nothing. When they ran this comparison, the results were startling. Every single version of the new adapter, including the one that had just won the head-to-head test, performed worse than simply leaving the model alone. The tiny adjustments the adapters made actually confused the models, leading to less accurate predictions than if the adapters had never been added.

The researcher dug further to understand why the results looked so different depending on which comparison was made. They discovered that one of the twelve datasets they used was behaving very differently from the others. This dataset, tracking deaths during a pandemic, had a massive spike in values during the testing period that was not present in the training period. Because the models were designed to normalize their scores based on the training data, this single dataset ended up weighing forty times more than any other dataset in the final average. It was skewing the entire result, making the adapters look like they were failing spectacularly when compared to a simple imputation method, and making the "doing nothing" approach look surprisingly strong. When the researcher removed this one outlier dataset and re-ran the numbers, the picture became clear: the adapters were consistently worse than the frozen model across the board.

The study concludes that while the specific training method of balancing different types of missing data did show a small advantage over the standard method, it was an advantage only in a race between two losing strategies. The real finding was that for these specific models and this specific setup, the intervention itself was harmful. The best way to handle incomplete data, according to this audit, was to leave the powerful, pre-trained model exactly as it was, rather than trying to fine-tune it with a small adapter. The researcher emphasizes that this does not mean adapters will never work, but rather that in this specific context, the cost of the adjustment outweighed the benefit. They also highlighted a critical lesson for the field: when evaluating new methods, scientists must always compare them against a "do nothing" baseline. Without that anchor, it is easy to declare a winner between two variations of a technique, only to realize later that both are inferior to the original approach. The study serves as a rigorous check on the enthusiasm for adding layers to complex models, showing that sometimes the most effective tool is the one you already have.

Technical Summary: Auditing Frozen Time-Series Foundation Models Under Informative Context Missingness

Problem Statement
Time-series foundation models (e.g., Chronos, TimesFM) are typically evaluated on complete historical contexts. However, in operational environments, data is often incomplete due to sensor outages, delayed reports, or event-driven recording. Crucially, this missingness is often informative (dependent on the underlying process) rather than random. The central question addressed is whether equipping a frozen foundation model with a small, global adapter trained on a mechanism-balanced curriculum improves probabilistic forecasts under informative missingness compared to standard average-risk adaptation.

Methodology
The study employs a pre-registered audit protocol designed to isolate the effect of adaptation interventions from the backbone's inherent capabilities.

Experimental Setup: The audit covers 12 datasets (48 series) across diverse domains (economic, energy, healthcare, etc.) with a context length of $L=80$ and forecast horizon $H=24$ . Two frozen backbones are tested: Chronos-T5-small and TimesFM 2.5.
Missingness Mechanisms: The protocol distinguishes between missingness rates ( $10\%, 20\%, 40\%$ ) and mechanisms. Training masks include MCAR, observed-history MAR, and geometric bursts. Critically, test masks are held out from the training bank and include "informative" families (periodic and event-aligned) whose generators never enter adapter fitting, ensuring an out-of-distribution evaluation.
Adaptation Objectives: A global adapter is trained to minimize loss. The primary comparison is between:
1. Average Adaptation: Minimizes pooled training loss.
2. Mechanism-Balanced Adaptation: Gives equal weight to each mechanism class (MCAR, MAR, Burst) in the curriculum.
  Secondary objectives include Group DRO, CVaR, and MCAR-only adaptation.
The "No-Adaptation" Anchor: A critical component of the protocol is the frozen backbone applied directly to the incomplete context without any adapter. This serves as the baseline to determine if any intervention is beneficial, rather than just which intervention is best.
Inference and Metrics:
- Primary Metric: Normalized Interval Score (at 20% missingness), which jointly assesses sharpness and calibration.
- Statistical Hierarchy: Inference respects the nested structure (Dataset $\to$ Series $\to$ Origin). The study uses a hierarchical bootstrap and an exact cluster-level sign-flip permutation test (enumerating all $2^{12}$ sign assignments) to avoid inflating precision by treating horizons or origins as independent.
- Normalization Audit: The study explicitly audits the validity of the per-series scale denominator, checking if the training prefix represents the test period.

Key Results

Confirmatory Contrast (Balanced vs. Average):
- Mechanism-balanced adaptation performs better than average-risk adaptation.
- For TimesFM 2.5, the improvement is statistically significant (Holm-adjusted $p = 0.0020$ , effect $-2.41$).
- For Chronos-T5-small, the improvement is directional but not statistically significant at the primary rate ( $p = 0.065$ , effect $-0.74$).
- This result holds across held-out informative mask families (periodic and event).
The No-Adaptation Anchor (The Critical Finding):
- When compared against the frozen backbone (no adapter), all seven learned adaptation objectives perform worse on 11 of the 12 datasets for both backbones.
- The "balanced" method, while the "best" among the adapters, is still harmful compared to doing nothing.
- Significance of the Anchor: The confirmatory family (comparing two adaptation strategies) structurally cannot detect that the entire family of interventions is detrimental. The study reveals that the "winning" adaptation is merely the least harmful member of a set of harmful interventions.
Normalization Validity and Sensitivity:
- One dataset, covid_deaths, violates the normalization assumption: its test period (epidemic peak) has a magnitude ~250x larger than its training prefix (pre-pandemic).
- In an unweighted mean across 12 datasets, this single dataset carries ~40x the weight of others, artificially inflating the performance of methods that happen to handle this specific scale shift (like SAITS imputation) and masking the true failure of adapters.
- Excluding covid_deaths (11-dataset view): The gap between adaptation and the frozen backbone becomes unambiguous and highly significant ( $p=0.001$ ). The frozen backbone is clearly superior to all learned adapters.
Secondary Findings:
- Rate Sensitivity: The deficit of adapters relative to the frozen backbone shrinks as missingness increases. At 40% missingness, the adapter for TimesFM finally outperforms the frozen backbone, suggesting adapters only help when the context is severely degraded.
- Calibration: Raw intervals substantially under-cover (nominal 80% vs. observed ~60%). Validation-only conformal calibration fixes coverage but does not significantly alter the interval score for the focal methods, though it drastically penalizes degenerate controls (e.g., mask/age-only).

Significance and Claims
The paper's primary contribution is methodological: it demonstrates that pre-registered comparisons between variants of an intervention (e.g., Balanced vs. Average) are uninformative about whether the intervention itself is useful.

The "Anchor" Requirement: A valid audit must include a "no-intervention" anchor (the frozen model) in the reported family. Without this, a statistically significant result between two adaptation strategies can misleadingly suggest the intervention is beneficial when it is actually harmful.
Cluster-Level Inference: With a small number of top-level units (12 datasets), mean-based tests can be dominated by a single heterogeneous cluster. The study advocates pairing mean-based tests with cluster-level direction counts to ensure robustness.
Normalization Auditing: Per-series normalization constants are modeling assumptions that must be audited against input data validity. Failure to do so can invert the apparent ranking of baselines.

Conclusion
The study concludes that for the specific panel and scale tested, mechanism-balanced adaptation is not an improvement over leaving the foundation model frozen; it is the least harmful of a set of harmful interventions. The paper does not claim that lightweight adapters can never help, but rather that these specific global adapters, fitted at this scale, do not pay for themselves under the tested conditions. The findings generalize to the evaluation practice of time-series foundation models, urging researchers to report no-adaptation baselines and to rigorously audit normalization assumptions.

Technical Summary: Auditing Frozen Time-Series Foundation Models Under Informative Context Missingness

More like this