🤖 machine learning

Evaluating Generative Time-Series Models on Data with Point Masses

This paper reveals that standard evaluation protocols for generative time-series models often fail to account for point masses (e.g., zeros), leading to misleading benchmarks where mismatched data structures can reverse conclusions and where autoregressive hurdle models significantly outperform conditional flows when evaluated with careful, matched protocols.

Original authors: Jian Xu

Published 2026-08-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jian Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to predict the future, but instead of guessing the weather or stock prices, you are asking it to predict things that mostly don't happen. Think about a ride-sharing app: for most hours of the day, no one requests a ride. Or a solar panel: for half the day, it produces zero energy. In the world of data science, these are called "time series with point masses." It's a fancy way of saying the data is full of zeros, interrupted by occasional bursts of activity.

To predict these patterns, scientists use "generative models." You can think of these models as incredibly sophisticated artists. They learn the shape of the data and then try to draw new, realistic pictures of what the future might look like. Most of the time, these artists use a technique called "flow matching," which is like stretching and twisting a smooth, continuous sheet of clay to match the data. The problem is, a smooth sheet of clay can never truly form a sharp, distinct dot. It can get very close, but mathematically, it can't create a perfect "zero" that stays a zero; it can only make a tiny, thin spike that looks like a zero.

Why does this matter? Because if you are building an app to manage electricity or deliver spare parts, you need to know exactly when nothing is happening. If your model thinks there's a tiny chance of a ride being requested when there actually is none, you might waste money or miss a critical delivery. This paper asks a simple but tricky question: Are these smooth, artistic models actually bad at predicting the "nothing" moments, and are we even measuring them correctly?

The author of this paper decided to put these smooth, artistic models to the test on data that is full of zeros, like ride-share requests and weather patterns. They wanted to see if a model that is mathematically designed to be "smooth" could ever truly master the art of predicting "nothing." What they found was a bit of a plot twist: the smooth models were actually losing the game on the specific task of predicting when things happen, but not because they were bad artists or because their smooth nature was the problem. They were losing because the judges were using the wrong scorecard.

First, the researchers discovered that the standard way of testing these models was like judging a marathon runner by only looking at the last few steps of the race. In many datasets, the "zeros" (the quiet moments) happen in specific patterns, like a dry spell before a storm. The standard testing method often picked windows of time that didn't have enough zeros to represent the real world. For example, on one dataset called "rideshare," the actual data was 47% zeros, but the testing windows only had 5% zeros. It was like testing a chef who specializes in making bread by only asking them to make a cake. When the researchers fixed this and made sure the test windows actually looked like the real data, the smooth models started to look even worse on the occurrence statistics, yet they still held the best overall score (CRPS) on three of the six datasets. This revealed a confusing reality: the model that was "winning" the general accuracy contest was actually "losing" the specific contest of predicting the zeros.

Next, they introduced a clever trick to see what the models were actually learning. They took the model's predictions and shuffled the order of the events, like shuffling a deck of cards. This kept the "what" (the amount of rain or rides) exactly the same but destroyed the "when" (the timing and sequence). They found that for some datasets, like rideshare, the model's ability to predict the timing of zeros was so poor that shuffling the cards didn't make it any worse. The model wasn't actually learning the pattern of the dry spells; it was just guessing. In fact, on the rideshare dataset, a much simpler model called an "autoregressive hurdle" (which is basically a logistic classifier that checks one step at a time) beat the fancy smooth model by a massive factor of 153. That's like a bicycle beating a Ferrari by a factor of 153.

The paper also showed that these smooth models are incredibly unstable. If you train the same model five times with slightly different starting points (called "seeds"), the results can swing wildly. On one dataset, the model's performance on predicting zeros varied by up to 62% just by changing the seed. Meanwhile, the simpler models were rock-solid and gave the same answer every time.

Finally, the author tried a "hybrid" approach: they took the smooth model's ability to predict the size of the events (like how much rain) and swapped out its bad timing with a simple, explicit rule for when zeros happen. They thought this would be the perfect fix. But it wasn't. The hybrid model only worked on one dataset and performed worse on the others. The real winner was the simple, step-by-step classifier, not the fancy hybrid.

The big takeaway is that the issue isn't that smooth models are inherently broken or incapable of the task. The problem is that the way we usually test them hides their flaws and gives a misleading ranking. The paper suggests that for data full of zeros, we need to fix how we evaluate these models to see the real picture. If we don't fix the testing methods, we might keep praising the wrong tools and missing the ones that actually work, regardless of whether they are "smooth" or "simple." The author is careful to say this isn't a failure of the model class itself, but a specific mismatch between the evaluation protocol and the data.

Technical Summary: Evaluating Generative Time-Series Models on Data with Point Masses

Problem Statement
Many time-series datasets used to benchmark generative models contain a significant probability mass on a single value (typically zero), representing events such as "no rain," "no ride requested," or "no part ordered." This creates a "mixture" data law: an atom at zero and a continuous distribution over positive values. However, state-of-the-art continuous-state generative models (e.g., flow matching, diffusion models) are constructed to transport absolutely continuous source distributions. Under standard regularity conditions, these models produce outputs that are absolutely continuous, meaning the probability of generating an exact zero is theoretically zero. Consequently, these models are tasked with reproducing a feature (the atom) they cannot inherently represent. The paper investigates whether this structural mismatch systematically handicaps continuous models and whether explicit occurrence processes offer a solution, while simultaneously identifying critical flaws in how these models are currently evaluated.

Methodology and Evaluation Protocols
The author conducts a rigorous re-evaluation of generative time-series models, focusing on six datasets with varying degrees of zero-inflation (from 1% to 94%). The study compares a conditional flow matching model against several baselines derived from the intermittent demand literature, including Markov chains, semi-Markov processes, and autoregressive hurdle models.

The methodology introduces three specific analytical controls to isolate evaluation artifacts from model performance:

Window Resemblance Analysis: The author analyzes the standard "rolling-origin" evaluation protocol, where models are scored on the final $H$ steps of each series. They demonstrate that for non-stationary atom processes, these evaluation windows can have a zero-rate drastically different from the overall dataset (e.g., a dataset with 46.9% zeros evaluated on windows with only 5.3% zeros).
The Decorrelation Control: To measure the specific contribution of temporal coupling to a statistic, the author proposes a control where sample indices are permuted independently at each time step. This operation destroys the temporal coupling (run-length structure) while leaving the per-timestep predictive marginals—and thus the Continuous Ranked Probability Score (CRPS)—invariant by construction. This allows for a direct measurement of how much a model's learned trajectory coupling contributes to a chosen metric.
Seed Sensitivity Analysis: The study benchmarks models over five training seeds to assess stability, contrasting the stochastic nature of flow models with deterministic baselines.

Key Results

Evaluation Artifacts: The standard rolling-origin protocol often evaluates models on regimes unrepresentative of the dataset. In one case (rideshare), the dataset is 46.9% zeros, but the evaluation windows are only 5.3%. This discrepancy was severe enough to invert the author's own conclusions, initially making a strong autoregressive hurdle model appear to be a "cautionary tale" due to exposure bias, only for it to emerge as the strongest performer under a series-wise split.
Model Performance: Under a matched protocol (series-wise split), an autoregressive occurrence hurdle (a logistic classifier rolled out step-by-step) outperforms the conditional flow on five of six datasets. On the rideshare dataset, the hurdle model beats the flow by a factor of 153 in terms of zero-run length error ( $W_1$ ).
Metric Divergence: The ranking of models depends heavily on the choice of metric. The autoregressive hurdle dominates on occurrence statistics (zero-run length, spell quantiles, survival functions), while the flow often achieves the best CRPS. The two families of metrics do not agree on model ordering.
Coupling Contribution: The decorrelation control reveals that the learned temporal coupling in flow models contributes little or nothing to zero-run statistics on several datasets (e.g., rideshare, car_parts). On rideshare, the flow's performance on zero-run length is indistinguishable from its own decorrelated samples, indicating the coupling adds no value for this specific statistic.
Instability: The flow model's occurrence statistics are highly unstable across training seeds, varying by up to 62% on certain datasets. In contrast, the occurrence baselines are deterministic.
Hybrid Failure: Substituting a semi-Markov occurrence process into the flow (a "hybrid" approach) improved performance on only one dataset and worsened it on five, suggesting that the specific choice of occurrence mechanism (autoregressive vs. semi-Markov) is more critical than simply adding an explicit process.

Significance and Claims
The paper claims that the observed performance gap is not necessarily due to the inherent inability of continuous-state models to handle atoms, but rather stems from measurement problems in current evaluation protocols. The author does not claim that continuous generative modeling is fundamentally broken; rather, they argue that:

Evaluation Windows Matter: The standard rolling-origin protocol can yield misleading results if the evaluation window's atom structure differs significantly from the dataset.
Coupling Must Be Verified: A model should not be credited with learning temporal structure unless it outperforms its own decorrelated samples on the relevant statistic.
Metric Selection is Critical: Different occurrence statistics (e.g., run-length vs. spectral) yield different model rankings, and practitioners must choose metrics aligned with their specific use cases.
Stability Reporting: Reporting single-run results for stochastic generative models on occurrence statistics is insufficient; the spread across seeds must be reported.

The paper concludes that the "obvious fix" of substituting a semi-Markov process into a flow is not the solution; instead, the autoregressive hurdle approach is the superior method for these specific datasets, and the evaluation framework must be adjusted to properly reflect the data's atom structure.

Technical Summary: Evaluating Generative Time-Series Models on Data with Point Masses

More like this