Information-theoretic analysis of temporal dependence in discrete stochastic processes: Application to precipitation predictability

This paper introduces a robust information-theoretic framework based on predictability gain to quantify temporal memory in discrete stochastic processes, demonstrating its superiority over traditional criteria and revealing that daily precipitation in the contiguous United States is well-described by low-order Markov chains with distinct regional and seasonal variations.

The Gist

The Big Idea: How Much Does the Past Matter?

Imagine you are trying to guess what happens next in a story.

• Scenario A: You are flipping a coin. If it lands on Heads, does that change the odds of the next flip? No. The coin has no memory. The past doesn't matter.
• Scenario B: You are watching a movie. If the hero just got shot, the odds of them running away next are very different than if they just won a prize. The story has memory.

This paper is about figuring out how much memory a system has. Specifically, the authors looked at rainfall. They wanted to know: Does it rain today because it rained yesterday? Or does it rain because it rained three days ago? Or is rain just random luck?

The Problem: The "Goldilocks" Trap

Scientists have tried to answer this before using two famous tools: AIC and BIC.

• Think of AIC as a student who loves to over-study. They think, "I need to remember everything from the last 100 years to predict the weather!" They pick models that are too complicated.
• Think of BIC as a student who is too lazy. They say, "I'll just guess based on today. Who needs history?" They pick models that are too simple.

Both tools often get it wrong because they are trying to pick the "best" model from a pre-made list, rather than actually listening to what the data is saying.

The New Solution: The "Predictability Gain" Meter

The authors invented a new tool called Predictability Gain (PG).

Imagine you are playing a guessing game.

1. Round 1: You guess the next word in a sentence. You get it right 50% of the time.
2. Round 2: You look at the previous word. Now you get it right 70% of the time.
3. Round 3: You look at the two previous words. Now you get it right 72% of the time.
4. Round 4: You look at the three previous words. Now you get it right 72.1% of the time.

The Predictability Gain is the extra "bonus" you get by looking further back.

• If looking back 1 day gives you a huge bonus, but looking back 2 days gives you almost nothing, the system has a short memory (1 day).
• If looking back 5 days gives you a huge bonus, the system has a long memory.

The authors built a "statistical flashlight" (using a method called Bootstrap Testing) to shine on this bonus. They ask: "Is this extra bonus real, or is it just a lucky coincidence?" If the bonus disappears after looking back 1 day, they stop there. They don't force a complex model if a simple one works.

The Experiment: The Great US Rain Hunt

They applied this new tool to daily rain data from thousands of weather stations across the United States (from 1990 to 2020). They turned the rain into a simple code: 0 (Dry) or 1 (Wet).

What they found:

1. Mostly Short Memories: For most places and most times, rain is like a First-Order Markov Chain.

   • Translation: Whether it rains tomorrow depends almost entirely on whether it rained today. Looking back further than yesterday usually doesn't help much.
   • The Metaphor: Rain is like a sticky note. If it's wet today, it's likely to be wet tomorrow. But by the third day, the note has dried up, and the past doesn't matter anymore.

2. Seasonal Shifts (The "Weather Personality"):

   • Winter on the West Coast: The memory is stronger. If it rains in California in January, it's very likely to rain the next day, and the day after.
     • Why? This is due to "Atmospheric Rivers"—massive, long-lasting bands of moisture that sweep across the coast for days. It's like a heavy blanket that stays on for a while.
   • Summer in the Southeast: The memory is also stronger here.
     • Why? This is due to "Subtropical Circulation." Hot, humid air gets stuck, causing daily thunderstorms. It's like a humidifier that keeps the air wet day after day.
   • The Rest of the Time: In many other places and seasons, the rain is more random (memory = 0). It's like a coin flip; just because it rained yesterday doesn't mean it will rain today.

Why Does This Matter?

1. Better Forecasts: If we know exactly how much "memory" the rain has, we can build better computer models. We don't need to use super-complex math for places where the rain is simple.
2. Saving Money: Complex models take a lot of computer power (and electricity) to run. By using this new tool, scientists can say, "Hey, for this specific town in July, we only need to look at yesterday's rain." This saves massive amounts of computing power.
3. A New Tool for Everything: While they tested this on rain, this "Predictability Gain" meter can be used for anything that changes over time: stock markets, heartbeats, or even how people talk to each other.

The Takeaway

The authors didn't just find out when it rains; they built a better ruler to measure how much the past influences the future. They proved that for most of the US, rain is a "one-day thinker," but in specific seasons and places, it has a longer memory. Their new method is more accurate and less biased than the old tools, helping us understand the hidden patterns in our chaotic world.

Technical Summary

1. Problem Statement

Understanding the temporal dependence (memory) of complex stochastic processes is crucial for improving predictability and developing efficient models. In many real-world systems, such as weather patterns, future states depend on past events, violating the assumption of memoryless (independent and identically distributed) processes.

• The Challenge: Determining the intrinsic "memory" ($m$) of a process—defined as the minimum number of past states required to predict the next state accurately—is difficult.
• Limitations of Existing Methods: Traditional model selection criteria like the Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC) are often model-dependent and prone to bias. AIC tends to favor overly complex models (overfitting), while BIC often favors overly simplistic ones (underfitting), especially with finite data. Furthermore, these methods select the "best" model from a predefined set but do not guarantee that the selected model captures the true underlying dynamics.

2. Methodology

The authors propose a novel information-theoretic approach based on Predictability Gain (PG) to estimate memory without relying on specific model assumptions.

A. Theoretical Framework: Predictability Gain

• Block Entropy ($H_r$): The method utilizes block entropy, a generalization of Shannon entropy for sequences of length $r$. For a process with memory $m$, $H_r$ becomes a linear function of $r$ for all $r \geq m$.
• Predictability Gain ($G_u$): Defined as the negative second discrete derivative of block entropy:
  $$G_u = -(H_{u+2} - 2H_{u+1} + H_u)$$
  Mathematically, $G_u$ is equivalent to the conditional mutual information between the next state and the state $u+1$ steps back, conditioned on the intermediate $u$ states.
  • If $G_u > 0$, the process has correlations extending beyond order $u$.
  • If $G_u = 0$ for all $u \geq m$, the process has memory $m$.
• Properties: The authors prove that $G_u$ is additive, bounded, and that the total predictability gain corresponds to the reduction in uncertainty when moving from a memoryless description to a full history-based description.

B. Estimation Algorithm (PG Estimator)

Since real-world data is finite and transition probabilities are unknown, the authors develop a robust estimation procedure:

1. Bootstrap Resampling: To handle statistical fluctuations in finite data, they use a bootstrap method. Under a null hypothesis $N(\eta)$ (that the process has memory $\eta$), they generate synthetic sequences using estimated transition probabilities of order $\eta$.
2. Hypothesis Testing: They test the global null hypothesis $N(\eta)$ against the alternative that memory is greater than $\eta$.
   • They calculate the p-value ($q_u^{(\eta)}$) for each order $u \geq \eta$ by comparing the observed predictability gain of the real data against the distribution of gains from the bootstrap samples.
3. Fisher's Method: To control the Type I error rate when performing multiple hypothesis tests (for $u = \eta, \dots, u_{max}$), they combine the individual p-values into a single statistic using Fisher's method.
4. Memory Selection: The estimated memory $\hat{m}_{PG}$ is the smallest $\eta$ for which the combined p-value exceeds the significance threshold $\alpha$ (i.e., the null hypothesis cannot be rejected).

3. Key Contributions

1. Novel Estimator: Introduction of the Predictability Gain (PG) estimator, which is model-free and directly quantifies the information provided by higher-order temporal dependencies.
2. Superior Performance: Demonstrated through simulations that the PG estimator outperforms AIC and BIC in accuracy and robustness, particularly for binary stochastic processes with varying memory orders ($m=0$ to $4$) and finite sequence lengths.
3. Robustness to Data Limitations: The method effectively handles the "undersampled regime" (where the number of possible states exceeds the data length) by using block entropy estimators (NSB estimator) and bootstrap resampling.
4. Diagnostic Capability: Unlike AIC/BIC which always return a model order, the PG method can identify when the data does not fit any tested memory order, providing a clearer indication of model inadequacy.

4. Results

A. Simulation Validation

• Setup: The authors tested the PG, AIC, and BIC estimators on synthetic binary sequences ($L=2$) with known memory orders ($m=0, 1, 2, 3, 4$) and varying lengths ($N=100, 200, 300$).
• Findings:
  • PG Estimator: Achieved the highest accuracy across almost all combinations. Its performance improved consistently as sequence length increased.
  • AIC: Tended to overestimate memory, with accuracy decreasing as $N$ increased (due to the expansion of the search space for higher orders).
  • BIC: Tended to underestimate memory, performing poorly for short sequences and higher memory orders, though it was more consistent than AIC for large $N$.

B. Application to Precipitation Data

The method was applied to daily precipitation occurrence data (binary: wet/dry) from ~8,000 stations across the contiguous United States (1990–2020).

• Memory Order: The vast majority of station-month pairs were best described by first-order Markov chains ($m=1$). Higher-order dependencies ($m \geq 2$) were rare.
  • Seasonality: Memory 0 (independent days) was more common in winter (Jan/Feb) and summer (Jul/Aug), while memory 1 dominated in spring and autumn.
• Spatial and Seasonal Correlations ($G_0$):
  • West Coast: Strongest correlations ($G_0$) occurred in winter, consistent with the passage of frontal systems and atmospheric rivers causing consecutive wet days.
  • Southeast: Strongest correlations occurred in summer, driven by persistent subtropical circulation and daily convective storms.
  • Central US: Generally showed weaker correlations throughout the year.
• Transition Probabilities: The study confirmed that dry days are highly likely to be followed by dry days ($\hat{p}(0|0) > 0.5$) everywhere, while rainy days are more likely to be followed by dry days, except in the specific seasonal regimes mentioned above where persistence is high.

5. Significance and Implications

• Parsimonious Modeling: The findings suggest that for most locations and times, simple first-order Markov models are sufficient to describe precipitation occurrence. This allows for significant reductions in computational cost for physical climate simulations without sacrificing predictive skill.
• Data-Driven Forecasting: The framework provides a rigorous, data-driven tool to identify regions and seasons where memory effects are strong, guiding the development of real-time forecasting schemes.
• General Applicability: While applied to precipitation, the methodology is general and applicable to any discrete stochastic process where understanding short-term temporal dependencies is critical, including ecology, neuroscience, social interactions, and AI.
• Methodological Advancement: The paper establishes a robust alternative to traditional model selection criteria, offering a more interpretable measure of temporal dependence that avoids the biases inherent in AIC and BIC.

In conclusion, the paper successfully bridges information theory and climatology, providing a powerful tool to quantify memory in stochastic processes and demonstrating that precipitation dynamics in the US are largely governed by first-order dependencies with distinct seasonal and regional patterns.