Horizon-Constrained Rashomon Sets for Chaotic Forecasting

This article bridges predictive multiplicity and chaotic dynamics by introducing horizon-constrained Rashomon Sets, a framework that describes how model diversity evolves with the prediction horizon in chaotic systems to enable decision-oriented model selection that significantly improves downstream utility in safety-critical applications.

The Gist

Imagine you are trying to predict the weather, control traffic lights, or forecast wind energy for a city. You have a team of prediction experts (computer models). In a perfect, calm world, if all these experts agree on short-term forecasts, you can trust that they will also agree on long-term forecasts.

But the real world is chaotic. This means that tiny differences in how a model starts its calculations can later lead to completely different outcomes. It is like the "butterfly effect": a butterfly flapping its wings in Brazil could trigger a tornado in Texas weeks later.

This article addresses a specific problem: What happens when you have many different models that are all equally good at predicting the near future, but they begin to contradict each other the further you look into the future?

Here is the breakdown of their solution using simple analogies:

1. The "Rashomon" Problem (Too Many Truths)

In machine learning, there is a concept called the Rashomon effect. Imagine a crime scene where five different witnesses tell slightly different stories, yet all of them are technically "correct" based on the evidence they have. In AI, this means you can have many different models that all achieve the same accuracy score today but make completely different predictions for tomorrow.

Normally, scientists simply pick a single "best" model and hope for the best. But in chaotic systems (like weather or traffic), this is dangerous because these "equally good" models could diverge wildly from each other over time.

2. The New Idea: "Horizon-Limited" Sets

The authors introduce a new way of thinking called horizon-limited Rashomon sets.

• The Analogy: Imagine a group of hikers starting on the same trail (the present). For the first hour (short-term), they all stay on the same path and agree on where they are going. This is the "Rashomon set."
• The Chaos: Because the terrain is treacherous (chaotic), the path splits the further they hike (longer time). One hiker goes left, another right. The group, once a tightly connected unit, begins to scatter.
• The Discovery: The article mathematically proves that this group of "consensus" models in chaotic systems does not simply scatter randomly; it shrinks exponentially. The further you look, the fewer models remain that are still "good enough" to be considered. The speed at which they scatter is determined by something called the Lyapunov exponent (think of this as the "speed of chaos" for this specific system).

3. The Solution: Choosing the Right Guide for the Job

Since the group of models shrinks and changes over time, you cannot simply pick the model that is best at predicting "in 5 minutes" and assume it will also be best for "in 5 hours."

The authors developed a method called decision-aligned selection.

• Old Way: Choose the model with the lowest error rate for the specific time point you are looking at.
• New Way: Consider the goal. If you are managing a wind farm, your goal is not just to have "accurate wind speed," but to "avoid unnecessarily shutting down turbines."
• The Metaphor: Imagine you need to cross a river.
  • Model A is great at predicting water levels for the next 10 minutes but poor for 1 hour.
  • Model B is okay for 10 minutes but surprisingly good for 1 hour.
  • If you need to cross in 1 hour, Model B is the better choice, even if Model A looked "more accurate" in a standard test.

The article's algorithm considers the "horizon" (how far ahead you need to plan) and the "utility" (what actually matters for the decision) to select the model that helps you make the best decision, not just the one with the lowest mathematical error.

4. The Results

The team tested this on:

• Synthetic Chaos: Mathematical simulations like the Lorenz system (a classic weather model).
• Real World: Real wind power data, traffic speeds in Los Angeles, and weather patterns.

The Result:
By using their method to select models based on the required decision rather than just raw accuracy, they improved the quality of decisions by 18% to 34%.

• For wind energy, this meant better energy planning.
• For traffic, it meant better signal control.
• For weather, it meant better resource allocation.

Summary

The article argues that in chaotic worlds, "accuracy" changes over time. A model that is perfect for tomorrow could be useless for next week. Instead of selecting a single "best" model, we should look at the entire group of "good enough" models, observe how they scatter over time, and select the one best suited for the specific decision we need to make at that specific moment. This approach bridges the gap between the mathematics of chaos and the practical needs of AI.

Technical Summary

Technical Summary: Horizon-Restricted Rashomon Sets for Chaotic Predictions

Problem Statement

Predictive multiplicity (the existence of multiple models with equal accuracy but diverging predictions) and chaotic dynamics (where small initial differences lead to exponentially diverging outcomes) are two fundamental challenges in machine learning that have historically evolved in isolation. While the Rashomon effect is well-characterized for static prediction tasks, existing frameworks do not account for temporal dynamics. Conversely, literature on chaotic prediction typically optimizes for a single "best" model, without characterizing the multiplicity of equally accurate alternatives.

This discrepancy poses critical risks in safety-critical applications (e.g., wind power dispatch, traffic management, weather forecasting). In chaotic systems, models that agree on short-term predictions can diverge exponentially once the prediction horizon is extended. Current selection methods based on static accuracy metrics like RMSE offer no principled way to choose between near-optimal models when downstream utility depends on long-term behavior. This work bridges the gap by investigating how chaos alters the structure of predictive equivalence over time and how models can be selected to maximize downstream decision utility rather than just prediction accuracy.

Methodology

Theoretical Framework

The authors introduce horizon-restricted Rashomon sets, a theoretical extension of the classical Rashomon set that varies with prediction lead time $k$.

• Definition: For a lead time $k$ and a tolerance $\epsilon_k$, the set $R^{(k)}_{\epsilon_k}$ contains all models $h$ whose loss $L_k(h)$ lies within $\epsilon_k$ of the optimal loss $L^*_k$.
• Exponential Contraction: The work proves (Theorem 1) that in chaotic systems, the effective size of this set contracts exponentially with lead time. The contraction rate is determined by the maximum Lyapunov exponent ($\lambda_{max}$), which quantifies the system's sensitivity to initial conditions.
• Lyapunov-Weighted Metrics: To address the varying importance of different horizons, the authors propose Lyapunov-weighted Rashomon ratios (Definition 2). These metrics weight model deviation with an exponential decay factor $e^{-\lambda_{max}k}$ and provide tighter bounds on effective predictive ambiguity than classical unweighted metrics (Theorem 2).

Algorithmic Approach

The work develops decision-aligned selection algorithms (Algorithm 1) that select models from the time-varying Rashomon set based on downstream utility functions rather than prediction errors.

• Process: The algorithm samples models from the intersection of horizon-restricted sets, computes the optimal downstream action for each model at various horizons, and aggregates utility using Lyapunov weights.
• Convergence: A theoretical analysis (Theorem 3) establishes bounds on the sample size required for this selection process and shows that the necessary sample size grows exponentially with the effective decision horizon, weighted by the Lyapunov exponent.

Experimental Implementation

• Model Construction: The authors construct Rashomon sets through systematic enumeration of Reservoir Computer variants. They vary hyperparameters including reservoir size ($N_r$), spectral radius ($\rho$), sparsity ($p$), and leakage rate ($\alpha$), resulting in 1,080 candidate configurations.
• Lyapunov Estimation: The maximum Lyapunov exponent is estimated using the Rosenstein algorithm on time-delay embedded phase spaces and verified against synthetic ground truths.
• Adaptive Tolerance: The tolerance $\epsilon_k$ is defined adaptively based on the empirical performance range at each horizon, keeping the set size manageable (10–100 models) while reflecting horizon-specific equivalence.

Key Results

Synthetic Systems

Experiments on the Lorenz-96 and Kuramoto-Sivashinsky systems validate the theoretical claims:

• Set Contraction: Empirical measurements confirm that the cardinality of the Rashomon set declines exponentially with lead time, with decline rates closely matching theoretical predictions based on $\lambda_{max}$ ($R^2 > 0.95$).
• Chaos Strength: Increasing chaos strength (via the forcing parameter $F$) accelerates set contraction. Under strong chaos, the set collapses to fewer than 10 models by horizon 30, while weaker chaos maintains approximately 50 models.

Real-World Applications

The framework was evaluated on three real-world datasets: SDWPF Wind Power, METR-LA Traffic, and ERA5 Weather.

• Decision Utility: The proposed decision-aligned selection method improved downstream decision quality by 18–34% compared to baselines (single best, ensemble average, random selection).
• Horizon Dependence: Improvements were monotonic with prediction horizon. For instance, utility gains in wind power increased from 18.4% at 6 hours to 27.2% at 24 hours. In traffic, gains rose from 21.3% (30 min) to 31.7% (60 min).
• Architectural Diversity: Including diverse model families (Reservoirs, CNNs, Transformers) in the candidate pool yielded additional utility gains (up to 38%) compared to selection within a single family, suggesting that architectural diversity complements hyperparameter diversity.

Significance and Claims

The work claims to establish the first rigorous connection between chaos theory and predictive multiplicity. Its main contributions are:

1. Theoretical Unification: It demonstrates that chaos fundamentally alters predictive equivalence, causing the space of "equally good" models to shrink exponentially as predictions approach the system's prediction horizon.
2. Practical Guidance: It provides a principled framework for model selection in safety-critical chaotic domains, moving beyond accuracy-centered metrics toward utility-centered selection.
3. Performance: The framework consistently outperforms standard selection strategies in real-world chaotic prediction tasks, offering significant improvements in decision quality while maintaining competitive prediction accuracy.

The authors acknowledge limitations, noting that the framework relies on accurate estimates of the Lyapunov exponent (which can be difficult in high-dimensional or partially observed systems) and requires access to a sufficiently large pool of near-optimal models. They suggest that future work should investigate online adaptation, multi-objective optimization, and applications in climate modeling and neural ODEs.