Revealing Stratosphere-Troposphere Interactions via Regime-Based Transfer Entropy

Technical Summary: Revealing Stratosphere-Troposphere Interactions via Regime-Based Transfer Entropy

Problem Statement
The macroscopic interaction between the stratosphere and troposphere is a primary driver of subseasonal-to-seasonal (S2S) climate predictability. However, quantifying the causal coupling within these high-dimensional, non-linear fluid systems remains a fundamental challenge. Classical statistical methods, such as cross-correlation and linear Granger causality, fail to capture non-linear dependencies and state-synchronizations, often producing spurious results due to strong internal autocorrelation. While information-theoretic tools like Transfer Entropy (TE) offer a model-free framework for detecting causality, their direct application to continuous spatial fields (e.g., geopotential height) is severely constrained by the "curse of dimensionality."

Methodology
To overcome these limitations, the authors introduce a regime-based Transfer Entropy framework that integrates probabilistic coarse-graining with information theory. The methodology proceeds in three stages:

1. Probabilistic Coarse-Graining via HMMs:

   • Daily geopotential height anomalies are projected into a low-dimensional state space using Kernel Principal Component Analysis (KPCA) with a Gaussian radial basis function kernel.
   • A Hidden Markov Model (HMM) with Gaussian mixture emissions is trained on these reduced trajectories. The number of initial "micro-states" is optimized to capture the intrinsic metastability of the circulation.
   • A network-based community detection algorithm clusters these micro-states into distinct, non-overlapping macro-circulation regimes (e.g., specific phases of the North Atlantic Oscillation or polar vortex states).
   • Instead of hard classification, the method utilizes the continuous posterior probabilities of these regimes as dynamic variables. This preserves noise-filtered information about the system's proximity to regime boundaries.

2. Lag-Specific Transfer Entropy (TE):

   • The authors define a modified, lag-specific TE to measure directed information flow from a source process $X$ to a receiver $Y$ at a specific time lag $\tau$.
   • Crucially, the conditioning in the TE formula is shifted to the receiver's past at the tested lag $\tau$ (rather than the immediate past) to account for the long-memory, non-Markovian inertia of macro-states.
   • A Normalized Transfer Entropy (NTE) is proposed, scaling the directed flow by the conditional Shannon entropy of the receiver given its own past. This normalization isolates the external coupling strength from the receiver's internal predictability, allowing for the detection of subtle triggers that would otherwise be masked by high internal inertia.

3. Statistical Significance and Spatial Mapping:

   • To address the non-Markovian nature of the data, a conservative ensemble-permutation surrogate framework is employed. This method shuffles the chronological order of independent realization blocks (e.g., winter seasons) while preserving internal block dynamics, creating a null distribution that maintains the exact spectral and memory properties of the original data.
   • Soft composite maps are constructed by weighting spatial fields with the posterior probabilities of the causal regimes at specific lags, revealing the physical footprint of information transfer.

Key Results
Applying this framework to ERA5 and NCEP/NCAR reanalysis data (1960–present) across 500 hPa (mid-troposphere), 100 hPa (lower stratosphere), and 20 hPa (middle stratosphere) yields the following findings:

• Asymmetric Vertical Coupling:

  • Upward Flow (Troposphere $\to$ Stratosphere): Dominates in magnitude with a rapid synoptic response (3–7 days). In ERA5, specific tropospheric regimes (particularly those associated with blocking patterns) evolve into highly persistent, multi-week saturation, sustaining significant information transfer for over 40 days. NCEP/NCAR shows a similar initial peak but lacks this long-term persistence, decaying by day 30.
  • Downward Flow (Stratosphere $\to$ Troposphere): Exhibits a weaker but distinct long-lived memory. The signal is highly state-selective, driven almost exclusively by the weakened polar vortex regime (Stratospheric Regime 1, associated with Sudden Stratospheric Warming events). This downward influence peaks at subseasonal timescales (30–40 days).

• Regime-Specific Non-Linearity:

  • The downward signal is non-linear and threshold-like; only the specific "weakened vortex" state triggers the long-delayed tropospheric response. Other stratospheric regimes show negligible downward impact.
  • The upward flow is also regime-dependent, with specific tropospheric blocking configurations acting as active triggers for stratospheric rearrangement.

• Dataset Discrepancies:

  • Comparisons between ERA5 and NCEP/NCAR reveal that the legacy NCEP/NCAR dataset artificially delays the downward signal (peaking at ~40 days vs. ~31 days in ERA5) and inflates the amplitude of the response. This is attributed to NCEP/NCAR's coarse vertical resolution and simplified radiative physics, which slow the descent of anomalies and filter out all but the most extreme SSW events.

• Macro-Scale Feedback Loop:

  • The analysis uncovers a multi-week, macro-scale positive feedback loop: Tropospheric blocking (Regime 3) drives the stratosphere toward a weakened vortex state (Regime 1), which subsequently descends over ~30 days to reinforce the original tropospheric blocking pattern.

Significance and Claims
The paper claims that this regime-based TE framework successfully isolates complex dynamical triggers and causal resonance within highly inertial Earth systems, overcoming the limitations of traditional linear methods and high-dimensional TE.

• Methodological Advancement: The study demonstrates that shifting from raw spatial fields to a latent probabilistic space via HMMs allows for the rigorous quantification of non-Markovian, long-memory dynamics without succumbing to the curse of dimensionality.
• Physical Insight: By decomposing TE into regime-specific components, the authors identify that stratosphere-troposphere coupling is not a continuous linear process but a state-dependent, threshold-driven mechanism. The findings provide a rigorous statistical basis for the physical understanding that specific tropospheric configurations can trigger stratospheric events, which in turn modulate surface weather weeks later.
• Predictability: The identification of a multi-week positive feedback loop between the polar vortex and tropospheric blocking highlights a potential source of S2S predictability, suggesting that the stratosphere acts as a "dynamical capacitor" that stores and reflects information back to the surface.

The authors conclude that while the current method utilizes time-homogeneous HMMs, the framework offers a robust avenue for future development toward non-autonomous systems to better capture seasonal evolution and critical transitions in a changing climate.