💰 quantitative finance

Emergent Latent-State Computation under Stochastic Volatility

This paper demonstrates that sequence models trained on stochastic volatility data develop interpretable, two-stage internal mechanisms to encode latent volatility states and forecast returns, revealing that performance degradation under noisy training often stems from readout misalignment rather than representation failure.

Original authors: Xiaoyu Huang, Lulu Wang

Published 2026-07-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiaoyu Huang, Lulu Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Secret Life of AI: Listening to the Invisible

Imagine you are trying to predict the weather. You can see the clouds, feel the wind, and measure the rain, but you can't see the invisible air pressure systems swirling miles above your head that actually cause the storm. In the world of artificial intelligence, scientists are trying to understand how computer brains (called neural networks) make sense of the world. Usually, they test these brains on simple, clear-cut puzzles where the answer is obvious, like solving a math problem or finishing a sentence. But the real world is messy. Often, the most important information is hidden, noisy, or changing in unpredictable ways. This is where a field called "mechanistic interpretability" comes in. It's like being a detective for AI: instead of just asking "what did the AI predict?", we ask "how did the AI figure it out?" and "what secret clues did it find inside its own brain?" This paper dives into a tricky version of this detective work, looking at how AI handles financial data where the true "volatility" (how wild the market is) is a hidden, shifting ghost that the AI can only guess at by watching the noisy numbers on the screen.

The Paper: Teaching AI to See the Invisible Ghost

In this study, researchers Xiaoyu Huang and Lulu Wang set up a digital playground to see if small AI models could learn to "see" a hidden ghost. They created a simulation of a stock market with six different assets. In their world, the actual "volatility" (how much the prices swing) is a hidden state that changes over time, like a ghost moving through a house. The AI can only see the "returns" (the price changes), which are noisy and blurry—like trying to guess where the ghost is by only hearing the creaks of the floorboards. The goal was simple: could the AI learn to predict the next price swing by first figuring out where the hidden ghost is?

The researchers trained two types of AI brains: a simple one (an MLP) that looks at each stock separately, and a more complex one (a Transformer) that can look at how the stocks talk to each other. They found that both types of AI learned a clever two-step trick. First, the AI's "hidden brain" (the middle layers) successfully built a mental map of the invisible ghost's location. Second, the AI's "output mouth" (the final layer) used that map to guess the next price swing. It's as if the AI first learned to track the ghost, and then used that tracking to predict the storm.

But here is where it gets really interesting. The researchers discovered that how the Transformer learned this trick depended on how fast the ghost was moving.

Slow Ghosts (Long Cycles): When the volatility changed very slowly (like a slow, lazy wave), the Transformer found a super-simple shortcut. It turned the raw data into a specific shape using a mathematical "filter" (a linear projection followed by a normalization step) and voila—the ghost's location was suddenly clear. It was like the AI found a magic lens that instantly focused the blurry image.
Fast Ghosts (Short Cycles): When the volatility changed quickly, the AI needed more help. It had to use its "attention" mechanism (looking at how stocks interact) and its internal "thinking" layers to piece the puzzle together.
The "Ghost" vs. The "Translator": The study also found a funny glitch. Sometimes, the AI's hidden brain knew exactly where the ghost was, but the part of the AI that made the final prediction (the output head) was a bit clumsy at translating that knowledge into a number. It was like having a brilliant detective who knows the killer's identity but stumbles when writing the report. When the researchers swapped the clumsy reporter for a better one, the AI's predictions got much better, proving that the AI had actually learned the secret all along; it just needed a better way to say it.

The paper suggests that these financial simulations are a fantastic new playground for AI scientists. Unlike simple puzzles where the answer is right there on the page, these models force the AI to do "implicit filtering"—to guess the hidden truth from noisy clues, just like humans do in the real world. By showing that AI can learn to extract these hidden states and that we can pinpoint exactly where and how it happens in the code, the authors suggest we are getting closer to truly understanding the secret lives of our digital brains.

Technical Summary: Emergent Latent-State Computation under Stochastic Volatility

Problem Statement

Mechanistic interpretability research has predominantly focused on language models and deterministic synthetic tasks where target computations are fully observable. There is a significant gap in understanding how neural sequence models internally represent latent stochastic dynamics in noisy, partially observed environments.

This paper addresses this gap using Stochastic Volatility (SV) models as a controlled benchmark. In SV models, asset returns are noisy observations of an unobserved latent log-volatility state ( $h_t$ ). Unlike ARCH/GARCH models where volatility is a deterministic function of past returns, SV models require the model to perform implicit filtering to infer hidden states from noisy data. The challenge is to determine if small neural networks trained solely to forecast future squared returns ( $r^2_{t+1}$ ) recover internal representations of the true latent volatility state ( $h_{t+1}$ ) and how these representations emerge across the network architecture.

Methodology

Experimental Setup

Data Generation: The authors utilize a controlled Multivariate Stochastic Volatility (MSV) process with $N=6$ assets. Returns ( $r_t$ ) are generated as noisy observations of a latent log-volatility state ( $h_t$ ) following a linear state-space model with cross-asset spillovers.
Task: Models are trained to predict the next-step squared returns ( $y = r^2_{T+1}$ ) given a history of $T=21$ past returns. The ground-truth latent state $h_{T+1}$ is known to the researcher but hidden from the model.
Architectures:
- Transformer: A compact encoder with one layer and one attention head. Assets are treated as tokens with return histories as features. The architecture includes linear embedding, $\ell_2$ normalization, positional embeddings, self-attention, and a feed-forward network (FFN) with GELU activation.
- MLP Baseline: A three-layer MLP that processes each asset's history independently, isolating the effect of attention-based cross-asset interactions.
Training Objectives: Two loss functions are compared:
1. MSE: Mean Squared Error on squared returns (a noisy proxy for conditional variance).
2. NLL: Gaussian Negative Log-Likelihood, which is theoretically aligned with conditional variance estimation.

Analysis Techniques

Linear Probing: Ridge regression probes are fitted to intermediate hidden representations ( $z_i$ ) to measure the linear decodability of the latent state $h_{i, T+1}$ .
Stage-wise Analysis: The authors track decodability ( $R^2$ ) at every architectural stage (embedding, normalization, attention, FFN, output) to identify where latent information emerges.
Causal Interventions:
- Output-head replacement: Replacing the learned output head with a probe-aligned readout to test if forecast degradation stems from representation failure or readout misalignment.
- Activation Ablation: Perturbing specific components (e.g., post- $\ell_2$ normalization) to measure downstream effects on decodability and forecast error.

Key Findings

1. Robust Two-Stage Latent-State Decomposition

Across both Transformers and MLPs, and regardless of loss function or activation, the models exhibit a consistent two-stage computation:

Latent Inference: Hidden representations encode substantial information about the next latent volatility state ( $h_{T+1}$ ). Linear probes achieve high $R^2$ (often >0.85) across architectures.
Forecast Mapping: The output head maps this representation to squared return forecasts.

Observation: Under MSE training, the learned output head often exhibits significantly lower alignment ( $R^2$ ) with the latent state compared to the probe, even when the hidden state is highly decodable. This suggests that forecast errors under MSE often arise from readout misalignment rather than a failure to encode the latent state.

2. Stage-wise Emergence in Transformers

The point at which latent-state decodability emerges depends critically on the volatility period ( $\omega$ ):

Long-Cycle Regimes ( $\omega \ge 365$ ): Decodability emerges immediately after $\ell_2$ normalization of the linear embedding. The representation becomes largely decodable before attention or FFN processing.
Intermediate Regimes ( $\omega \in \{30, 90\}$ ): Decodability requires additional refinement by the FFN after attention.
Short-Cycle Control ( $\omega = 21$ ): Decodability is near zero until positional embeddings are added, after which it jumps to $\approx 1.0$ . This suggests the model relies on periodicity memorization rather than dynamic filtering in this specific regime.
Comparison: The MLP shows a less modular, more diffuse emergence of decodability across layers, lacking the sharp stage-specific transitions seen in Transformers.

3. Explicit Latent Filters in Long-Cycle Regimes

In long-cycle regimes, the Transformer's computation simplifies into an explicit filter:
$\hat{h}_{i, t+1} \approx \text{RidgeProbe}\left( \frac{x_i W_{in}^\top}{\|x_i W_{in}^\top\|_2} \right)$
This indicates that most latent-state information is extracted by a learned linear projection followed by $\ell_2$ normalization. Causal ablation confirms that the post- $\ell_2$ -norm representation is the earliest stage where the latent signal becomes causally relevant for downstream forecasting; perturbing this stage causes the largest collapse in decodability.

Significance and Claims

The paper argues that stochastic volatility models provide a useful benchmark for mechanistic interpretability under noisy latent dynamics and partial observability.

Bridging Gaps: Unlike deterministic toy tasks, SV models require implicit state inference, yet the ground truth remains available for evaluation. This allows for controlled study of how models represent, transform, and use latent stochastic structure.
Mechanistic Insight: The study reveals that even small, compact Transformers can learn to perform implicit filtering that mirrors classical econometric pipelines (inference followed by prediction).
Interpretability of Financial Models: The findings suggest that financial time series forecasting tasks can serve as a bridge between mechanistic interpretability, time-series modeling, and state-space inference, offering a setting to study how neural networks handle irreducible uncertainty.

The authors maintain modesty regarding their claims, noting that their results are based on synthetic data with known latent states and small architectures. They do not claim these mechanisms scale directly to large-scale real-world financial models or that they provide a complete circuit-level account of all SV computations, but rather that they identify specific, interpretable emergent mechanisms in controlled settings.