💰 quantitative finance

A hidden Markov model for statistical arbitrage in international crude oil futures markets

This paper proposes a hidden Markov model-based statistical arbitrage strategy for international crude oil futures, demonstrating that while traditional pairs like Brent, WTI, and Dubai are unprofitable, incorporating the newly introduced Shanghai crude oil futures yields significant profits even under conservative transaction costs.

Original authors: Viviana Fanelli, Claudio Fontana, Francesco Rotondi

Published 2026-02-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Viviana Fanelli, Claudio Fontana, Francesco Rotondi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to find hidden patterns in the chaotic world of oil prices. This paper is about a team of researchers (Fanelli, Fontana, and Rotondi) who built a sophisticated "oil price radar" to find free money—or at least, very profitable money—by betting on how different types of crude oil prices move together.

Here is the story of their discovery, broken down into simple concepts.

1. The Three Oil Brothers

Imagine the global oil market has three main characters:

Brent: The old, reliable British oil (the "Grandfather").
WTI: The classic American oil (the "Uncle").
Shanghai: The new, energetic Chinese oil (the "Young Cousin"), which only started trading in 2018.

Historically, oil traders knew that the Grandfather and the Uncle usually move in sync. If one goes up, the other usually goes up. They are "cointegrated," which is a fancy math way of saying they are tied together by an invisible elastic band. If they drift too far apart, the band snaps them back together.

The researchers asked a simple question: Does the Young Cousin (Shanghai) also get tied to this elastic band? And if so, can we make money betting on when the band stretches and snaps back?

2. The Problem: The "Hidden Mood Swings"

In the past, traders used simple rules: "If the price gap gets too big, bet that it will shrink." This is like a dog chasing a ball; it works until the dog gets tired or the ball disappears.

The problem is that the oil market isn't a calm lake; it's a stormy sea with different "moods." Sometimes the market is calm, and prices snap back quickly. Other times, it's chaotic, and prices might drift apart for a long time. A simple rule doesn't know the difference between a calm day and a stormy day.

3. The Solution: The "Hidden Mood Detective" (Hidden Markov Model)

The researchers built a smarter system. They didn't just look at the prices; they built a Hidden Markov Model (HMM).

Think of this model as a mood detective inside a foggy room.

The prices are the things you can see.
The market regime (calm vs. stormy) is the "mood," which is hidden in the fog. You can't see the mood directly, but you can guess it by watching how the prices behave.

The model uses a "filter" (like a pair of smart glasses) to guess the current mood based on the price movements. It asks: "Is the market in a 'Calm' state where prices snap back fast, or a 'Chaos' state where they might drift?"

Once the model guesses the mood, it adjusts its strategy. If it's calm, it bets aggressively. If it's chaotic, it waits.

4. The Strategy: Playing the "Elastic Band"

The researchers tested five different ways to play the game:

The "Always On" Strategy: Bet every time the prices move even a tiny bit. (Too expensive because of trading fees).
The "Backward Looking" Strategy: Bet only when the gap is huge compared to the past. (Like looking in a rearview mirror).
The "Forward Looking" Strategy (The Winner): Use the "Mood Detective" to predict where the prices will go next. Bet only when the model predicts the gap is about to snap back.

The Result: The "Forward Looking" strategy, which used the hidden mood detective, made the most money. It was like having a crystal ball that told you exactly when the elastic band was about to snap.

5. The Big Surprise: The "New Kid" is the Star

The most exciting finding was about the Shanghai oil futures.

The researchers found that the Shanghai oil is actually faster at snapping back to the group than the older Brent or WTI oils.
It's like the Young Cousin is the most obedient one; when the group drifts apart, the Cousin runs back to the group first.
Because the Shanghai market is newer and less mature, there are more "mistakes" (temporary price errors) to exploit. The researchers showed that including this new oil in their trio made the strategy much more profitable, even after paying for trading fees.

6. Why This Matters

It works even when pairs fail: Sometimes, Brent and WTI don't move together perfectly. But when you add Shanghai into the mix, the three of them do move together. It's like a three-legged race; if two legs stumble, the third can keep the team moving.
It beats the "Buy and Hold": Just buying oil and waiting (like holding a stock) lost money during the test period. This active strategy made money by constantly adjusting.
It's robust: Even if trading fees go up, this strategy still works because it's so smart about when to trade.

The Takeaway

The paper proves that in the complex world of oil trading, you don't just need to look at the numbers; you need to understand the hidden moods of the market. By using a mathematical "mood detective" to track the relationship between old and new oil markets, investors can find profitable opportunities that simple traders miss. It's a reminder that in finance, the smartest move is often to understand the context (the regime) before making a bet.

1. Problem Statement

The paper addresses the challenge of identifying and exploiting statistical arbitrage opportunities in international crude oil futures markets. While classical pairs trading strategies (e.g., between Brent and WTI) are well-established, the authors investigate whether the introduction of the Shanghai crude oil futures (launched in 2018) creates new, profitable arbitrage opportunities when combined with traditional benchmarks.

Key challenges addressed include:

Market Segmentation: The Shanghai market has structural differences (currency denomination in CNY, trading restrictions, specific benchmark composition) that may cause temporary deviations from global benchmarks.
Non-Stationarity of Relationships: Cointegration vectors between oil futures are not constant; they suffer from structural breaks and time-varying regimes, which can degrade the performance of static pairs trading strategies.
Latent Regimes: Market participants cannot directly observe the underlying "regime" (e.g., high volatility vs. low volatility, or different mean-reversion speeds) governing the spread between prices.

2. Methodology

A. Data and Cointegration Analysis

Assets: The study analyzes three futures: Brent (B), WTI (W), and Shanghai (S).
Data: Daily and weekly continuous price series from March 26, 2018, to June 30, 2023.
Cointegration: The authors confirm that the three series are jointly cointegrated (integrated of order one, $I(1)$ ), forming a single cointegrating relationship. This is established via the Johansen trace test and Vector Error Correction Model (VECM) estimation.
Spread Definition: The cointegration spread ( $S_t$ ) is defined as a linear combination:
$S_t = \lambda_0 + \lambda_B F^B_t + \lambda_S F^S_t + \lambda_W F^W_t$
Empirical results show that deviations from equilibrium are primarily corrected by adjustments in the Shanghai futures price ( $F^S_t$ ), indicating a unidirectional long-run causal relationship from Brent/WTI to Shanghai.

B. Stochastic Modeling: AR-HMM

To capture the time-varying nature of the spread, the authors model $S_t$ as an Autoregressive process modulated by a Hidden Markov Model (AR-HMM).

Dynamics: The spread follows a regime-switching autoregressive process:
$S_{t+1} = \gamma(X_t) + \alpha(X_t)S_t + \eta(X_t)z_t$
Where:
- $X_t$ is an unobservable Markov chain with $N$ states (regimes).
- $\gamma(X_t)$ , $\alpha(X_t)$ , and $\eta(X_t)$ are regime-dependent intercept, autoregressive coefficient, and volatility, respectively.
- $|\alpha_i| < 1$ ensures mean reversion within each regime.
Continuous-Time Analogy: The discrete model approximates a continuous-time Ornstein-Uhlenbeck process modulated by a hidden Markov chain (OU-HMM).

C. Parameter Estimation: Filter-Based EM Algorithm

Since the Markov chain $X_t$ is unobserved, the authors employ Stochastic Filtering combined with the Expectation-Maximization (EM) algorithm.

Filtering: Recursive equations (based on a change of measure technique) estimate the conditional probability of the current regime ( $\hat{X}_t$ ) given the history of spread observations.
Online Estimation: The EM algorithm updates parameter estimates ( $\hat{\gamma}, \hat{\alpha}, \hat{\eta}, \hat{\Pi}$ ) recursively as new data arrives. This allows the model to adapt dynamically to changing market conditions without retraining on the full historical dataset.
Initialization: The algorithm uses OLS estimates from a short initial window to initialize parameters, with robustness checks performed via random restarts to avoid local optima.

D. Trading Strategies

Five statistical arbitrage strategies are implemented and tested:

Plain Vanilla (PV): Opens a position whenever the spread is non-zero (benchmark).
Probability Interval (ProbI): Backward-looking; opens when the spread exceeds a dynamic Bollinger band (based on rolling mean/variance).
Prediction Interval (PredI): Forward-looking; opens when the spread falls outside the one-step-ahead prediction interval derived from the AR-HMM model.
Realized Increment (RI): Backward-looking; trades based on significant deviations in spread increments (z-score of changes).
Predicted Increment (PI): Forward-looking; trades based on significant deviations in the predicted spread increments from the AR-HMM.

Exit Rule: Positions are closed when the spread crosses zero (mean reversion).
Costs: Transaction costs are explicitly modeled based on bid-ask spreads (Shanghai has the highest costs due to market immaturity).

3. Key Contributions

Tripartite Cointegration: First study to jointly assess cointegration among Brent, WTI, and the newly introduced Shanghai futures, demonstrating that a three-contract portfolio can be profitable even when pairwise cointegration fails (e.g., between Brent and WTI in certain windows).
Regime-Switching Framework: Proposes a discrete-time AR-HMM for the cointegration spread, generalizing previous work by allowing parameters (mean, speed of reversion, volatility) to switch based on a latent market state.
Dynamic Filtering: Implements an online, filter-based EM algorithm that continuously updates model parameters, ensuring the strategy remains tuned to current market regimes rather than relying on static historical estimates.
Forward-Looking Superiority: Demonstrates that strategies utilizing the model's predictive capabilities (PredI and PI) significantly outperform traditional backward-looking strategies (ProbI and RI).

4. Empirical Results

Performance Metrics

The strategies were tested on an out-of-sample period (July 2022 – June 2023) using annualized returns ( $R$ ), Sharpe Ratios ($SR$), and trading-window returns ($RTW$).

Model-Based Strategies Win: The PredI (Prediction Interval) strategy consistently delivered the highest performance.
- Annualized Return: ~15.18% (PredI) vs. ~10.43% (ProbI) vs. -0.91% (PV).
- Sharpe Ratio: ~1.18 (PredI) vs. ~0.83 (ProbI).
- Statistical Significance: The PredI strategy's Sharpe ratio is statistically significant at the 10% level, and the White Reality Check (WRC) confirms it as the best-performing strategy in the set.
Impact of Shanghai Futures: Including the Shanghai contract enhanced profitability. The authors attribute this to the faster speed of adjustment of Shanghai prices to restore long-run equilibrium compared to Brent and WTI.
Transaction Costs: Strategies remained profitable even under conservative transaction cost assumptions. While higher costs reduced returns, the AR-HMM strategies (PredI/PI) were robust.
Risk Profile: The proposed strategies exhibited lower Value-at-Risk (VaR) and Maximum Drawdown (MDD) compared to passive benchmarks (like holding an Oil ETF), which suffered significant losses during the test period.

Robustness

Initialization: Results were stable across 100 random initializations of the EM algorithm, indicating the model does not rely on specific local optima.
Contract Selection: The framework remained effective when replacing Shanghai with Dubai futures or using four-contract portfolios, provided cointegration existed among the group.
Sample Size: Performance degraded if the training sample was too small (e.g., starting too close to the test period), highlighting the need for sufficient data to estimate the regime-switching parameters accurately.

5. Significance and Conclusion

The paper provides strong evidence that statistical arbitrage in crude oil markets can be significantly enhanced by:

Expanding the asset universe beyond traditional pairs to include emerging market benchmarks (Shanghai), which offer faster mean-reversion dynamics.
Adopting a regime-switching stochastic model that accounts for structural breaks and latent market states, rather than assuming static relationships.
Utilizing forward-looking signals derived from online filtering, which allow traders to anticipate spread behavior more effectively than historical statistical thresholds.

The findings suggest that the "newness" of the Shanghai futures market creates a window of opportunity for arbitrageurs, and that sophisticated filtering techniques are essential to capture these opportunities profitably in a partially segmented global market.