💻 computer science

An Ensemble PPO Trading Agent Across Real Limit Order Book and Long-Horizon OHLCV Data: Diagnosis, Correction, and Honest Evaluation

This paper presents a transparent empirical study of a PPO-based Bitcoin trading agent that diagnoses and corrects critical training failures to achieve modest long-horizon outperformance over Buy-and-Hold, while simultaneously demonstrating that real-time limit order book data often yields a rational "no-trade" policy due to transaction costs outweighing micro-price movements, ultimately advocating for the value of reproducible negative-result reporting over polished success narratives.

Original authors: william darryl towa

Published 2026-07-03

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: william darryl towa

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to trade Bitcoin. You want it to be smarter than just buying and holding the coin, but you also want to avoid it losing all its money. This paper is a "behind-the-scenes" documentary of that process, where the author, William Darryl Towa, decides to be brutally honest about everything that went wrong, what went right, and why the robot sometimes just decides to sit on the couch and do nothing.

Here is the story of the research, broken down into simple concepts:

1. The Two Different Training Camps

The author didn't just use one set of data. He trained his robot in two very different "gyms":

Gym A (The High-Speed Microscope): This used a super-detailed, second-by-second view of the Bitcoin order book (the list of people waiting to buy or sell) from a specific week in 2021. It's like watching a race car driver in slow motion, seeing every tiny twitch of the steering wheel.
Gym B (The Long-Haul Highway): This used 2.4 years of hourly price charts. It's like watching the same driver over several years, seeing how they handle different weather, traffic, and road conditions.

2. The First Big Crash: "The Frozen Robot"

When the author first tried to train the robot, it immediately broke. The robot made one trade, and then for 195,000 steps, it just froze. It stopped learning and stopped moving.

The Diagnosis: The author found two reasons for this "frozen" state:

Confusing Numbers: The robot was fed numbers that were wildly different in size. Imagine trying to compare the weight of a feather (0.001) to the weight of a blue whale (85,000). The robot got confused because the whale drowned out the feather. The author fixed this by "normalizing" the data (putting everything on the same scale).
A Broken Scorecard: The robot was being graded on a test that was mathematically broken. Sometimes, the test gave it a score of "1,000" and sometimes "-1,000" for the exact same action, just because of a tiny math error. This made the robot think the world was chaotic and unpredictable, so it gave up. The author fixed the math so the scores made sense.

3. The "Lazy" Robot: When Doing Nothing is the Smartest Move

After fixing the crash, the robot trained on the High-Speed Microscope (Gym A) data. It learned a very strange lesson: Never trade.

At first, the author thought the robot was broken or didn't have enough data to learn. But then he realized the truth: The robot was actually being smart.

The Analogy: Imagine you are at a flea market. Every time you buy something and try to sell it 10 seconds later, the market charges you a 30% fee. Even if the price of the item goes up by 1%, you still lose money because of the fee.
The Reality: In the high-speed data, the price of Bitcoin barely moved in one second (often 0.00000%). But the fee to trade was huge (0.30%).
The Conclusion: The robot realized that any trade would lose money. So, the most profitable strategy was to sit still and do nothing. The author calls this "economically rational inaction." It wasn't a failure; it was the robot correctly figuring out that the game was rigged against trading.

4. The Long-Haul Success: A Modest Win

When the robot trained on the Long-Haul Highway (Gym B) data (the 2.4 years of hourly charts), it did something different. It learned to trade, but very carefully.

The Result: The market dropped by about 21% during the test period.
- If you just held Bitcoin (Buy-and-Hold), you lost 20.82%.
- The robot lost 19.42%.
The Takeaway: It didn't make a fortune. It didn't beat the market by a huge margin. But it lost less money than the person who just held on. It was a small, consistent victory, like a runner who finishes a marathon slightly ahead of the pack in a very bad race.

5. The "Ensemble" Filter: A Gatekeeper That Did Nothing

The author tried to combine the two robots. He built a "gatekeeper" (an ensemble filter) that would only let the Long-Haul robot trade if the High-Speed robot saw a good opportunity.

The Test: He tested this on the one week where both data sets overlapped.
The Result: The gatekeeper blocked 64 potential trades, but it didn't change the final result. Why? Because the Long-Haul robot was already sitting on its hands, doing nothing, most of the time. The gatekeeper was like a bouncer at a club that was already empty; he stopped people from entering, but since no one was trying to get in anyway, the club stayed empty.
The Honesty: Instead of hiding this "null result," the author published it. He argued that admitting "this part didn't work" is more valuable than faking a success.

6. The Big Lesson: "Honest Failure" is Better than "Polished Lies"

The most important part of this paper isn't the trading strategy; it's the attitude.

In the world of AI and finance, researchers often only publish the "happy endings" where the robot makes millions. This paper is different. It says:

"Here is exactly why our first attempt crashed."
"Here is why our second robot decided to do nothing (and why that was actually correct)."
"Here is a part of our system that didn't work, and here is the proof."

The author believes that sharing these "negative results" and "diagnoses" helps the whole community learn faster than sharing a polished, perfect story that might be hiding hidden flaws. He released all his code and notes so anyone can check his work, proving that transparency is the best way to move science forward.

In short: The paper is a story about a robot that learned to trade Bitcoin, crashed because of bad math, learned that sometimes the best trade is no trade, and finally learned to lose a little less money than everyone else—all while the author promised to tell the whole truth, even the embarrassing parts.

Technical Summary: An Ensemble PPO Trading Agent Across Real Limit Order Book and Long-Horizon OHLCV Data

Problem Statement
The paper addresses the challenge of training Reinforcement Learning (RL) agents for cryptocurrency trading under two contrasting data regimes: a short window of real, high-frequency Limit Order Book (LOB) data and a long window of lower-frequency OHLCV data. The study highlights a critical methodological tension in financial RL: the provided LOB datasets (often from contests) are typically too narrow in time to train sample-efficient policies with statistical confidence, while useful ensemble methods require long-horizon data to cover multiple market regimes. Furthermore, the paper investigates the phenomenon of "policy collapse," where agents fail to learn or converge to degenerate behaviors, and seeks to distinguish between training failures and economically rational inaction.

Methodology
The authors propose a two-model architecture trained using Proximal Policy Optimization (PPO) within a custom Gymnasium-compatible environment, diverging from the official contest simulator to allow full control over state, action, and reward specifications.

Data Sources:
- Model A (Short-Horizon/LOB): Trained on the official FinRL-AlphaSeek contest dataset (494,749 second-level snapshots of BTC order book depth from April 2021). Features include order imbalance, cancellation pressure, market-order flow, and spread.
- Model B (Long-Horizon/OHLCV): Trained on 2.4 years of hourly BTC/USDT data from Binance (2024–2026). Features include log-returns, volatility, volume, and momentum.
MDP Formulation: The action space is discrete {SELL, HOLD, BUY} with full-position transitions. The reward function is a Sortino-ratio proxy on mark-to-market log-returns, penalized for drawdowns.
Diagnostic Process: The study treats the training process as a diagnostic case study. It documents an initial "total policy collapse" (a single trade followed by 195,000 steps of frozen behavior) and systematically isolates the root causes:
1. Unnormalized Inputs: Raw price levels ($85,000) mixed with small features ( $10^{-3}$ ) created a magnitude ratio of $7.1 \times 10^7$ , destabilizing the network.
2. Numerically Unstable Reward: The Sortino-ratio denominator used a floor ( $10^{-8}$ ) far below empirical volatility, causing reward saturation (clipping) on 88–97% of transitions.
3. Hypothesis Falsification: The authors tested whether "insufficient exploration" caused near-Buy-and-Hold convergence by increasing the entropy coefficient 4x, which failed to improve performance, ruling out exploration deficiency.
Ensemble Mechanism: A proposed filter uses Model A's LOB-derived signals (order imbalance and spread) to gate Model B's trading decisions during the temporal overlap of the two datasets.

Key Results

Long-Horizon Agent (Model B): After correcting input normalization and reward scaling, the agent converged to a policy that modestly but consistently outperformed a Buy-and-Hold baseline on held-out test data (a period with a 21% underlying price decline).
- Return: -19.42% (PPO) vs. -20.82% (Buy-and-Hold).
- Risk Metrics: Higher Sharpe (-1.274 vs. -1.377) and Sortino ratios, and lower maximum drawdown (31.45% vs. 32.54%).
- Behavior: The improvement stemmed entirely from learned entry timing (one realized trade) rather than active trading frequency.
Short-Horizon Agent (Model A): The agent trained on second-level LOB data converged to a policy of never trading (0 trades, 0.00% return).
- Diagnosis: This was not a training failure but an economically rational inference. At one-second granularity, the median price movement is 0.00000%, and even the 99th percentile (0.038%) is roughly 8x smaller than the configured round-trip transaction cost (0.30%). Under these economics, inaction is the optimal policy.
Ensemble Filter Evaluation: On the single window where data sources overlapped (April 2021), the filter intercepted 64 of 165 signals but changed zero realized trades. This occurred because the base policy (Model B) emitted redundant "no-op" actions on that specific window. The authors report this as a null result rather than suppressing it.

Significance and Claims
The paper claims its primary contribution is methodological transparency rather than a state-of-the-art trading performance.

Diagnostic Value: It provides a precise root-cause analysis of PPO policy collapse in financial settings, identifying that unnormalized features and ill-conditioned reward functions can lead to degenerate behavior that mimics "failure" but is actually a response to signal saturation.
Economic Rationality: It demonstrates that an RL agent converging to "no trading" can be a correct inference about market microstructure (specifically, transaction costs dominating price movements at high frequencies), rather than a lack of learning capability.
Negative Result Reporting: The authors argue that reporting null results (e.g., the ensemble filter having no effect on the specific test window) and falsifying their own hypotheses (e.g., sample scarcity vs. transaction costs) is more valuable to the research community than polished but unverifiable success narratives.
Reproducibility: The study emphasizes the release of a full codebase, decision logs, and evaluation harness to support reproducibility, noting that all results are based on a single random seed (42) and should not be considered statistically robust without multi-seed validation.

The paper concludes that while the long-horizon agent showed modest outperformance, the ensemble approach requires further validation across multiple market regimes and seeds before performance claims can be generalized.