🤖 machine learning

Plan Before You Trade: Inference-Time Optimization for RL Trading Agents

The paper introduces FPILOT, a plugin inference-time optimization framework that enhances pre-trained reinforcement learning trading agents by dynamically optimizing portfolio allocations at each decision step using predicted price trajectories, thereby consistently improving returns and risk-adjusted metrics without requiring model retraining.

Original authors: Eun Go, Rohan Deb, Arindam Banerjee

Published 2026-05-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Eun Go, Rohan Deb, Arindam Banerjee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a professional stock trader who has spent years training a robot to manage a portfolio. This robot, trained using Reinforcement Learning (RL), is very good at looking at the current market and deciding what to buy or sell right now. It's like a chess player who has memorized millions of games and knows the best move for any position on the board.

However, there's a catch: once the robot is trained, it's "frozen." It makes decisions based only on what it sees today. It doesn't have a way to use a weather forecast for tomorrow's market, even if a separate expert (a "forecaster") has one.

The paper introduces FinPILOT, a clever "plugin" that lets this frozen robot think ahead before it makes a move. Here is how it works, using simple analogies:

1. The Problem: The "Static" Robot

Think of the trained robot as a driver who only looks at the road directly in front of the car. It reacts to potholes and traffic lights as they appear, but it doesn't look at the GPS map to see a traffic jam 10 miles ahead. In the financial world, this means the robot misses out on opportunities to adjust its strategy based on predicted future price movements.

2. The Solution: The "Imaginary Road Trip" (FinPILOT)

FinPILOT acts like a co-pilot who gives the driver a quick "what-if" simulation before every turn.

The Forecaster: A separate tool (in this case, an XGBoost model) predicts what the stock prices will look like for the next 50 days.
The Simulation: Before the robot executes a trade in the real world, FinPILOT asks: "If we follow our current plan, but the prices change exactly as the forecaster predicts, how much money would we make?"
The Adjustment: The robot then quickly tweaks its decision-making "brain" (its policy) to maximize that imaginary profit. It does this on the fly, without needing to retrain the whole robot from scratch.
The Execution: It takes that single, improved decision, executes it in the real market, and then repeats the process for the next day.

3. The Key Insight: "The Market Doesn't Care About You"

The paper points out a unique feature of the stock market that makes this easy: One small trader's actions don't change the price.

In video games or robotics (where other AI agents might be), if you move, the world changes.
In the stock market, if you buy a tiny bit of Apple stock, the price of Apple doesn't change because of you.
Why this matters: Because the robot's actions don't change the future prices, the "forecaster" can just predict the future prices once, and the robot can imagine all its possible moves against that fixed future. It's like planning a hike on a map that doesn't change, rather than a map that shifts every time you take a step.

4. The "Cheating" Experiments

To prove their system works, the researchers did something they call "cheating."

They created fake forecasts that were perfectly accurate (knowing the future exactly).
They found that even a tiny bit of "future knowledge" (a very weak forecast) made the robot significantly smarter.
The Threshold Effect: They discovered a "tipping point." Once the forecast is slightly better than random guessing (even just 1% accurate), the robot's performance jumps up. Making the forecast even more accurate helps, but the biggest leap happens just by crossing that tiny threshold from "useless" to "slightly useful."

5. The Results: Better Returns, Less Risk

When they tested this on real stock data (the Dow Jones 30) and currency data:

Better Profits: The robots using FinPILOT made more money than the standard robots.
Smarter Risk Management: They added a "safety brake" to the imaginary simulation. If a potential future looked risky (like a high chance of a big loss), the robot adjusted its plan to avoid it.
Works with Any Robot: It didn't matter which specific learning algorithm they used (like PPO, SAC, etc.); the plugin worked for all of them.
Stochastic vs. Deterministic: Robots that make "randomized" decisions (stochastic) benefited more than robots that make "fixed" decisions (deterministic). It's like a driver who is willing to try different routes benefiting more from a GPS than a driver who stubbornly sticks to one path.

Summary

FinPILOT is a tool that lets a trained trading AI "dream" about the future before it acts. By using a simple price forecast to imagine a few days ahead, the AI can tweak its strategy instantly to make more money and avoid some risks, all without needing to be retrained. It turns a reactive robot into a proactive planner.

Technical Summary: FinPILOT – Inference-Time Optimization for RL Trading Agents

Problem Statement

Reinforcement learning for financial trading (RLFT) has established itself as a framework for sequential portfolio decision-making. However, current approaches suffer from a disconnect between forecasting and decision-making. Existing methods typically fall into two independent threads:

Supervised Forecasting: Models predict short-horizon returns but offload the decision problem to external heuristics.
Reinforcement Learning: Agents are trained as static policies. While they may ingest price predictions as state features during training, they do not utilize forecasts as a planning signal to adapt behavior at inference time.

Furthermore, while Model Predictive Control (MPC) and world-model-based RL (e.g., Dreamer, MuZero) exist, they generally assume high-fidelity dynamics suitable for robotics or require co-designing the policy with the world model. They do not offer a mechanism to improve an arbitrary, pre-trained RL agent at deployment using a separately trained forecaster without retraining the agent.

The core challenge addressed is: How can inference-time policy optimization be driven by a separately trained price forecaster without retraining the RL agent?

Methodology: FinPILOT

The authors propose FinPILOT (Financial Plugin Inference-time Learning for Optimal Trading), a plugin framework inspired by Model Predictive Control (MPC). The key structural insight is that in financial markets, future prices are largely independent of a single agent's portfolio allocation. This allows the use of a predictive model to generate multi-step price trajectories without the need for iterative, action-conditioned rollouts typical in standard RL.

Core Components

Pre-trained Agent: A standard model-free RL agent (e.g., PPO, SAC) trained on historical data with no access to future price forecasts.
Exogenous Forecaster ( $F$ ): A separate model (e.g., XGBoost) trained on historical data strictly preceding the test window. It generates an $H$ -step price trajectory.
Inference-Time MPC Loop: At each decision step $t$ $t$ :
- The forecaster predicts a trajectory of future states $\{\hat{s}_{t+1}, \dots, \hat{s}_{t+H}\}$ .
- The pre-trained actor $\pi_\theta$ is adapted by optimizing an imagined objective $J_t$ over this predicted trajectory.
- Only the first resulting action is executed in the real environment; the policy parameters are then reset or updated for the next step.

The Imagined Objective

The framework defines a surrogate return over the planning horizon $H$ . To handle the uncertainty of forecasts and improve risk-adjusted performance, the objective includes a downside-risk penalty:
$J_t(\theta) = \bar{J}_t - \lambda \sqrt{D_t + \epsilon}$
Where:

$\bar{J}_t$ is the mean return across $K$ particles (Monte Carlo samples).
$D_t$ is the downside variance (semi-deviation) of per-particle returns.
$\lambda$ is a risk-aversion coefficient.
Forecast Noise Injection: To prevent overfitting to a single point estimate, Gaussian noise is added to the predicted trajectories. The noise scale is calibrated based on the variance of the forecaster's predictions, not ground truth, ensuring the method remains forecaster-agnostic.

Two-Phase Rollout

The method exploits the fact that price transitions are independent of the agent's actions to split the rollout:

Phase 1 (No Gradients): Precompute the deterministic forecast trajectory and generate $K$ noisy particles. Evaluate the terminal value estimate $V_\theta$ for each.
Phase 2 (Gradient Ascent): Iterate over the cached particles to compute the imagined reward using the closed-form portfolio return equation. Gradients are tracked only through the actor parameters $\theta$ to update the policy.

Key Contributions

FinPILOT Framework: A plugin inference-time MPC framework compatible with any pre-trained RL agent, featuring a downside-risk-penalized variant. It requires no modifications to the training procedure.
Empirical Validation: Demonstrated consistent improvements in total return and risk-adjusted metrics (Sharpe, Sortino, Calmar) across five RL algorithms (PPO, SAC, A2C, TD3, DDPG) on the TradeMaster DJ30 benchmark using a practical XGBoost forecaster with modest predictive power ( $R^2 \approx 0.01$ ).
Generalization: The framework transfers to the Foreign Exchange (FX) dataset without hyperparameter retuning, suggesting robustness across asset classes.
Cheating Experiments Methodology: A diagnostic approach that constructs synthetic forecasts at calibrated $R^2$ levels to measure the scaling of policy gains against forecast quality.

Experimental Results

Evaluated on the TradeMaster DJ30 benchmark (29 stocks, 2012–2021) and FX dataset:

Performance Gains: With a practical XGBoost forecaster ( $R^2 \approx 0.01$ $R^{2} \approx 0.01$ ), FinPILOT produced consistent gains.
- PPO: Total return increased from 17.74% to 23.46%.
- SAC: Total return increased from 17.78% to 22.21%.
- Risk-Adjusted Metrics: Significant improvements in Sharpe, Sortino, and Calmar ratios were observed, particularly for stochastic policies (PPO, SAC).
Stochastic vs. Deterministic: Stochastic policies (PPO, SAC) benefited more significantly from inference-time optimization than deterministic ones (TD3, DDPG).
Forecast Quality Scaling: "Cheating experiments" revealed a threshold-like behavior. Performance gains were nearly indistinguishable between $R^2 = 0.001$ and $R^2 = 0.01$ , but continued to improve as $R^2$ increased toward 0.8. This suggests that even weak, economically meaningful signals are sufficient to drive the framework's benefits.
Horizon Sensitivity: A planning horizon of $H=50$ days was found to be optimal, balancing the need for structural market regime capture against high-frequency noise.

Significance and Claims

The paper claims that FinPILOT bridges the gap between forecasting and decision-making in RLFT without the computational cost of retraining agents. By treating the price forecaster as a surrogate world model, the framework allows pre-trained agents to "plan" against imagined futures at inference time.

The authors emphasize that the structural property of financial markets (closed-form rewards and price-taker assumptions) simplifies world modeling to pure price forecasting. The framework is presented as a plugin solution: it does not alter the training of the RL agent, making it applicable to a wide range of existing algorithms.

The results suggest that the performance of RL trading agents is not limited by the need for high-fidelity world models (which are difficult to learn in finance) but rather by the ability to leverage even modest predictive signals ( $R^2 \approx 0.01$ ) through inference-time adaptation. The authors conclude that as financial forecasting models improve, the performance of agents using FinPILOT should correspondingly improve.

Limitations: The authors note that while risk-adjusted metrics improved, maximum drawdown increased slightly for some stochastic policies (PPO, SAC) relative to baselines, indicating a trade-off between total return and downside exposure that remains a direction for future work. The evaluation is bounded to two benchmarks (DJ30 and FX) and relies on backtesting, which does not account for market impact or latency in live trading.