Bridging Behavioral Finance and Robust Agentic DRL: A novel computational framework for Portfolio Selection

Technical Summary: Bridging Behavioral Finance and Robust Agentic DRL for Portfolio Selection

Problem Statement
Financial markets are characterized by dynamic volatility, macroeconomic shocks, and significant behavioral responses that conventional portfolio optimization models often fail to capture. Traditional approaches, including standard Reinforcement Learning (RL) algorithms, typically rely on return-based reward structures. These structures frequently overlook critical investor characteristics such as loss aversion, asymmetric reactions to gains and losses, and the inherent uncertainty of market states. Consequently, standard RL agents may struggle to generalize under volatile conditions, leading to suboptimal timing, excessive risk-taking, or premature exits during market turbulence. There is a distinct gap in existing literature regarding the systematic integration of perturbation-based robustness and Prospect Theory-inspired behavioral modeling within a unified RL architecture for portfolio management.

Methodology
The study proposes a novel computational framework, RPPO-MPT, which extends the Proximal Policy Optimization (PPO) algorithm through two primary mechanisms: perturbation-based robust learning and behavioral reward shaping. The portfolio selection problem is formulated as a Markov Decision Process (MDP) involving 15 large-cap U.S. stocks.

The methodology progresses through three comparative stages:

1. Baseline PPO: A standard PPO agent is initialized with uniform weights and pre-trained using a Mean-Variance Portfolio (MPT) strategy via supervised learning. The agent learns to maximize cumulative returns penalized by transaction costs ($\gamma = 0.01$). The state space includes historical returns over a 14-week window and previous portfolio weights.
2. Robust PPO (RPPO): This variant introduces a perturbation-based training mechanism to enhance stability against market noise. For each state-action pair, the agent generates $K=5$ perturbed states using Gaussian noise ($\sigma = 0.03$). A robust reward is computed by penalizing the variance in returns across these perturbed states, controlled by a trade-off parameter $\beta = 0.2$. This forces the policy to be less sensitive to small state variations.
3. RPPO-MPT (Proposed Model): This model integrates the robustness of RPPO with a reward structure inspired by Prospect Theory. Instead of optimizing raw returns, the agent optimizes a "hope-fear" utility function.
   • Reference Dependence: Returns are evaluated relative to a reference point (set to zero).
   • Loss Aversion: The reward function decomposes outcomes into "hope" (gains) and "fear" (losses). Losses are weighted by a loss-aversion coefficient ($\lambda = 2.25$), while gains and losses are subject to diminishing sensitivity ($\alpha = \beta = 0.88$).
   • Reward Shaping: The final reward is a ratio of the hope component to the fear component (plus a small constant $\epsilon_1$ to ensure numerical stability), minus transaction costs. This structure explicitly penalizes downside risk more heavily than it rewards equivalent gains, aligning the agent's objective with behavioral investor preferences.

Experimental Setup
The models were trained on historical adjusted price data for 15 S&P 500 stocks from January 2010 to mid-June 2022. Out-of-sample testing was conducted from mid-June 2022 to September 2025. Performance was evaluated using a comprehensive suite of metrics, including annualized return, Sharpe ratio, Information ratio, $\alpha$, maximum drawdown, longest drawdown, annual turnover, and behavioral utility under varying risk-aversion settings.

Key Results
The comparative analysis demonstrates that the proposed RPPO-MPT framework achieves the strongest out-of-sample performance profile among the three models:

• Superior Risk-Adjusted Returns: RPPO-MPT outperformed both the baseline PPO and the Robust PPO (RPPO) in terms of Sharpe ratio, Information ratio, and $\alpha$.
• Behavioral Alignment: The model achieved higher behavioral utility, indicating better alignment with investor preferences regarding loss aversion and asymmetric risk perception.
• Drawdown Control: While maintaining competitive drawdown control, the RPPO-MPT model effectively managed downside risk without sacrificing return generation.
• Stability: The integration of perturbation-based robustness allowed the agent to maintain stability during the volatile testing period (2022–2025), avoiding the over-concentration and erratic rebalancing sometimes observed in the baseline PPO.

Significance and Claims
The paper claims that the primary contribution of this work is the successful bridging of robust optimization and behavioral finance within a single Deep Reinforcement Learning architecture. By jointly optimizing for financial performance, resilience to state uncertainty, and investor-consistent utility, the RPPO-MPT framework addresses the limitations of traditional RL models that rely solely on return maximization.

The authors assert that this approach offers a practical path for deploying "resilient, psychologically grounded decision-making agents" in real financial systems. The study demonstrates that explicitly accounting for market uncertainty (via perturbations) and investor psychology (via Prospect Theory reward shaping) enhances the adaptability and investor alignment of RL agents in volatile markets, providing a more robust alternative to conventional portfolio optimization methods.