💰 quantitative finance

Decision-Induced Ranking Explains Prediction Inflation and Excessive Turnover in SPO-Based Portfolio Optimization

This paper investigates the prediction inflation and excessive turnover issues in SPO-based decision-focused learning for portfolio optimization, offering a KKT-based ranking interpretation and demonstrating that output constraints and turnover controls effectively enhance strategy stability and implementability.

Original authors: Yi Wang, Takashi Hasuike

Published 2026-05-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yi Wang, Takashi Hasuike

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to cook the perfect meal (a portfolio) for your guests. To do this, you need a sous-chef (the prediction model) to tell you which ingredients (stocks) will taste the best tomorrow.

The Old Way vs. The New Way

Traditionally, chefs trained their sous-chefs by asking, "How close was your guess to the actual taste?" If the sous-chef guessed the soup would be salty and it turned out salty, they got a gold star. This is called Predict-then-Optimize (PtO). The problem is, in finance, being "right" about the exact number doesn't always mean you make the best meal. A tiny error in guessing the saltiness might make the sous-chef suggest a completely different recipe.

Enter Decision-Focused Learning (DFL), specifically a method called SPO. Instead of training the sous-chef to guess the exact taste, you train them to guess in a way that leads to the best possible meal. You don't care if they say "salty" or "very salty," as long as they point you to the right ingredients to make a delicious dish.

The Problem: The "Hype" Sous-Chef

The authors of this paper discovered a weird side effect of this new training method. Because the sous-chef is only rewarded for making the best decision, they start acting like a hype man.

Instead of giving realistic estimates (e.g., "This stock might go up 1%"), the SPO-trained model starts screaming, "This stock will go up 500% and that one will crash!"

Why? Because the math behind the decision (the optimizer) is like a strict judge. If the judge sees a tiny difference in the numbers, they might not change the recipe. But if the numbers are huge and dramatic, the judge is forced to make a big, bold change. So, the sous-chef learns to inflate the numbers to force the judge to pick the "winners" clearly.

The Result:

Prediction Inflation: The model predicts wild, unrealistic returns.
Excessive Turnover: Because the predictions are so dramatic, the chef keeps throwing out the old ingredients and buying new ones every single month. This is like constantly changing your entire menu because the sous-chef screamed "New ingredients!" even when the difference was tiny. In the real world, this costs a fortune in transaction fees and is impossible to manage.

The "KKT" Explanation (The Secret Sauce)

The paper uses some fancy math (called KKT conditions) to explain why this happens. They show that the chef isn't actually looking at the raw numbers. Instead, the chef is looking at a ranking list.

Think of it like a race. The chef doesn't care if Runner A is 10 seconds faster than Runner B. They just care that Runner A is ahead of Runner B. The SPO model learns to stretch the gap between the runners so the judge can't possibly make a mistake about who is first. It turns the portfolio problem into a ranking game rather than a "guess the exact score" game.

The Fix: Calming the Sous-Chef Down

The authors tested three simple ways to stop the chaos without firing the sous-chef:

Clipping (The "Censor" Button):
Imagine telling the sous-chef, "No matter how excited you get, you can only say the stock will go up or down by a maximum of 10%." If they try to say 500%, you just clip it to 10%. This stops the extreme outliers but keeps the general order of who is better than whom.
Rescaling (The "Translator"):
This takes the sous-chef's wild list of predictions and squishes them all into a realistic range (like -10% to +10%) while keeping the order the same. If they said Stock A is "way better" than Stock B, it still says that, but now the numbers look normal.
Partial Adjustment (The "Slow Cooker"):
Even with normal numbers, the chef might still want to change the whole menu every day. This strategy says, "Don't change the whole menu. Just move 10% of the ingredients toward the new recipe." It smooths out the changes so you aren't frantically buying and selling every month.

What They Found

Just adding more risk rules didn't work: Making the chef more "cautious" didn't stop the wild predictions.
The "Censor" (Clipping) + "Slow Cooker" (Partial Adjustment) was the winner: This combination kept the turnover (trading activity) low and the portfolio stable, while still making good money.
The "Translator" (Rescaling) + "Slow Cooker" made the most money: This kept the aggressive "ranking" signal strong but smoothed out the trading, leading to higher profits, though with slightly more risk than the "Censor" method.

The Bottom Line

The paper concludes that Smart Predict-then-Optimize is a powerful tool, but it naturally wants to turn predictions into dramatic ranking signals rather than accurate forecasts. To use it in the real world, you can't just let it run wild. You need to put "guardrails" on the predictions (clipping/rescaling) and slow down how much you actually trade (partial adjustment). This turns a chaotic, high-speed trading machine into a stable, investable strategy.

1. Problem Statement

The paper addresses a critical instability in Decision-Focused Learning (DFL) applied to portfolio optimization, specifically within the Smart Predict-then-Optimize (SPO) framework.

The Context: Traditional "Predict-then-Optimize" (PtO) trains models to minimize prediction error (e.g., Mean Squared Error). DFL (and SPO) trains models to minimize "regret" (the loss in downstream decision quality), aligning the predictor directly with the portfolio optimizer.
The Issue: While DFL theoretically improves decision quality, empirical observations suggest it leads to prediction inflation (extremely large, unrealistic return forecasts) and excessive portfolio turnover (frequent, aggressive rebalancing).
The Gap: It is unclear why SPO causes these instabilities and whether standard risk-aversion parameters in the optimizer are sufficient to mitigate them. The paper seeks to explain the mechanism behind these behaviors and propose practical stabilization methods.

2. Methodology

Theoretical Framework: KKT-Based Interpretation

The authors provide a theoretical explanation using Karush–Kuhn–Tucker (KKT) conditions to analyze the SPO mechanism.

Ranking over Magnitude: They demonstrate that the portfolio decision is not driven by the absolute level of predicted returns ( $\hat{r}_i$ $\overset{r}{^}_{i}$ ) but by transaction-cost- and risk-adjusted marginal scores.
- In a return-maximization setting with transaction costs, the active assets satisfy: $\hat{r}_i - \kappa s_i = \nu$ .
- In a Mean-Variance Optimization (MVO) setting, the condition becomes: $\hat{r}_i - 2\lambda(\Sigma w)_i - \kappa s_i = \nu$ .
The Insight: The optimizer effectively performs a ranking of assets based on these adjusted scores. Consequently, the SPO-trained predictor learns to manipulate the margins between these scores to induce a specific ranking, often by inflating the magnitude of predictions to create clearer separation between assets, rather than learning calibrated return estimates.

Empirical Setup

Datasets: Three asset universes: DOW (Dow Jones constituents), ETF_A (broad-market ETFs), and ETF_B (sector/style ETFs).
Timeframe: Backtested from 2020 to 2026 (with 2019 as a warm-up).
Models:
- Predictor: Linear model trained via SPO+ (a differentiable surrogate for the non-differentiable SPO loss).
- Optimizer: Mean-Variance Optimization (MVO) with $L_1$ transaction costs.
- Baselines: Standard MVO (historical returns) and PtO-MVO (predictor trained on MSE).
Interventions Tested: To stabilize the SPO output, the authors tested three mechanisms:
1. Prediction Clipping: Restricting predicted returns to a realistic range $[-\gamma, \gamma]$ .
2. Min-Max Rescaling: Mapping the entire prediction vector to a fixed range (e.g., $[-0.1, 0.1]$ ) while preserving cross-sectional ranking.
3. Partial Portfolio Adjustment: Instead of fully rebalancing to the target portfolio $w^*_t$ , the actual portfolio is updated incrementally: $w_t = w_{t-1} + \delta(w^*_t - w_{t-1})$ , where $\delta \in (0, 1]$ .

3. Key Contributions

Mechanistic Explanation: The paper establishes that SPO-based DFL behaves as a decision-induced ranking mechanism. The predictor inflates return magnitudes to maximize the "gap" in adjusted marginal scores, ensuring the optimizer selects the desired asset ranking, even if the absolute predictions are numerically unrealistic.
Diagnosis of Instability: The authors prove that increasing the risk-aversion parameter ( $\lambda$ ) in the MVO objective is insufficient to solve the turnover problem. The instability stems from the nature of the learned signal (ranking margins), not just the lack of risk penalization.
Practical Stabilization Strategies: The paper evaluates and validates two distinct types of interventions:
- Input-side control: Clipping and Rescaling to manage prediction magnitude.
- Output-side control: Partial adjustment to smooth the realized trading path.
Ablation Analysis: The study isolates the effects of these interventions, showing that while clipping reduces volatility, partial portfolio adjustment is the primary driver for reducing turnover.

4. Results

Prediction Inflation: SPO-trained models produced predicted returns with magnitudes far exceeding realized returns and PtO predictions. This confirmed the hypothesis that SPO prioritizes decision-inducing signals over calibrated forecasts.
Turnover: Standard SPO portfolios exhibited monthly turnover rates exceeding 80–95% across all datasets and risk-aversion levels, rendering them practically unimplementable due to transaction costs.
Effectiveness of Interventions:
- Rescaling Alone: Failed to reduce turnover significantly because it preserved the cross-sectional ranking (the core decision signal).
- Clipping: Reduced volatility and turnover moderately by removing extreme outliers, but did not fully stabilize the portfolio path.
- Partial Adjustment ( $\delta < 1$ ): The most effective mechanism. It drastically reduced turnover (to ~5–10%) and volatility.
- Combinations:
  - Clip + Adj: Produced the most conservative and stable strategy with the lowest turnover and volatility.
  - Rescale + Adj: Achieved the highest Sharpe ratios and returns, suggesting that preserving the ranking signal while smoothing the execution path yields a superior risk-adjusted performance.

5. Significance and Implications

Theoretical Shift: The paper reframes the understanding of DFL in finance. It suggests that DFL does not necessarily learn "better" return forecasts in the traditional statistical sense; rather, it learns ranking signals optimized for the specific constraints of the downstream optimizer.
Practical Implementation: It provides a roadmap for deploying DFL in real-world trading. Pure SPO is often too volatile for live trading. The study demonstrates that post-processing constraints (clipping/rescaling) and execution smoothing (partial adjustment) are essential to make SPO-based strategies implementable.
Trade-off Identification: The results highlight a trade-off: strategies that strictly control turnover (Clip+Adj) are safer but may sacrifice some return potential, while strategies that preserve the ranking signal (Rescale+Adj) offer higher returns but require careful turnover management.
Future Directions: The authors suggest that future DFL research should incorporate financial constraints (liquidity, turnover limits) directly into the learning objective or optimization layer, rather than treating them as post-hoc heuristics.

In summary, this paper bridges the gap between the theoretical appeal of Decision-Focused Learning and the practical realities of portfolio management, offering a mechanistic explanation for SPO's instability and a validated toolkit for stabilizing it.