Technical Summary: PORTBENCH

Problem Statement

While Large Language Models (LLMs) have demonstrated proficiency in diverse financial tasks such as knowledge retrieval and numerical reasoning, Portfolio Management (PM) remains inadequately benchmarked. Existing financial benchmarks suffer from two critical gaps:

1. Neglect of Cross-Asset Correlation: Most benchmarks restrict coverage to single asset classes or evaluate assets in isolation. This design fails to distinguish between genuinely diversified portfolios and highly concentrated ones, ignoring the correlation structures essential for risk management (Markowitz, 1952; Qian et al., 2005).
2. Incomplete Decision Pipelines: Current evaluations rely on static, single-step predictions or partial multi-step assessments. They fail to capture the full sequential workflow of PM—spanning market interpretation, signal generation, weight optimization, execution, and risk monitoring—where errors in early stages can cascade into catastrophic downstream failures. Furthermore, existing benchmarks typically evaluate under normal market conditions with a single implicit risk profile, leaving the resilience of strategies under stress and their alignment with specific investor tolerances untested.

Methodology

The authors introduce PORTBENCH, a correlation-aware, full-pipeline benchmark designed to evaluate LLMs across six heterogeneous asset classes (equities, commodities, bonds, cryptocurrency, real estate, and cash equivalents) over a ten-year period (2015–2025). The framework consists of four core modules:

1. Market Base Dataset

The dataset comprises 183 distinct financial instruments, including daily price histories, return series, and associated news text. It includes macroeconomic indicators (interest rates, inflation, credit spreads, volatility indices) and is structured to highlight the disparity between low inter-class correlations and strong intra-class correlations.

2. Dual-Layer Evaluation Framework

PORTBENCH evaluates LLMs through two complementary layers:

• Static QA Layer: A dataset of 6,269 correlation-based questions across seven task templates (T1–T7) ranging from single-asset prediction to regime-driven rebalancing. Questions and ground truths are generated automatically from historical data using analytical formulas.
• Dynamic Five-Stage Pipeline: A sandbox that mirrors the full PM decision cycle at each rebalance date:
  • S1 (Market Interpretation): Assigns sentiment scores and identifies market regimes.
  • S2 (Signal Generation): Maps sentiment to directional trading signals (buy/sell/hold).
  • S3 (Weight Optimization): Proposes portfolio weights subject to constraints.
  • S4 (Execution Simulation): A deterministic pass-through applying transaction costs and measuring turnover deviation from the oracle.
  • S5 (Risk Monitoring): Computes Value-at-Risk (VaR), drawdown, and weight drift, triggering rebalancing if thresholds are breached.

3. Novel Metrics

To address the limitations of existing scoring methods, PORTBENCH introduces:

• Two-Layer Correlation Scoring (for S3): Decomposes the weight optimization score into accuracy (proximity to the optimal Sharpe ratio) and correlation structure. The correlation component penalizes intra-class concentration and rewards inter-class hedging, ensuring that portfolios are evaluated on their diversification quality, not just return optimization.
• CEPS (Cross-stage Error Propagation Score): A metric that quantifies how reasoning errors compound across the pipeline. Unlike a simple average, CEPS penalizes models where performance drops significantly between consecutive stages (e.g., strong signals leading to poor execution), distinguishing cascading failures from uniformly mediocre performance.

4. Stress and Investor Profile Evaluation

Models are evaluated under three historical stress regimes (2015 China Shock, 2020 COVID Crash, 2022 Crypto Collapse) and three investor profiles (Conservative, Balanced, Aggressive). Performance is measured by whether models pass a "stress gate" (maintaining drawdowns within profile-specific tolerances) and by their ability to adapt allocations to different risk constraints (Profile Alignment Score).

Key Results

The authors evaluated ten frontier LLMs (including DeepSeek-V4, Qwen3, GLM-5, and Kimi-K2) against five classical baselines (Equal-Weight, 60/40, Risk Parity, Covariance Risk Parity, Minimum Variance).

• Knowledge-Competence Gap: Despite strong performance on static QA tasks (particularly formula-driven tasks where covariance matrices are provided), 90% of model-profile combinations failed to outperform a basic equal-weight (1/N) baseline in terms of risk-adjusted returns.
• Execution Collapse: A critical failure mode identified is "universal under-trading." Models often generate strong signals (S2) but fail to translate them into meaningful portfolio rebalancing (S4). This is attributed to models treating the covariance matrix as noise or failing to parse it, resulting in near-uniform weights that require minimal turnover, thereby collapsing execution scores.
• Stress Fragility: High performance in normal markets does not predict stress resilience. Six of ten models failed the stress gate under the conservative profile during the 2022 Crypto Collapse, despite satisfying all procedural constraints. These models suffered catastrophic drawdowns because their allocations did not effectively hedge against correlated asset crashes.
• Rank Dissociation: There is a weak negative correlation ($\rho = -0.32$) between static QA accuracy and dynamic pipeline performance (CEPS). Models that excel at isolated factual recall often fail to sustain reasoning across the causal chain of portfolio management.
• Value of Adaptation: The primary value of LLMs identified is not raw return generation, but constraint adaptation. Models that successfully adjusted their allocations to specific investor risk profiles (e.g., tightening equity caps for conservative investors) demonstrated a capability static baselines lack, even if their absolute returns did not always surpass simple heuristics.

Significance and Claims

The paper claims that PORTBENCH provides the first comprehensive, correlation-aware benchmark for LLM-driven portfolio management. Its significance lies in:

1. Exposing the Limits of Current LLMs: It demonstrates that strong static financial knowledge does not translate into robust, real-world portfolio management capabilities, particularly regarding the utilization of covariance structures and execution of rebalancing.
2. Redefining Evaluation Standards: By introducing CEPS and correlation-aware scoring, the benchmark shifts the focus from terminal outcomes to the quality of the decision process and the propagation of errors.
3. Highlighting the "Compliance Trap": The study reveals that models can satisfy all procedural constraints (e.g., asset caps) while still violating outcome safety (e.g., excessive drawdowns) under stress, a failure mode previously unmeasured in standard benchmarks.

The authors conclude that the value of LLMs in portfolio management currently lies in constraint adaptation and tail-risk awareness within a single decision framework, rather than in generating superior returns over naive diversification strategies. They caution against deploying LLM-generated allocations in live trading without rigorous human oversight, as even frontier models exhibit fragile behavior under stress conditions.