Constructing a Portfolio Optimization Benchmark Framework for Evaluating Large Language Models

This paper introduces a novel benchmark framework using mathematically explicit portfolio optimization problems to evaluate the quantitative reasoning capabilities of large language models, revealing distinct performance patterns among GPT-4, Gemini 1.5 Pro, and Llama 3.1-70B in handling risk-based objectives, return-based tasks, and investment constraints.

The Gist

Imagine you are hiring a new financial advisor. You have three candidates: GPT, Gemini, and Llama. You want to know who is actually good at managing money, not just who is good at writing essays about money.

Most previous tests for these AI "brains" were like asking them to read a news article and summarize it. They passed those tests easily. But in the real world, investing isn't just about reading; it's about doing complex math to figure out the perfect mix of stocks to get the most money with the least risk.

This paper introduces a new, tougher test called PortBench. Think of it as a "driving test" for AI, but instead of steering a car, they have to steer a portfolio of investments.

The Test: A Math Puzzle, Not a Reading Quiz

The researchers built a massive question bank with 9,500 puzzles. Each puzzle looks like a multiple-choice question:

• The Scenario: "Here are 5 stocks. You want to minimize risk (or maximize profit). Here are the rules (e.g., you can't put more than 20% in one stock)."
• The Options: The AI must pick the one mathematically perfect answer from four choices.
• The Trap: Three of the choices are "distractors." They look plausible but are mathematically wrong. Some are slightly off, while others are wildly incorrect.

The beauty of this test is that there is one single, mathematically correct answer. There's no guessing or "maybe." It's like a math problem where you can check the answer key with a ruler.

The Results: Who Passed the Driving Test?

The researchers ran the test on three popular AI models. Here is how they performed, using some everyday analogies:

1. GPT-4 (The Cautious Risk Manager)

• Performance: The clear winner, especially when the goal was to avoid losing money (minimizing risk).
• Analogy: Imagine GPT is a prudent captain steering a ship through a storm. When the goal is "don't crash," GPT is incredibly steady. It understands the math of safety very well and doesn't get confused by the rules (constraints).
• Weakness: It struggled a bit with the most complex puzzles that required balancing both high speed (return) and safety (risk) at the same time, like the "Sharpe Ratio."

2. Gemini 1.5 Pro (The High-Roller Gambler)

• Performance: Did very well when the goal was simply to make the most money (maximizing return).
• Analogy: Think of Gemini as a race car driver who loves speed. If the track is clear and the goal is "go fast," it wins. But as soon as you add obstacles (constraints) or ask it to drive carefully, it starts to panic. It often ignores the rules to chase a higher number, leading to mistakes.
• Weakness: It fell apart when the rules got complicated or when the "wrong" answers looked very similar to the "right" ones.

3. Llama 3.1 (The Student Who Needs More Study)

• Performance: The lowest scores overall.
• Analogy: Llama is like a student who memorized the textbook but hasn't learned how to apply the math to real life. It struggled with almost everything, especially when the rules got strict. It often picked the wrong answer even when the math was simple.

The Big Takeaway

The study reveals a surprising truth about current AI:

• They are great at reading and talking about finance.
• They are still learning how to do finance.

When the task was simple (like "pick the safest option"), GPT was reliable. But when the task required juggling multiple goals (high return + low risk + strict rules), all three models started to make mistakes.

Why Does This Matter?

If you are building an app that uses AI to manage your retirement fund, this paper says: "Be careful."

• You can trust these AIs to help you analyze news or explain concepts.
• You cannot yet trust them to automatically pick your stocks without a human expert double-checking the math, especially for complex strategies.

The researchers hope this new "driving test" will help developers build better, safer AI tools for the future, ensuring that when we let robots manage our money, they don't crash the car.

Technical Summary

Here is a detailed technical summary of the paper "Constructing a Portfolio Optimization Benchmark Framework for Evaluating Large Language Models."

1. Problem Statement

While Large Language Models (LLMs) have demonstrated proficiency in natural language processing (NLP) tasks, their ability to perform quantitative reasoning and structured decision-making in finance remains under-evaluated. Existing financial benchmarks (e.g., FinQA, FinBEN) primarily focus on language-oriented tasks such as document summarization, information extraction, and question answering based on financial texts. These benchmarks fail to assess the core competencies required for real-world asset management: mathematical optimization, risk-return trade-off analysis, and portfolio construction under specific constraints.

The authors identify a critical gap: there is no standardized framework to test whether LLMs can solve mathematically defined portfolio optimization problems where a single, objectively correct solution exists, distinct from subjective text generation.

2. Methodology

The study proposes PortBench, a novel benchmark framework grounded in Modern Portfolio Theory (MPT). The methodology involves three core components:

A. Benchmark Construction

• Dataset Scale: The framework automatically generates 9,500 multiple-choice questions.
• Problem Formulation: Each question is a portfolio optimization problem derived from the mean-variance framework. The problem is defined by:
  • Assets: A specific list of investment assets (e.g., ETFs like BND, VTI).
  • Objective: One of five optimization goals:
    1. Minimizing Volatility (Variance)
    2. Maximizing Expected Return
    3. Maximizing Sharpe Ratio
    4. Minimizing Maximum Drawdown (MDD)
    5. Minimizing Conditional Value-at-Risk (CVaR)
  • Constraints: Various conditions including no constraints, lower/upper bounds on asset weights (e.g., 0%–20% minimum, 60%–90% maximum), and cardinality constraints (forcing a specific number of assets).
• Ground Truth: The "correct" answer is the mathematically optimal portfolio weight vector calculated using a convex optimization solver.

B. Distractor Generation

To ensure rigorous evaluation, the framework generates three incorrect options (distractors) for each question using four distinct methods to vary difficulty:

1. Distance-based: Selecting portfolios with specific Euclidean distances from the optimal solution.
2. Threshold-based: Selecting portfolios where the deviation in the objective function value (e.g., variance) falls within a specific range.
3. Quantile-based: Selecting based on statistical distribution of performance.
4. Dual-Criteria: Combining structural deviation (weight vector distance) and performance deviation (objective value) to create complex, non-trivial distractors.

C. Evaluation Protocol

The framework tests three state-of-the-art LLMs:

• GPT-4 (specifically GPT-4o)
• Gemini 1.5 Pro
• Llama 3.1-70B

The models are prompted to select the optimal portfolio from the four options based on the natural language description of the objective and constraints.

3. Key Contributions

1. First Mathematically Explicit Financial Benchmark: Introduces the first benchmark specifically designed to evaluate LLMs on optimization-based financial decision-making rather than just text comprehension. It ensures every question has a unique, mathematically verifiable correct answer.
2. Scalable and Reproducible Framework: The system dynamically generates problems by varying objectives, assets, constraints, and time horizons, allowing for fine-grained analysis of model performance across thousands of scenarios.
3. Comprehensive Performance Analysis: Provides a granular breakdown of model capabilities across different risk profiles, return objectives, and constraint complexities, offering actionable insights for developing AI-driven financial advisory tools.

4. Experimental Results

The evaluation of the three models revealed distinct performance patterns:

• GPT-4 (GPT):

  • Performance: Achieved the highest overall accuracy, particularly excelling in risk-based objectives (minimizing variance and MDD).
  • Robustness: Demonstrated stable performance across varying constraint types. It showed a strong ability to interpret and apply mathematically defined objective functions.
  • Weakness: Struggled significantly with the Sharpe Ratio maximization (accuracy <10%), indicating difficulty with multi-criteria optimization involving complex trade-offs.

• Gemini 1.5 Pro:

  • Performance: Performed well on return-based objectives but showed lower accuracy than GPT in risk-based tasks.
  • Limitations: Performance degraded sharply as problem complexity increased or when constraints were introduced. It tended to favor portfolios with higher raw returns even when they violated the theoretical optimality under constraints, suggesting a reliance on direct return comparison rather than structural constraint interpretation.

• Llama 3.1-70B:

  • Performance: Recorded the lowest overall accuracy across all categories.
  • Limitations: Struggled significantly with complex objectives (Sharpe, CVaR) and restrictive constraints, often failing to differentiate between structurally similar distractors.

• General Trends:

  • All models performed reasonably well on simple, unconstrained problems.
  • Accuracy dropped significantly for all models when constraints (bounds, cardinality) were added.
  • Multi-objective problems (like Sharpe Ratio) proved extremely difficult for all current LLMs, with accuracy rates falling below 10% across the board.

5. Significance and Implications

• Current Limitations: The study highlights that while LLMs possess strong linguistic capabilities, their quantitative reasoning and optimization logic are currently insufficient for autonomous financial decision-making, especially in complex, constrained environments.
• Practical Application: GPT-4 shows potential as a decision-support tool for risk-focused strategies, but none of the models are reliable enough for fully autonomous portfolio management without expert oversight.
• Future Directions: The framework suggests that future LLM development must focus on integrating numerical analysis with linguistic interpretation and improving multi-objective optimization reasoning. The benchmark serves as a scalable foundation for tracking progress in this domain and developing more robust AI financial agents.

In conclusion, this paper establishes a rigorous, mathematically grounded standard for evaluating LLMs in finance, moving beyond text generation to test the core logical and quantitative skills required for investment management.