Technical Summary: MARKET-BENCH

Problem Statement

The field of quantitative trading operates in a high-stakes environment where model hallucinations or incomplete inputs can result in significant financial loss. While existing evaluations of Large Language Models (LLMs) in finance focus on high-level tasks—such as summarizing earnings calls, sentiment analysis, or fundamental modeling—there is a critical gap in assessing models on their ability to assist traders with day-to-day operational tasks. Specifically, current benchmarks do not evaluate whether an LLM can understand and apply fundamental trading concepts (e.g., order execution, inventory tracking, risk management) to construct a functional, executable backtester from a natural language description. The core problem addressed is the inability of current LLMs to reliably translate strategy descriptions into code that produces numerically accurate and structurally sound trading simulations.

Methodology

The authors introduce MARKET-BENCH, a benchmark designed to evaluate LLMs on introductory quantitative trading tasks. The evaluation framework requires models to generate executable Python code that simulates trading strategies based on provided natural language descriptions and market data.

Benchmark Design

The benchmark consists of three canonical strategies, each capturing distinct aspects of market dynamics:

1. Scheduled Trading (Strategy 1): A single-asset strategy on Microsoft (MSFT) involving market orders at pre-specified timestamps. It tests order-book interaction, position tracking, FIFO accounting for realized P&L, and unrealized P&L based on mid-prices.
2. Pairs Trading (Strategy 2): A mean-reversion strategy between Coca-Cola (KO) and Pepsi (PEP). It requires calculating spread z-scores, managing joint capital across two symbols, and executing entry/exit logic based on threshold crossings.
3. Delta Hedging (Strategy 3): An options strategy on MSFT where the model must hedge a predefined options delta time series using the underlying stock. This tests the handling of fill-or-kill orders, exchange delays, and the aggregation of net delta.

Data and Simulation

• Data Sources: Market data is derived from Databento (L10 order book data) or synthetically generated (random walk for options delta).
• Synthetic Book: A critical component of the benchmark is the requirement for models to track liquidity removed from the book. Models must maintain a "synthetic book" that nets raw order book data against consumed liquidity to ensure accurate price and volume tracking.
• Evaluation Protocol:
  • Each strategy is tested across 5 distinct rounds (different parameter configurations and data seeds), totaling 15 combinations per model.
  • Models are allowed up to 3 independent, one-shot attempts per round.
  • Temperature: Set to 0.0 to ensure determinism.
  • Constraints: A 10-minute time limit and a restricted set of allowed Python packages (e.g., pandas, numpy, datetime).

Metrics

The evaluation separates structural reliability from numerical accuracy:

• Executable Passes: Counts the number of rounds where the model successfully generates code that runs without syntax errors or runtime assertions.
• Mean Absolute Error (MAE): Measures the deviation between the model's output metrics (total P&L, max drawdown, position paths) and a verifiable reference implementation.
  • Failure Handling: If a model fails to execute, it is assigned a "duplicated-metrics baseline MAE," calculated by repeating the initial state metrics for all timestamps. This penalizes non-executable outputs heavily.
• Aggregation: Models are ranked by Mean MAE (averaged across all rounds, including failures), Best Run MAE (lowest error achieved in any single round), and total Executable Passes.

Key Results

The study evaluated 13 state-of-the-art models, including GPT-5 variants, Claude 4.5, Gemini 3 Pro, and open-source models like Qwen3 and Llama 4.

Performance Trends

1. Reliability vs. Accuracy: There is a significant divergence between a model's ability to produce running code and its ability to produce correct code.
   • GPT-5.2 demonstrated strong overall performance, achieving perfect executability (15/15 rounds) with a low overall Mean MAE (969.39).
   • Qwen3 Max achieved perfect executability (15/15) but suffered from catastrophic numerical divergence, particularly in Strategy 2, resulting in an overall Mean MAE of ~1.59 × 10⁸.
   • GPT-5.1 Codex-Max achieved the lowest single-run error (0.002 MAE) on Strategy 1 but showed high variance across rounds.
2. Strategy Difficulty:
   • Strategy 1 (Single Stock): The easiest task. Most models (average 4.08/5 rounds) could execute it, though numerical errors varied by orders of magnitude.
   • Strategy 2 (Pairs): More structurally challenging. Average Executable Passes dropped to 3.46. Models struggled with z-score computation and multi-asset capital management.
   • Strategy 3 (Delta Hedging): The most complex. Average Executable Passes fell to 3.38. Errors in delta aggregation and timing led to massive P&L discrepancies.
3. Failure Modes:
   • Structural Failures: Syntax errors, missing columns, or type mismatches preventing execution.
   • Semantic Failures: Code runs but implements logic incorrectly (e.g., wrong hedge ratios, incorrect FIFO accounting, ignoring capital constraints). These result in high MAE despite successful execution.

Key Contributions

• MARKET-BENCH Benchmark: A novel evaluation suite specifically targeting the intersection of code generation and quantitative finance, moving beyond text-based financial analysis to executable strategy implementation.
• Multi-Round, Multi-Metric Evaluation: A rigorous protocol that distinguishes between code that runs and code that works, using a baseline penalty for non-executable outputs to reflect the severity of failure in trading contexts.
• Synthetic Book Simulation: The introduction of a mechanism requiring models to track liquidity consumption and maintain synthetic order books, a critical real-world constraint often omitted in code-generation benchmarks.

Significance and Claims

The paper concludes that while current LLMs can scaffold basic trading infrastructure, they struggle to reason robustly about prices, inventory, and risk.

• Current Limitations: The authors assert that using model output as a direct replacement for quantitative analysts is "extremely risky." Even for introductory tasks, models fail to consistently implement basic strategies with numerical fidelity.
• Role of LLMs: The paper suggests that current models are better suited as "coding supplements" rather than autonomous agents for synthesizing trading ideas.
• Future Outlook: The authors posit that with better training on trading data and improved reasoning capabilities, models may eventually bridge the gap between describing strategies and implementing them with deep understanding of market dynamics. However, as of the study, the gap remains substantial.

The authors release the benchmark and a public leaderboard to facilitate future research in this domain.