Technical Summary: BacktestBench

Problem Statement

Quantitative backtesting is a critical component of evaluating trading strategies, yet it remains hindered by high technical barriers and limited scalability. The traditional workflow requires specialized expertise to construct factor combinations, query historical data with precise temporal logic, and implement error-free code to execute complex strategies. While Large Language Models (LLMs) offer transformative potential for automating this interdisciplinary workflow through code generation and agentic planning, their practical application is currently stalled. The primary bottleneck is the absence of a large-scale, high-quality benchmark dedicated to automated quantitative backtesting. Existing datasets (e.g., Text-to-SQL or general code generation benchmarks) fail to capture the unique temporal logic, multi-step reasoning, and rigorous reproducibility requirements inherent in financial strategy evaluation.

Methodology

1. Dataset Construction: BacktestBench

The authors introduce BacktestBench, the first large-scale benchmark for automated quantitative backtesting.

• Data Source: The dataset is constructed from over 6 million real market records (daily trading data) from China's three major exchanges (Shanghai, Shenzhen, Beijing) spanning January 2020 to September 2025.
• Scale and Composition: It comprises 18,246 meticulously annotated question-answering (QA) pairs.
• Core Concepts: The dataset distinguishes between factors (numerical variables), signals (Boolean decisions), strategies (executable trading policies), and KPIs (performance metrics).
• Task Categories: The benchmark evaluates models across four distinct decision-making tasks:
  1. Metrics Calculation: Computing specific KPIs (e.g., Sharpe Ratio, Max Drawdown) for a given strategy.
  2. Ticker Selection: Identifying the best-performing asset from a candidate universe under a fixed strategy.
  3. Parameter Confirmation: Selecting optimal hyperparameters (e.g., lookback windows) for a strategy.
  4. Strategy Selection: Choosing the superior logic among competing strategies under identical market conditions.
• Generation Pipeline:
  • Code-to-Text: The authors first generated 43 theoretical factors and 7 KPIs, encapsulating them into 80 atomic strategy functions. They synthesized executable Python backtesting code (92,170 buy strategies and 112,791 sell strategies).
  • Reverse Engineering: Using five state-of-the-art LLMs, they performed "code-to-text" reverse engineering to generate natural language strategy descriptions aligned with the actual code logic.
  • Quality Control: A multi-model peer-review process ensured high fidelity. A description was retained only if four independent LLM auditors unanimously agreed on its code fidelity and strategy validity.
  • Human Validation: A random sample of 200 instances achieved a 97.5% pass rate for description completeness and 98.5% for code correctness.

2. Baseline System: AutoBacktest

To address the challenges of the dataset, the authors propose AutoBacktest, a robust multi-agent collaborative system designed to mimic the workflow of a quantitative researcher. It decouples the complex pipeline into three specialized agents:

1. Summarizer: Extracts semantic-level financial indicators (factors and KPIs) from natural language descriptions. It utilizes a BM25 retrieval mechanism to map informal terms to a standardized library of indicator names, reducing ambiguity.
2. Retriever: Handles data-level operations. It maps identified indicators to unique Short Codes (compact, token-efficient identifiers) and generates executable SQL queries to fetch historical market data from a PostgreSQL database. It employs an execution-based validation loop to ensure the SQL returns non-empty, valid results.
3. Coder: Focuses on logic-level implementation. It takes the retrieved data (as a Pandas DataFrame) and the standardized indicator context to generate and execute Python backtesting code. It iteratively debugs the code to calculate the required metrics or execute the strategy logic under strict backtesting protocols (e.g., long-only, buy-at-open/sell-at-close, no look-ahead bias).

Key Contributions

1. BacktestBench: A pioneering large-scale benchmark containing 18,246 QA pairs derived from real market data, covering four core quantitative tasks. It bridges the gap between general-purpose code generation and the rigorous requirements of quantitative finance.
2. AutoBacktest Framework: A multi-agent baseline system that translates natural language strategies into reproducible backtests. It introduces a modular architecture (Summarizer, Retriever, Coder) and a "Short Code" mechanism to ground LLMs in precise database schemas and financial definitions.
3. Comprehensive Evaluation: The paper evaluates 23 mainstream LLMs (both open-source and closed-source) on BacktestBench, providing the first systematic analysis of LLM capabilities in this vertical domain.

Experimental Results

The evaluation of 23 LLMs reveals several critical insights:

• Performance Gap: Closed-source models generally outperform open-source counterparts. Gemini 3 Pro achieved the highest Overall Accuracy (OA) of 67.41%, leading significantly in complex logical reasoning tasks like Metrics Calculation (51.67%). The best open-source model, GLM 4.7, achieved an OA of 56.83%.
• Scaling Sensitivity: Logical reasoning tasks (e.g., Metrics Calculation) are highly sensitive to parameter scale. Reducing model size from 235B to 4B caused the Metrics Calculation score to plummet from 34.71% to 1.77%. Conversely, indicator retrieval tasks showed lower sensitivity, with smaller models maintaining relatively high retrieval accuracy.
• The Role of Chain-of-Thought (CoT): Models lacking CoT reasoning (e.g., Kimi Linear 48B) exhibited a decoupling between syntax and logic. They achieved high SQL execution correctness (99.22%) but low execution accuracy (48.46%) and near-zero strategy logic scores, indicating they could generate executable code that was logically flawed.
• Complexity Bottlenecks: Performance degraded sharply as KPI complexity increased. Simple metrics like Win Rate and Maximum Drawdown saw higher accuracy, while complex statistical indicators like Volatility and Sharpe Ratio became "disaster zones" for most models, with pass rates often below 20% for smaller or non-CoT models.
• Ablation Studies:
  • Short Codes: The inclusion of Short Codes (standardized database identifiers) significantly improved SQL Execution Accuracy (EA) across all models, with some seeing gains of over 20 percentage points.
  • Ground Truth: Providing ground-truth Short Codes was more impactful than providing ground-truth SQL, suggesting that understanding financial indicator semantics is the primary bottleneck rather than SQL syntax generation.

Significance and Claims

The paper claims that BacktestBench establishes a standardized evaluation protocol and a rigorous data resource, effectively bridging the gap between general-purpose code generation and quantitative investment research. By demonstrating that current LLMs struggle with the "high-order logic" required for complex financial metrics (like annualized volatility and Sharpe ratios), the work highlights the specific limitations of current AI in financial decision-making.

The authors position BacktestBench not as a tool for generating immediate alpha or live trading advice, but as a foundational baseline for testing LLMs' strategy understanding, data retrieval, and code generation capabilities in a structured financial environment. The work lays the groundwork for future advancements in intelligent financial decision-making by identifying that while LLMs are capable of procedural code generation, they still require significant improvements in rigorous logical deduction and statistical reasoning to automate complex quantitative workflows reliably.