Backtrader-Bench: Benchmarking LLM Agents on Algorithmic Trading with Self-Generated MCQs
The paper introduces Backtrader-Bench, a novel framework for evaluating LLM coding agents in algorithmic trading through self-generated, code-verified multiple-choice questions, demonstrating that tool-augmented agents significantly outperform non-tool baselines while establishing a scalable infrastructure for future reinforcement learning training.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a world where computers don't just chat with you but actually do things: they write code, run programs, and solve complex problems on their own. These are called "AI agents." Think of them as super-smart digital interns who can open a terminal, type commands, and execute software to get a job done. In the world of finance, these agents are being trained to act like quantitative traders—people who use math and computers to decide when to buy or sell stocks. The big question is: Can these AI interns actually do the job, or are they just good at pretending?
To find out, we need to test them. But testing a trader is tricky. You can't just ask them, "What's the profit?" because they might guess the right number without actually doing the math. It's like asking a chef to describe a cake they've never baked; they might say "it's sweet and fluffy," but that doesn't mean they know how to mix the ingredients. In finance, getting the numbers wrong isn't just a bad grade; it means losing real money. So, scientists need a way to test if an AI can actually run the trading software, crunch the numbers, and tell the truth about how a strategy performed.
This is exactly what the paper "Backtrader-Bench" tackles. The researchers built a giant, automated testing ground to see if AI agents can handle the messy, math-heavy reality of algorithmic trading. They created a system that generates thousands of tricky questions about stock market simulations and then watches to see if the AI can solve them by writing and running code, or if it just tries to guess the answer from its memory.
The "Backtest" Video Game
First, let's understand the playground. The researchers used a tool called Backtrader, which is like a video game simulator for stock trading. You give it a set of rules (a "strategy"), like "Buy when the 10-day average price goes above the 30-day average," and it runs a simulation using historical stock data. It tells you how much money you would have made or lost.
The problem is that these simulations are full of tiny, sneaky details. A small mistake in the code—like forgetting to pay a tiny transaction fee or miscounting how many shares you can afford—can turn a winning strategy into a losing one. If an AI agent is just "hallucinating" (making things up that sound plausible), it might give you a perfect-looking profit number that is completely fake.
The Two-Part Test: The Quiz and the Filter
To catch these fake answers, the authors built Backtrader-Bench, a framework with two clever parts.
Part 1: The Deterministic Quiz
Imagine a teacher who writes a test, solves it themselves, and then checks the answer key against their own work to make sure it's 100% correct. That's what the first part of the system does. It takes a trading strategy, runs the simulation, and automatically generates multiple-choice questions.
- Easy questions: "What was the stock price on January 1st?" (Just looking up a number).
- Medium questions: "How many trades were profitable?" (Counting and filtering).
- Hard questions: "What was the maximum amount of money the portfolio lost at any point?" (This requires complex math and tracking the highest and lowest points over time).
Crucially, the system has a "checker" robot that re-runs the exact same simulation to verify the answer key. If the answer key doesn't match the re-run, the question is thrown out. This ensures the test is fair and the answers are undeniably true.
Part 2: The "Hard Mode" Filter
The first part is great, but what if the AI is too smart and just memorizes the answers? To fix this, the researchers added a second, more aggressive pipeline. They used an AI "Generator" to invent brand-new, weird questions on the fly.
Then, they ran a "No-Tool Solver" (an AI that isn't allowed to use a computer) to try and answer them.
- If the No-Tool Solver gets it right, the question is too easy. It's discarded because the AI just guessed or knew it from training data.
- If the No-Tool Solver fails, but a "Tool-Augmented Solver" (an AI allowed to write and run code) gets it right, the question is kept.
This creates a special set of "Hard Mode" questions that require the AI to actually do the math and run the code. You can't guess these; you have to execute the program.
The Results: Guessing vs. Doing
The researchers put 11 different AI models to the test. Some were allowed to use tools (write code, run simulations), and some were forced to answer without them (just using their brain).
The "With-Tools" Champions:
When the AI agents were allowed to use code execution, they were fantastic. The top models (GPT-5.5 and Opus 4.7) got 90.0% of the questions right in a single try. They wrote the code, ran the simulation, and read the result. They were like a chef actually baking the cake and tasting it.
The "No-Tools" Struggle:
When the AI agents were banned from using code and had to guess from memory, the scores dropped. The best of them only got 73.0% right on average. But here's the scary part: when the researchers tested them on the "Hard Mode" questions (the ones the filter kept because they were too hard to guess), the scores plummeted.
- Half of the models dropped to around 25% accuracy.
- Since there are four options in a multiple-choice question, 25% is exactly what you get if you just close your eyes and point at a letter.
This suggests that without the ability to run code, many of these AI agents are essentially guessing on complex financial tasks. They might sound confident, but they are just making up numbers.
Why This Matters
The paper shows that for AI to be useful in real-world finance, it can't just be a chatbot that knows financial terms. It needs to be an agent that can actually do the work. The "Backtrader-Bench" framework proves that while AI is getting better, it still makes dangerous mistakes if it tries to trade without running the numbers first.
The authors suggest that this testing system isn't just for grading AI; it's a training ground. By feeding these verified, hard questions back into the AI, they hope to train a specialized "quant agent" that can reliably handle the complex, math-heavy world of algorithmic trading without making silent, costly errors. Until then, the paper suggests we should be very careful about trusting an AI's trading advice unless it can prove it actually ran the simulation.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.