🤖 AI

OpenFinGym: A Verifiable Multi-Task Gym Environment for Evaluating Quant Agents

This paper introduces OpenFinGym, a unified and verifiable multi-task gym environment that addresses the fragmentation and lack of financial relevance in existing benchmarks by integrating diverse quantitative finance workflows—from forecasting and strategy construction to real-time trading and fraud detection—along with automated task generation and robust verification mechanisms to better evaluate large language model agents.

Original authors: Kaicheng Zhang, Wen Ge, Lei Jiang, Weixin Yang, Jordan Langham-Lopez, Jialin Yu, Lukasz Szpruch, Hao Ni

Published 2026-06-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kaicheng Zhang, Wen Ge, Lei Jiang, Weixin Yang, Jordan Langham-Lopez, Jialin Yu, Lukasz Szpruch, Hao Ni

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to be a professional stock trader. In the past, researchers would test this robot on just one specific skill at a time, like "Can you guess tomorrow's price?" or "Can you spot a fake transaction?"

The problem is that real-life finance is messy and connected. A good trader doesn't just guess prices; they guess prices, build a strategy, manage risks, and react to live news all at once. If you only test the robot on guessing prices, you might think it's a genius, even if it would lose all its money the moment it tried to actually trade.

OpenFinGym is a new, all-in-one "training gym" designed to fix this. It's a digital playground where AI agents (the robots) can practice the entire lifecycle of a financial job, not just isolated tricks.

Here is how it works, broken down into simple concepts:

1. The Four Main Drills

Instead of just one test, OpenFinGym offers four different types of "drills" that mimic real financial work:

Forecasting (The Crystal Ball): The agent tries to predict future prices, volatility, or market trends based on history.
Trading (The Action): The agent actually buys and sells assets. It can do this on historical data (like a video game replay) or on live, real-time data (like a live sports match).
Market Generation (The Simulator): The agent tries to create fake market data that looks so real, it's indistinguishable from the real thing. This helps stress-test strategies.
Fraud Detection (The Security Guard): The agent looks at complex webs of transactions to spot suspicious or illegal activity.

2. The "Glass Box" Rule (No Cheating)

In many AI tests, the robot might accidentally "peek" at the answer key while it's studying. OpenFinGym prevents this with a strict Container System.

Think of the AI agent as a student taking an exam in a sealed room. They have all their textbooks (training data) and the exam questions (test inputs).
However, the Answer Key (the true future prices or the actual fraud labels) is locked in a separate, secure vault that only the Proctor (the Verifier) can see.
The student submits their answers, and the Proctor checks them against the key. This ensures the AI isn't cheating by memorizing the answers.

3. The "Recipe Book" Machine

One of the coolest features is how they built the gym. Instead of humans manually writing 78 different tests, they built an Automated Pipeline.

Imagine a machine that reads academic finance papers (the "recipes" for how to trade or predict), extracts the data and rules, and automatically builds a playable game level out of them.
This means the gym is constantly growing and stays up-to-date with the latest scientific research without needing a human to code every single new task.

4. The Results: Specialized Athletes, Not Superheroes

The authors tested several of the world's most advanced AI models in this gym. They found something interesting: There is no single "best" robot.

One AI was great at predicting prices but bad at spotting fraud.
Another was amazing at trading but struggled with generating fake data.
It's like a sports team: You don't want a single player who tries to do everything; you want specialists who excel at their specific positions. OpenFinGym helps us identify these specialists.

5. Training the Robots

The paper also shows that this gym is great for training the AI, not just testing it.

They took a basic AI model that couldn't even write a working trading script (it failed 100% of the time).
They used the gym to teach it (a process called "Supervised Fine-Tuning" and "Reinforcement Learning").
The Result: The AI went from failing every task to succeeding 100% of the time, and its trading strategies became significantly more profitable.

Summary

OpenFinGym is a unified, cheat-proof training ground for financial AI. It moves away from testing robots on single, isolated tasks and instead puts them through a realistic, multi-stage simulation of the financial world. It ensures that when we say an AI is "good at finance," we mean it can actually handle the complex, interconnected reality of markets, not just solve a simple math puzzle.

Technical Summary: OpenFinGym

Problem Statement

The evaluation of Large Language Model (LLM) agents in quantitative finance remains fragmented. Existing benchmarks typically isolate single tasks (e.g., forecasting or trading) or rely on static datasets that fail to capture the interdependent, multi-stage nature of real financial workflows. Furthermore, current platforms often overlook financial relevance, lack rigorous controls for data leakage (look-ahead bias), and do not support a unified interface for diverse task families such as forecasting, market generation, trading, and fraud detection. This fragmentation leads to an overestimation of agent competence and a failure to reveal weaknesses in generalization and real-market interaction.

Methodology

OpenFinGym addresses these gaps through a unified gym environment designed for rigorous, scalable, and leakage-resistant evaluation. The system is built on three core components:

1. Task Specifications and Taxonomy

Tasks are formalized as machine learning problems represented by a triplet $(D, T, E)$ :

$D$ (Dataset): Includes training and test splits.
$T$ (Task Description): Defines data sources, input-output interfaces, objectives, and output formats.
$E$ (Evaluation Protocol): Specifies metrics and aggregation rules.

The environment covers four distinct task families:

Forecasting: Predicting future financial quantities (returns, prices, volatility) using statistical and finance-specific metrics (e.g., $R^2$ , Sharpe ratio, $\Delta$ -risk).
Trading: Sequential decision-making on historical (offline) or live (real-time) data, evaluated on PnL, drawdown, and turnover.
Market Generation: Creating realistic financial time series or scenarios, evaluated via distributional, temporal, and financial diagnostics (e.g., autocorrelation, tail-risk).
Fraud Detection: Classification or anomaly detection on transactional and graph-structured data.

2. Automated Task Construction Pipeline

OpenFinGym introduces a pipeline to convert quantitative finance publications into executable task packages. This process involves:

Paper Harvesting: Filtering academic papers based on experimental protocols and data availability.
Dataset and Reward Construction: Using a generator-reviewer loop (LLM + unit tests) to materialize datasets and evaluation metrics, ensuring they match source specifications.
Task Assembly: Bundling datasets, rewards, and specifications into self-contained packages. A key design principle is the separation of agent-visible information from verifier-only ground truth to prevent leakage.

3. Containerized Runtime and Verification

Leakage Control: Agents run inside containers where held-out test labels are explicitly excluded. A host-side verifier service computes rewards using the ground truth, ensuring agents cannot access evaluation data during execution.
Real-Time Infrastructure: For trading and long-horizon forecasting, the system employs a low-latency data streaming service (using WebSockets) and a paper trading engine. It supports deferred resolution, where outcomes for long-horizon forecasts are resolved asynchronously via a persistent SQLite ledger once market data becomes available.
Training Integration: The environment supports Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) post-training, allowing agents to learn from successful trajectories.

Key Contributions

Unified Task Suite: A curated collection of 78 verified quantitative-finance tasks spanning forecasting, market generation, trading, and fraud detection, covering multiple asset classes (equities, crypto, FX, commodities, etc.).
Automated Construction Pipeline: A system that transforms academic literature into executable, verifiable benchmark tasks, reducing manual engineering effort and ensuring reproducibility.
Leakage-Resistant Runtime: A containerized execution environment with a host-side verifier that strictly separates training data from test ground truth, preventing temporal contamination and look-ahead bias.
Post-Training Support: An integrated framework for SFT and RL, enabling the improvement of agent capabilities on specific financial workflows.

Results

The authors benchmarked four frontier LLMs (GPT-4o, GPT-5.1-codex-mini, Haiku 4.5, Sonnet 4.6) and evaluated the impact of SFT and RL on Qwen3 base models.

Agent Performance: No single model dominated across all task families.
- Forecasting: GPT-5.1-codex-mini achieved the best reward (0.036).
- Market Generation: Sonnet 4.6 performed best (lowest reward of 7.114).
- Trading: GPT-4o achieved the highest reward (19.912).
- Fraud Detection: Sonnet 4.6 outperformed others with a reward of 1.630.
- All models achieved 100% success rates on forecasting, trading, and fraud detection, indicating reliable interface interaction.
Post-Training Efficacy:
- Base Qwen3 models (1.7B, 4B, 8B) failed to produce executable submissions (0% success) on held-out forecasting tasks.
- SFT improved success rates to approximately 50–80%.
- RL (GRPO) further boosted success rates to 100% across all models and tasks, while reducing mean rewards (improving performance) by an order of magnitude on specific tasks like Treasury and Crypto.

Significance and Claims

The paper positions OpenFinGym as a critical infrastructure for advancing autonomous financial agents. Its significance lies in:

Holistic Evaluation: Moving beyond isolated tasks to assess agents across interconnected financial workflows, providing a more realistic measure of generalization.
Rigorous Verification: Establishing a standard for leakage-resistant evaluation in high-stakes domains, ensuring that benchmark results reflect genuine capability rather than data contamination.
Scalability: Demonstrating that automated pipelines can generate complex, literature-grounded benchmarks, facilitating rapid expansion of the evaluation landscape.
Training Utility: Proving that the environment serves not just as a benchmark but as an effective training ground, where SFT and RL can transform base models from non-functional to highly competent agents in quantitative finance.

The authors note that while the current SFT and RL results are promising, they represent an initial demonstration. Future work aims to scale trajectory collection, incorporate stronger backbone models, and expand the task families to include risk management and broader execution settings.