💰 quantitative finance

FinanceHarness: Autonomous Financial Deep Research Framework

The paper introduces FinanceHarness, an autonomous framework for end-to-end financial deep research that integrates specialized tools and practitioner-guided workflows, alongside FinanceGym, a rigorous benchmark demonstrating significant performance gaps in current models and measurable improvements through the proposed system.

Original authors: Yijia Xiao, Rujun Han, Yanfei Chen, Zifeng Wang, Ke Jiang, Zhongying CuiZhu, Vishy Tirumalashetty, Wei Wang, Burak Gokturk, Tomas Pfister, Chen-Yu Lee

Published 2026-07-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yijia Xiao, Rujun Han, Yanfei Chen, Zifeng Wang, Ke Jiang, Zhongying CuiZhu, Vishy Tirumalashetty, Wei Wang, Burak Gokturk, Tomas Pfister, Chen-Yu Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but there's a strict rule: you can only use clues that existed before the crime happened. You can't peek at the police report written the next day, and you certainly can't read the newspaper article about the trial that hasn't started yet. This is the world of financial deep research. It's a field where artificial intelligence (AI) tries to act like a human stock analyst, digging through mountains of data to figure out if a company is a good investment. To do this well, the AI needs two things: a massive library of past news and reports (the "clues"), and a way to think critically about what might happen next without cheating by looking at the answer key.

For a long time, AI was great at writing general reports, like summarizing a movie plot or explaining how a volcano works. But finance is a different beast. It's not just about finding facts; it's about connecting the dots between a company's earnings, a competitor's new product, and a change in government policy to predict the future. The problem is that most AI tools are like students who cheat by looking at the test answers before they finish the exam. They accidentally "leak" future information, making them look smarter than they really are. To fix this, scientists needed a way to test AI on financial puzzles where the "future" is strictly locked away, forcing the AI to rely on its own reasoning skills rather than a lucky guess.

This is where a new project called FinanceHarness comes in, along with its training ground, FinanceGym. Think of FinanceGym as a high-stakes video game level designed specifically to test financial AI. The creators built a giant, time-travel-proof library containing millions of articles, but with a magical twist: for every question asked, the library locks the door to any article published after a specific date. If the AI tries to peek at a report from next month, the door stays shut. This ensures that when the AI makes a prediction, it's actually using its brain, not just copying a future fact.

The researchers then built FinanceHarness, which is like a specialized toolkit and a coach for the AI. Instead of just letting the AI wander around the internet, this harness gives it a structured way to work. It forces the AI to gather evidence, check its sources, and write a report that cites exactly where it found its information. It's like giving a detective a magnifying glass, a notepad, and a strict rulebook that says, "You must prove every claim you make."

When they tested the smartest AI models available on this new, tough game, the results were surprising. Even the most advanced AI systems, which usually ace general knowledge tests, struggled mightily. In fact, the best models only got about 40% of the questions right. This suggests that while AI is getting better at reading, it still finds it very hard to act like a seasoned financial analyst who can look at past data and confidently predict what will happen next. The paper shows that simply having a bigger brain (a larger AI model) isn't enough; the AI needs the right tools and a strict environment to learn how to think like a pro.

The team found that their new system, FinanceHarness, helped a standard open-source AI model jump from a score of 25.3% to 32.4%. While that might sound like a small number, in the world of expert financial analysis, it's a significant step forward. It proves that by giving AI the right "harness"—a mix of specialized tools, strict time limits, and expert guidance—we can start to build machines that don't just read the news, but actually understand the complex story of the financial world. The paper concludes that there is still a lot of room for improvement, as even the best systems are far from perfect, but this new framework gives us a clear, fair way to measure how much progress we are really making.

Technical Summary: FinanceHarness: Autonomous Financial Deep Research Framework

1. Problem Statement

While "deep research" agents powered by Large Language Models (LLMs) have become widely adopted for generating general-purpose reports, they are inadequate for financial deep research. Financial analysis requires specialized capabilities beyond generic search, including:

Specialized Knowledge: Analyzing historical earnings, competitor dynamics, and macroeconomic trends to forecast future events.
Temporal Constraints: Professional financial reasoning often involves "post-cutoff" forecasting (predicting outcomes after a specific date), which requires a Point-in-Time (PIT) evaluation framework to prevent the leakage of future information into the research process.
Evaluation Gaps: Existing benchmarks either focus on isolated NLP tasks (e.g., sentiment analysis, filing QA) or general deep research that lacks the rigorous, time-constrained, and expert-validated rubrics necessary to measure the quality of comprehensive financial reports.

2. Methodology

The paper introduces a complete ecosystem comprising a search sandbox, a benchmark, and an execution harness.

A. Point-in-Time (PIT) Financial Search Sandbox

To ensure temporal integrity, the authors constructed a large-scale web corpus (100+ million articles) with reliably extracted publication dates.

Architecture: Articles are embedded using Qwen3-Embedding-4B and stored in a FAISS index.
Access Control: The search server enforces a strict contract where agents can only retrieve documents with publication dates $\le$ the assigned cutoff date for a specific query. This prevents future information leakage during the research phase.

B. FinanceGym: The Benchmark

FinanceGym is a large-scale, expert-validated benchmark constructed on top of the PIT sandbox.

Construction:
1. Entity Graph: A finance entity graph is mined from the corpus, extracting 4.37M edges (relations between entities, events, and sectors).
2. Situation Mining: Research questions are generated by mining "situations" (e.g., cross-category linkages, temporal narratives, divergences like upgrades vs. downgrades) rather than broad clustering.
3. Cutoff Selection: Cutoff dates are selected based on event volume, entity diversity, and relation entropy to ensure high-quality, non-trivial research scenarios.
4. Unconstrained Generation: An LLM generates analyst-style questions, investment theses, and two-tier rubrics without category priming.
Rubric Design:
- Pre-cutoff criteria: Test the retrieval and synthesis of facts available before the cutoff date.
- Post-cutoff criteria: Test the agent's ability to anticipate outcomes verifiable only after the cutoff date.
Validation: A 500-question subset was filtered via an Integer Linear Program (ILP) for balance across topics, sectors, and reasoning types. External professional experts annotated these, resulting in a final benchmark of 400 high-quality questions with 2,464 rubric items (82% expert pass rate).

C. FinanceHarness: The Execution Framework

FinanceHarness is an expert-knowledge-guided agent harness designed to run and optimize agents within the PIT environment.

Layered Design:
- Model Layer: Hosts the orchestrator backbone (e.g., Qwen3.6-27B) and a lightweight reader.
- Runtime Layer: A control plane managing the agent loop, schema validation, tool dispatch, and recovery policies.
- Tool Surface: A tiered interface including core search, deferred financial tools (valuation, risk analysis), and extension connectors.
Modes: Supports research (web-first), analytical (data/computation), and automatic modes.
Training & Optimization: The harness allows for end-to-end optimization. The same rubric judge used for evaluation serves as a reward signal for Group Relative Policy Optimization (GRPO) training, ensuring the model learns the specific tool-use distribution and citation requirements of the financial domain.

3. Key Contributions

FinanceGym Benchmark: A reproducible, PIT-constrained benchmark of 400 expert-annotated questions that separates evidence retrieval (pre-cutoff) from outcome anticipation (post-cutoff).
FinanceHarness Framework: An expert-guided harness that automates environment construction, agent execution, and reward modeling under a strict temporal contract.
Comprehensive Evaluation: A benchmarking of 17 leading systems (including fine-tuned open-weight models, agentic search systems, and proprietary LLMs) demonstrating that current state-of-the-art agents struggle significantly with financial deep research.

4. Results

The authors evaluated 17 baselines and FinanceHarness on FinanceGym. Key findings include:

Low Overall Performance: Even leading LLMs and specialized agents scored below 40% on the overall rubric.
- Top Proprietary: Claude-Opus-4.7 (34.1%), Gemini-3.1-Pro (33.2%).
- Top Open-Weight: FinanceHarness (32.4%) on a 27B backbone.
The Pre/Post-Cutoff Gap: A persistent performance gap exists across all systems. Pre-cutoff scores (retrieval/synthesis) are significantly higher (e.g., ~45%) than post-cutoff scores (forecasting/anticipation, e.g., ~12%), indicating that retrieval alone does not solve financial deep research.
Harness Effectiveness:
- Using the same Qwen3.6-27B backbone, the FinanceHarness configuration (32.4%) significantly outperformed a vanilla search-only model (25.3%) and a naive harness (29.6%).
- Further GRPO training on a separate machine-curated set added a marginal 0.4 points (32.8%), suggesting the harness structure itself provides the primary value.
Efficiency: FinanceHarness achieved performance competitive with much larger open-weight models and approaching the proprietary frontier, despite using a 27B parameter backbone.

5. Significance and Claims

The paper claims that financial deep research is not solved by stronger retrieval alone. The persistent gap between pre-cutoff and post-cutoff performance highlights that the core difficulty lies in transforming historical evidence into forward-looking judgments.

The significance of this work lies in:

Establishing a Standard: Providing the first scalable, reproducible, and temporally constrained benchmark for financial deep research, addressing the leakage issues of previous benchmarks.
Validating the Harness Approach: Demonstrating that an expert-knowledge-guided harness, which aligns tool use, environment constraints, and reward modeling, is essential for automating complex financial analysis.
Defining the Challenge: Showing that even the most advanced agents currently fail to meet professional standards for financial reporting, leaving substantial headroom for future research in agentic financial reasoning.

The authors conclude that evaluation, tool use, and optimization must share the same temporally controlled environment to advance the field effectively.