EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial Statements

This paper introduces EDINET-Bench, a challenging open-source benchmark derived from ten years of Japanese financial reports to evaluate LLMs on complex tasks like fraud detection and earnings forecasting, revealing that current models struggle significantly without specialized scaffolding and highlighting the need for more realistic evaluation frameworks.

The Gist

Imagine you are trying to teach a brilliant, super-smart robot how to be a financial detective. You want to see if this robot can read a company's annual report, spot hidden lies, predict if the company will make more money next year, or guess what kind of business they are in just by looking at the numbers.

This paper, EDINET-Bench, is essentially a "final exam" created to test these robots (Large Language Models, or LLMs) on exactly those difficult tasks.

Here is the breakdown of the paper in simple terms:

1. The Problem: The Robot is Good at Math, Bad at Finance

We've seen AI get really good at math problems and writing code. But finance is different. It's like the difference between solving a Sudoku puzzle and diagnosing a patient with a rare disease. Finance requires reading between the lines, connecting tiny clues across hundreds of pages, and understanding human behavior and complex rules.

Until now, there haven't been many "hard" tests for AI in finance. Most existing tests are like asking, "What is the profit of Company X?" (a simple lookup). This paper wanted to ask, "Is Company X lying about its profits?" (a complex investigation).

2. The Solution: EDINET-Bench (The "Financial Gym")

The researchers built a new training ground called EDINET-Bench.

• The Source: They used real documents from Japan's "EDINET," which is like the US SEC's EDGAR system. It's a massive library of annual reports filed by Japanese companies over the last 10 years.
• The Tools: They built a special tool (called edinet2dataset) to dig through these reports, turning messy PDFs and tables into clean data the AI can read.
• The Three Challenges (The Exam Questions):
  1. Fraud Detection: Can the AI spot a company that is cooking the books? (Like finding a needle in a haystack of numbers).
  2. Earnings Forecasting: Can the AI look at this year's report and guess if the company will make more or less money next year?
  3. Industry Prediction: Can the AI look at a company's balance sheet and guess if they are a bank, a car manufacturer, or a food company?

3. The Results: The Robots Struggled

The researchers tested the smartest AI models available (like GPT-4o, Claude 3.7, and others) on this exam.

The Verdict: The AI did not do well.

• The Analogy: Imagine a student who has memorized the entire dictionary but fails a basic math test because they don't understand how to apply the numbers.
• The Reality: The AI models performed only slightly better than a very simple, old-school statistical method called "Logistic Regression." In some cases, they were barely better than flipping a coin.
• Why? The AI was given the whole report and asked to "just read it." But financial fraud is subtle. It's not usually a big red flag; it's a tiny inconsistency in a footnote that contradicts a table three pages away. The AI, in its current "read-only" mode, missed these connections.

4. The Big Lesson: Reading Isn't Enough

The paper concludes that just handing an AI a document isn't enough to make it a financial expert.

• The Metaphor: Giving an AI a stack of annual reports is like handing a detective a 500-page mystery novel and saying, "Solve the crime." The detective needs a magnifying glass, a whiteboard to connect clues, and maybe a team of experts to talk to.
• The Future: To make AI useful in finance, we need to build "scaffolding." This means giving the AI tools to simulate real-world scenarios, let it ask questions, let it cross-reference data, and support its reasoning process. We need to move from "Passive Reader" to "Active Analyst."

Summary

EDINET-Bench is a wake-up call. It shows that while AI is amazing at many things, it is still very clumsy when it comes to the high-stakes, nuanced world of financial analysis. The researchers are releasing their data and tools to the public so that other scientists can help build better, more "financially literate" AI for the future.

In short: The AI is smart, but it's not ready to be a Wall Street analyst yet. It needs more training and better tools to understand the hidden stories behind the numbers.

Technical Summary

Here is a detailed technical summary of the paper "EDINET-BENCH: EVALUATING LLMS ON COMPLEX FINANCIAL TASKS USING JAPANESE FINANCIAL STATEMENTS."

1. Problem Statement

While Large Language Models (LLMs) have achieved human-level or superhuman performance in domains like mathematics and coding, their capabilities in the financial domain remain underexplored and lag behind. Existing financial benchmarks (e.g., FinQA, FinanceBench) primarily focus on:

• Simple knowledge-based question answering.
• Data extraction from isolated tables.
• Tasks solvable with basic financial knowledge and simple computations.

These benchmarks fail to capture the complexity of real-world financial analysis, which requires expert-level reasoning: synthesizing information across multiple tables (Balance Sheet, Profit & Loss, Cash Flow) and textual sections (narrative disclosures, risk factors) to detect subtle anomalies, forecast earnings, or classify industries. Furthermore, there is a lack of open-source benchmarks specifically for Japanese financial reports, which have unique structural and linguistic characteristics.

2. Methodology

Data Source and Construction

The authors constructed EDINET-Bench using data from Japan's Electronic Disclosure for Investors' NETwork (EDINET), a platform managed by the Financial Services Agency (FSA).

• Corpus: They collected approximately 40,000 annual reports (including amended reports) from 2014 to 2024, covering roughly 4,000 listed companies.
• Tooling: They developed edinet2dataset, an open-source tool (inspired by edgar-crawler) to download and parse EDINET reports. It extracts structured data (BS, PL, CF, Summary) in TSV format and unstructured text (Text blocks) using Polars for high-speed processing.
• Labeling Strategy:
  • Accounting Fraud Detection: They identified 668 fraudulent cases by analyzing amended annual reports. Using pdfminer to extract text and Claude 3.7 Sonnet to classify amendment reasons, they filtered for reports indicating fraud (e.g., misstated assets, revenue overstatement). Non-fraud cases were randomly sampled. The dataset was split into 865 training and 224 test samples (after parsing errors reduced the count).
  • Earnings Forecasting: A binary classification task (Increase/Decrease) predicting the next fiscal year's "Profit Attributable to Owners of the Parent Company." They created pairs of consecutive reports for 1,000 companies, resulting in 549 training and 451 test samples.
  • Industry Prediction: A multi-class classification task predicting one of 16 consolidated industry categories (based on TOPIX-33 and MIC standards). The test set contains 476 samples.

Evaluation Setup

• Models: Evaluated state-of-the-art closed-source models (GPT-4o, GPT-5, o4-mini, Claude 3.5/3.7 Sonnet) and open-weight models (Kimi-K2, DeepSeek-V3/R1, Llama 3.3 70B).
• Baselines: Compared against classical machine learning models (Logistic Regression, Random Forest, XGBoost) trained on the same data.
• Input Modalities: Tested three configurations to assess information density:
  1. Summary only.
  2. Summary + Tabular Data (BS, CF, PL).
  3. Summary + Tabular Data + Text (unstructured narrative).
• Metrics: ROC-AUC and Matthews Correlation Coefficient (MCC) for fraud/earnings; Accuracy for industry prediction.

3. Key Contributions

1. EDINET-Bench: The first open-source benchmark for accounting fraud detection using real-world annual reports, specifically tailored for the Japanese market.
2. Expert-Level Reasoning Tasks: Unlike previous benchmarks, EDINET-Bench requires models to process entire reports and integrate cross-modal data (tables + text) to solve high-stakes tasks.
3. Toolkit Release: Public release of edinet2dataset and the benchmark dataset/code to facilitate reproducibility and future research.
4. Contamination Analysis: The authors rigorously tested for data contamination by evaluating models on company name prediction (given only financial data) and analyzing performance trends by fiscal year, confirming that test set leakage is unlikely to be the primary driver of results.

4. Key Results

The experiments revealed that even state-of-the-art LLMs struggle significantly in this domain, often performing only marginally better than simple logistic regression.

• Accounting Fraud Detection:
  • Performance: Random Forest achieved the highest ROC-AUC (0.75). The best LLM (Claude 3.5 Sonnet) achieved 0.73 ROC-AUC, barely outperforming Logistic Regression (0.68).
  • Insight: Incorporating textual information significantly boosted LLM performance (e.g., Claude 3.5 Sonnet jumped from 0.64 to 0.73 ROC-AUC when text was added), suggesting that narrative disclosures contain critical fraud signals. However, models showed bias toward auditor reputation (e.g., trusting "reputable" firms).
• Earnings Forecasting:
  • Performance: All models struggled, with GPT-5 reaching only 0.65 ROC-AUC.
  • Insight: Unlike fraud detection, adding text did not improve performance, suggesting that earnings direction is harder to predict from narrative alone and relies more on complex numerical trends that current LLMs fail to extrapolate accurately.
• Industry Prediction:
  • Performance: This was the easiest task. Claude 3.5 Sonnet achieved 0.41 accuracy (vs. random chance of ~0.06), significantly outperforming baselines.
  • Insight: Industry classification relies heavily on asset composition (e.g., high leverage for banks, specific inventory types for manufacturing), which is more discernible from tabular data than fraud patterns.

5. Significance and Future Directions

• Limitations of Current LLMs: The results indicate that simply feeding annual reports to LLMs in a zero-shot setting is insufficient for complex financial analysis. The "benchmark hill" in finance is steeper than in math or coding.
• Need for New Frameworks: The authors argue that future benchmarks must move beyond simple QA to agent-based frameworks. Real-world financial professionals use simulations, access external news, and perform iterative reasoning. Benchmarks should incorporate:
  • Realistic simulations.
  • Task-specific reasoning support (scaffolding).
  • Agentic capabilities (web browsing, code execution for quantitative analysis).
• Global Applicability: While currently focused on Japan, the authors suggest the pipeline can be extended to other markets (e.g., US EDGAR) to create multilingual, cross-border financial benchmarks.

In conclusion, EDINET-Bench establishes a new standard for evaluating financial reasoning, highlighting a critical gap between current LLM capabilities and the expert-level reasoning required for real-world financial decision-making.