Technical Summary: Are Large Language Models Reliable Reviewers? A Benchmark for Error Detection in Financial Documents

Problem Statement

Ensuring the accuracy of financial documents is critical for economic analysis, regulatory compliance, and corporate decision-making. Despite the prevalence of human errors in financial workflows (with surveys indicating 18% of practitioners make daily errors), the capability of Large Language Models (LLMs) to identify subtle, complex errors in financial texts remains unexplored. Existing benchmarks primarily focus on general-domain grammatical errors, hallucination detection in short texts, or standard financial tasks like sentiment analysis and question answering. They lack the capacity to evaluate the multi-hop reasoning and long-context understanding required to detect semantic, domain-specific, and logical inconsistencies in lengthy financial documents.

Methodology

1. FinED-Bench Construction

The authors introduce FinED-Bench, the first publicly available benchmark specifically designed for financial error detection. The construction process involves a semi-automated pipeline to ensure data freshness and prevent training data contamination:

• Data Collection: Financial documents published after February 2025 were collected to postdate the knowledge cutoffs of evaluated models. Documents were converted to plain text, with tables preserved in Markdown format.
• Error Taxonomy: A two-tier taxonomy was developed, categorizing errors into three cognitive levels:
  1. General Knowledge Errors (GKEs): Surface-level issues like illegal dates (e.g., "February 30"), redundant statements, or missing values.
  2. Financial Domain Knowledge Errors (FKEs): Misuse of terminology (e.g., confusing "price-to-earnings" with "price-to-book"), incorrect legal references, or ambiguous expressions.
  3. Financial Reasoning Errors (FREs): Complex logical inconsistencies requiring cross-document verification, such as conflicting numerical data between text and tables, time contradictions, or clause conflicts.
• Error Injection: Documents were segmented into semantically complete fragments. Domain experts provided error seeds, which were then used to guide LLMs (GPT-4o) to inject errors contextually.
• Filtering: A two-stage filtering process was applied:
  • Model-based: GPT-4o re-evaluated injected errors for contextual appropriateness.
  • Manual Verification: Five financial experts manually verified the remaining errors to eliminate bias and ensure realism.
• Dataset Composition: The benchmark comprises 973 documents (average length ~3,784 words) with 4,123 annotated error instances. A "Hard" subset (FinED-Bench-Hard) includes 24 documents ranging from 32K to 120K words to test long-context capabilities.

2. Experimental Setup

The study evaluated a diverse set of LLMs, including:

• General-purpose models: GPT-4o, GPT-4o-mini, GPT-3.5-turbo, Qwen3 series (8B, 14B), and DeepSeek-R1 derivatives.
• Domain-specific models: Fin-R1 and Dianjin-R1-7B.
• Evaluation Metrics: Precision, Recall, and F1-score. A prediction was considered correct only if the error span matched the ground truth and the error type was accurately classified.
• Fine-tuning: A supervised fine-tuning (SFT) dataset was constructed using the same injection pipeline (without human verification) to test if task-specific adaptation improves performance.

Key Results

1. Current LLM Performance

Experimental results indicate that even state-of-the-art models struggle significantly with financial error detection:

• Overall Performance: The best-performing model, GPT-4o, achieved an overall F1 score of 48.34%.
• Error Type Disparity: Performance degrades as cognitive complexity increases. GPT-4o scored 52.33% on General Knowledge Errors but dropped to 38.00% on Financial Reasoning Errors. Similar trends were observed across Qwen3-14B (dropping from 51.41% to 29.10%) and other models.
• Long-Context Degradation: Document length severely impacts performance. F1 scores declined from 40.16% for documents under 2.5K words to 16.66% for documents exceeding 50K words.
• Domain-Specific Limitations: Specialized financial models (Fin-R1, Dianjin-R1-7B) did not outperform their base general-purpose counterparts significantly. In some cases, performance even declined, suggesting that fine-tuning for tasks like stock prediction or QA does not transfer to error detection.

2. Impact of Reasoning and Fine-tuning

• Reasoning Capabilities: Models with explicit reasoning capabilities (e.g., Qwen3-14B with "thinking" enabled) significantly outperformed their "no-thinking" variants. For instance, Qwen3-14B's F1 on General Knowledge Errors improved from 33.72% to 51.41% when reasoning was enabled.
• Supervised Fine-Tuning (SFT): Fine-tuning on financial error detection data substantially boosted performance. Qwen3-14B saw a 10.70% increase in overall F1 score (from 43.15% to 53.85%) after SFT, with notable gains in detecting conflicting expressions and ambiguous terms. Crucially, this improvement did not degrade the model's performance on other financial tasks (e.g., text classification, translation), indicating preserved generalization.

3. Human Baseline

Even human annotators with finance backgrounds achieved an F1 score of 63.63%, highlighting that the task remains highly challenging even for experts, particularly in long documents.

Significance and Claims

The paper positions FinED-Bench as a critical tool for evaluating the reliability of LLMs in professional financial contexts. Its primary contributions are:

1. Benchmarking Gap: It fills the void in systematic research regarding error detection in lengthy, complex financial documents, moving beyond short-text or general-domain evaluations.
2. Realistic Evaluation: By using post-cutoff data and a rigorous injection/verification pipeline, it provides a realistic assessment of current model capabilities without data contamination.
3. Diagnostic Insights: The results reveal that while LLMs possess strong contextual understanding, they lack the robust reasoning and domain-specific logic required for high-stakes financial review. The study demonstrates that reasoning capabilities and domain-specific fine-tuning are essential for improving error detection, whereas generic domain adaptation (e.g., for stock prediction) is insufficient.

The authors conclude that while LLMs show promise, they are not yet fully reliable reviewers for financial documents, particularly in high-complexity, long-context scenarios. The benchmark serves as a foundation for future research into developing more robust, reasoning-enhanced models for financial compliance and analysis.