FinRule-Bench: A Benchmark for Joint Reasoning over Financial Tables and Principles

1. Problem Statement

While Large Language Models (LLMs) are increasingly used for financial analysis, their ability to perform diagnostic auditing—verifying structured financial statements against explicit accounting principles—remains unexplored. Existing benchmarks (e.g., FinQA, TAT-QA) primarily focus on:

• Question Answering: Retrieving facts or performing arithmetic.
• Anomaly Detection: Identifying inconsistencies in synthetically corrupted data.

The Gap: Real-world auditing requires diagnostic completeness. Auditors must not only detect errors but also:

1. Exhaustively enforce a formal system of interdependent rules.
2. Identify which specific principle was violated.
3. Localize the violation to specific records within the table.
4. Handle multiple simultaneous violations without relying on noisy or corrupted data inputs (as real financial statements are often constructed correctly, and errors arise from reasoning failures).

Current benchmarks fail to test whether models can distinguish between competing rules or localize violations in clean, ground-truth data.

2. Methodology

A. Dataset Construction (FinRule-Bench)

The benchmark is built on real-world corporate filings (Form 10-K), ensuring high fidelity and avoiding synthetic noise.

• Data Source: 2024 Form 10-K filings converted into structured Markdown tables.
• Statement Types: Covers four canonical types: Balance Sheets (BS), Cash Flow Statements (CF), Statements of Equity (SE), and Income Statements (SI).
• Ground Truth: Rules are defined by human-curated accounting principles (based on U.S. GAAP/FASB) and implemented as deterministic validators (code-based functions).
• Error Injection: Instead of corrupting data randomly, the authors use controlled, minimal error injection. They take clean tables and deterministically modify specific values or labels to violate exactly one or multiple rules, preserving the original table structure.
• Statistics: The dataset contains ~4,385 single-violation records and 400 multi-violation records across 16 distinct rule categories.

B. Task Formulation

The benchmark defines three progressively complex auditing tasks:

1. Rule Verification: Given a table and a single rule, determine if the table complies (Binary: True/False).
2. Rule Identification: Given a table and a set of rules, identify the single violated rule from the set.
3. Joint Rule Diagnosis: Given a table and a set of rules, detect if any violations exist and, if so, identify all violated rules and localize them to specific records. This is the most challenging task, requiring multi-step reasoning.

C. Evaluation Protocols

• Prompting Strategies: Models are tested under Zero-shot, Few-shot, and Few-shot with Causal-Counterfactual Reasoning.
  • Causal-Counterfactual Protocol: The few-shot examples explicitly show the cause of a violation and a minimal counterfactual edit that would fix it. This tests if models can understand causal dependencies rather than just pattern matching.
• Metrics:
  • Verification: Accuracy, Precision, Recall, F1.
  • Identification: Top-1 Accuracy, Balanced Accuracy.
  • Diagnosis: Step-1 (Violation Detection) Accuracy; Step-2 (Exact Match of all violated rules and their locations).
• Models Evaluated: GPT-4o, Gemini 2.5 Pro (Thinking), Gemini 2.0 Flash, and LLaMA 3.3.

3. Key Contributions

1. FinRule-Bench Dataset: The first benchmark for financial auditing that pairs ground-truth financial statements with explicit, human-curated accounting principles and deterministic validators, moving beyond synthetic anomaly detection.
2. Task Suite: Formalization of three tasks that scale from simple compliance checks to complex, multi-violation diagnostic reasoning.
3. Causal-Counterfactual Reasoning Framework: Introduction of a prompting protocol that forces models to reason about causes and minimal interventions, serving as a diagnostic tool to reveal inconsistencies between decisions and explanations.
4. Comprehensive Analysis: A detailed breakdown of failure modes (e.g., mislocalization, partial coverage) and performance-cost trade-offs across different rule complexities (arithmetic vs. structural vs. conditional).

4. Experimental Results

A. Performance Trends

• Rule Verification: Models perform relatively well (60–74% accuracy), indicating that basic arithmetic and single-rule checks are tractable.
• Rule Identification: Performance drops significantly (40–60% accuracy). Models struggle to discriminate between competing rules when multiple principles could apply.
• Joint Rule Diagnosis: Performance degrades sharply. While Step-1 (detecting if an error exists) is moderate (60–80%), Step-2 (Exact Match of all violations and their locations) is very low (15–34% for top models).
  • Key Finding: Even the strongest models (GPT-4o, Gemini 2.5 Pro) fail to exhaustively localize multiple simultaneous violations.

B. Failure Modes

The primary errors are not random but systematic:

• Incomplete Coverage: Models detect some violations but miss others in multi-violation scenarios.
• Mislocalization: Models identify the correct rule but attribute the violation to the wrong record/row.
• Rule Complexity: Performance is highest on simple arithmetic rules and lowest on conditional and multi-record rules (e.g., rollforward constraints).

C. Impact of Causal-Counterfactual Prompting

• Lightweight Models: Shows clear benefits for smaller models (Gemini 2.0 Flash, LLaMA 3.3), improving performance across all tasks by clarifying causal links.
• Reasoning-Oriented Models: Shows inconsistent or even negative gains for larger "thinking" models (GPT-4o, Gemini 2.5 Pro). The explicit causal scaffolding sometimes introduces redundancy or interferes with the model's internal reasoning processes, particularly in rule discrimination tasks.

D. Cost-Efficiency

• Adding causal-counterfactual reasoning significantly increases token consumption (cost).
• The trade-off is only justified for Joint Rule Diagnosis on complex statements (like Income Statements), where the reasoning overhead helps achieve better localization. For simple verification, the cost outweighs the marginal accuracy gain.

5. Significance and Conclusion

• Diagnostic Completeness: The paper demonstrates that high accuracy on standard financial QA benchmarks does not translate to reliability in auditing. Models lack the "diagnostic completeness" required to exhaustively enforce rule systems.
• Systematic Failures: The benchmark reveals that LLMs struggle with compositionality (handling multiple interacting rules) and attribution (linking a rule to a specific data point), which are critical for high-stakes financial compliance.
• Future Directions: The work calls for new architectures or training paradigms that can handle structured, rule-governed reasoning without relying solely on massive parameter counts or verbose prompting.
• Open Science: The authors release the full benchmark, validators, and evaluation code to enable reproducible research into financial reasoning failures.

In summary, FinRule-Bench establishes that while LLMs are capable of surface-level financial analysis, they currently lack the rigorous, exhaustive reasoning capabilities required for automated financial auditing and compliance verification.