🤖 machine learning

V-FiLLM: Verified Financial LLM Reasoning Benchmark

The paper introduces V-FiLLM, a scalable benchmark framework that generates verified financial reasoning tasks from executable computation trees to evaluate and improve LLMs' robustness and accuracy in handling complex, structured financial data.

Original authors: Alicia Larsen, Victoire Laurent, Aulia Kharis Rakhamsari, Lara Turgut, Nino Antulov-Fantulin

Published 2026-08-12

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Alicia Larsen, Victoire Laurent, Aulia Kharis Rakhamsari, Lara Turgut, Nino Antulov-Fantulin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to be a financial analyst. You want it to look at a messy spreadsheet, find the right numbers, and do some math to tell you how much money a company made. This is the world of "Large Language Models" (LLMs)—computer brains that are great at writing stories and answering questions, but sometimes stumble when they have to do precise math or read complex tables. Think of these models like a brilliant student who can write a beautiful essay about money but might accidentally add the wrong numbers if you ask them to calculate a profit margin. The big question researchers are asking is: Can we build a fair test that shows exactly where this student gets confused, without the test itself being messy or unfair?

This is where the paper "V-FiLLM" comes in. The researchers created a special, super-organized test called a "benchmark." Instead of asking the robot to read real, messy bank reports (which might have typos or missing pages), they built a factory that generates perfect, fake financial spreadsheets. They then use a "computation tree"—imagine a family tree, but instead of ancestors, it's a map of math steps—to create questions. Because the math is built into the tree, the answer is guaranteed to be correct by the time the question is even written. This lets them test the robot's brain on pure logic, without the robot getting tripped up by bad handwriting or confusing layouts. They found that as the math gets deeper (more steps), the robot gets much worse at answering, and even tiny changes to the numbers (like turning a "0" into an "O") can make it crash. However, they also showed that if you give the robot a little bit of extra training on how to show its work step-by-step, it gets significantly better.

The Paper Explained

The Problem: The "Messy Desk" vs. The "Clean Lab"
Imagine you are trying to teach a child to do math. If you hand them a crumpled receipt from a grocery store with smudged ink and torn corners, and they get the answer wrong, you don't know if they failed at math or just couldn't read the receipt. This is the problem with existing tests for financial AI. Most previous tests used real documents, which are full of "noise"—missing numbers, weird formatting, and confusing text. This makes it hard to tell if the AI is bad at reasoning or just bad at reading.

The authors of this paper decided to build a "clean lab" instead. They created V-FiLLM, a system that generates its own financial questions from scratch. They start with a perfect, synthetic spreadsheet (like a video game inventory screen) and build a "computation tree" for every question.

The Tree Analogy: Think of a question like a recipe. A simple question is just "What is the price of the apple?" (Depth 0). A harder question is "What is the price of the apple plus the price of the banana?" (Depth 1). The hardest questions are like a complex cake recipe that requires mixing three different bowls, baking them, and then combining the results (Depth 4 or higher).
The Magic: Because the computer builds the tree first, it knows the answer before it even writes the question. This means every single test item has a "ground truth" that is mathematically guaranteed to be correct. There is no human guessing involved, and no "labeling cost" (no one has to sit there and check the answers).

The Discovery: The "Depth" Trap
The researchers used this clean lab to test several different AI models, including some very famous ones like Gemma, Llama, and Qwen. They asked a simple question: What happens when the math gets harder?

They found that the AI's performance drops like a stone as the "depth" of the reasoning increases.

The Drop: On simple questions (just looking up a number), the best models got almost everything right (around 98%). But as soon as the question required 8 steps of reasoning (like a complex cake recipe), the accuracy of some models plummeted to as low as 26%.
The Adversarial Twist: They also tried to "trick" the models by messing with the data in ways that shouldn't matter. For example, they replaced numbers with "garbage" values or changed the units (like swapping dollars for cents). They found that OCR-style character corruption (changing a "0" to a letter "O" or a "1" to an "l") was the most effective way to break the models. When they messed with the units on real-looking documents, the best model's score crashed from nearly 100% down to just 3.0%. This suggests the models are reading the numbers but often ignoring the labels that tell them what those numbers mean.

The Solution: Teaching the AI to "Show Its Work"
The paper also tested a way to fix these problems. They used a technique called LoRA (Low-Rank Adaptation), which is like giving the AI a specific, lightweight training manual instead of retraining its entire brain.

The Method: They took the "computation trees" and turned them into step-by-step "Chain-of-Thought" explanations. They only used the examples where the AI's reasoning was verified to be correct.
The Result: After this targeted training, a smaller model (Qwen3-4B) improved its accuracy from 81.1% to 85.6% on new, unseen problems. It also did better on a different, real-world dataset called FinQA, beating its original version by 5 percentage points. This suggests that teaching the AI to break problems down into small, verified steps is a promising way to make it smarter, without needing massive amounts of computing power.

The Verdict
The paper concludes that while AI is getting better at financial reasoning, it still struggles significantly when the logic gets deep or the data gets messy. The "depth" of the reasoning is the biggest hurdle. However, the new benchmark they built (V-FiLLM) gives researchers a reliable way to measure this progress without the confusion of real-world noise. The authors suggest that with more training on how to show their work step-by-step, these models could become much more reliable financial assistants, but for now, they are still prone to making silly mistakes when the math gets complicated or the numbers look a little weird.

Technical Summary: V-FiLLM: Verified Financial LLM Reasoning Benchmark

Problem Statement

While existing benchmarks have advanced the evaluation of Large Language Models (LLMs) in STEM and general numerical reasoning, financial reasoning over structured tabular data remains under-explored. Current financial QA datasets (e.g., FinQA, TAT-QA) rely heavily on human annotation or model-assisted extraction from real-world documents. This approach introduces significant limitations:

Noise Conflation: It is difficult to distinguish between genuine reasoning failures and errors caused by source-level noise (formatting irregularities, missing values, ambiguous phrasing).
Scalability and Control: Manually annotated benchmarks are costly to scale and lack explicit, controllable difficulty axes, making it hard to isolate specific failure modes such as reasoning depth or compositional complexity.
Ground Truth Reliability: Heuristic extraction or manual labeling can introduce errors into the ground truth, complicating the assessment of model performance.

Methodology

The authors introduce V-FiLLM, a deterministic framework for generating financial reasoning benchmarks from synthetic, executable computation trees grounded in real-table structures. The pipeline ensures that answers are correct by construction, removing the need for manual annotation and eliminating generator error rates.

1. Data Generation

Synthetic Financial Data: The framework constructs two data sources: (a) synthetic 10-Q filings mimicking real company reports, and (b) regularized financial sheets with consistent column names and units to isolate reasoning from formatting noise.
Typed Atoms: Spreadsheet cells are decomposed into "typed financial atoms" containing numerical values and semantic metadata (concept, company, year, unit).
Computation Trees: Questions are generated by sampling typed binary trees where leaves are financial atoms and internal nodes are operations (addition, subtraction, multiplication, ratios, min/max/average).
- Controllable Difficulty: The framework explicitly controls four axes:
  - Depth: The number of reasoning steps (tree depth).
  - Breadth: The number of parallel sub-results required (balanced trees).
  - Concept Complexity: Inclusion of derived financial concepts (e.g., Gross Profit, Net Income) stored as special atoms with hidden executable formulas.
  - Context Size: The number of relevant table entries.
Rendering: Trees are rendered into natural language questions using a bottom-up semantic procedure. To test robustness, the pipeline applies lightweight linguistic data augmentation (typos, capitalization changes, phrasing variations) while preserving the underlying computation graph.

2. Benchmark Extensions

Multi-Turn Evaluation: Complex single-turn problems are decomposed into conversational turns, where each turn resolves one node of the expression tree. This allows for the isolation of intermediate arithmetic errors versus compositional planning failures.
Adversarial Robustness: The framework perturbs input tables (leaving the question/answer unchanged) to simulate real-world pipeline failures:
- Missing values (empty cells).
- Garbage values (outliers like -10^7 or "ERROR").
- OCR look-alikes (digit substitution, e.g., 0→O).
- Cross-sheet contamination (swapping rows between companies).
- Distractor information (irrelevant text appended to questions).

3. Fine-Tuning Strategy

The authors employ Low-Rank Adaptation (LoRA) to fine-tune open-source models (specifically Qwen3.5-4B) on verified Chain-of-Thought (CoT) traces. Only examples where the generated CoT matches the ground-truth value derived from the expression tree are retained for training (88% retention rate).

Key Results

1. Model Performance and Reasoning Depth

Depth Sensitivity: Accuracy declines sharply as reasoning depth increases. On simplified statements, accuracy for models like DeepSeek-v4-Flash drops from ~84% at 6 steps to ~26% at 8 steps. Gemma-31B shows higher resilience but still drops from 85% to 55% over the same range.
Real vs. Simplified: Real 10-Q filings are approximately 20 percentage points harder than simplified statements at depths 3–4, attributed to messy layouts and mixed units.
Model Rankings: Gemma-31B consistently outperforms larger models (e.g., GPT-OSS-120B, Llama-3.3-70B) and matches or exceeds Qwen3.7-Plus, suggesting that specialized training data and table parsing mechanisms are more critical than parameter scale.

2. Multi-Turn Evaluation

Decomposing complex prompts into sequential sub-queries significantly mitigates performance degradation.

Qwen3.5-9B improved from 55.2% (single-turn) to 86.8% (multi-turn).
Llama-3.3-70B improved from 80.8% to 86.8%.
This suggests that query decomposition eases the context load required for single-step reasoning.

3. Adversarial Robustness

General Impact: All models suffer significant accuracy drops under perturbations. Gemma-31B drops by ~27 points on 10-Q filings, and DeepSeek-v4-Flash by ~30 points.
Unit/Scale Shifts: This is the most destructive perturbation. Changing units or scales causes Gemma-31B's accuracy to crash to 3.0% on 10-Q filings, indicating models often read numbers without verifying unit labels.
Combined Noise: When all noise types are combined, accuracy on real documents falls to 17–26%, suggesting that error accumulation severely degrades performance.

4. LoRA Fine-Tuning

Fine-tuning Qwen3.5-4B on verified CoT traces improved accuracy on held-out problems from 81.1% to 85.6%.
The best configuration (Rank $r=8$ , Scaling $\alpha=8$ , Dropout 0.10) showed that the $\alpha/r$ ratio is critical; configurations where $\alpha/r \neq 1$ failed to improve over the baseline.
On the external FinQA dataset, the LoRA-finetuned model improved from 27/100 to 32/100 correct answers, suggesting the gains are not purely due to overfitting.

Significance and Claims

The paper claims to provide a reproducible, annotation-free benchmark that enables systematic evaluation of compositional financial reasoning. Its primary contributions are:

Isolation of Failure Modes: By decoupling reasoning depth, breadth, and domain complexity, V-FiLLM allows researchers to pinpoint exactly where and why models fail (e.g., distinguishing between arithmetic errors and planning failures).
Scalability: The framework generates items at arbitrary scale without annotation costs, addressing the prohibitive cost of scaling existing benchmarks.
Robustness Insights: The study highlights that OCR-style character corruption and unit/scale shifts pose greater challenges than missing data or distractors, emphasizing the need for robust numerical extraction in financial applications.
Targeted Adaptation: The results suggest that lightweight, targeted adaptation (LoRA on verified traces) is a promising direction for improving compositional reasoning in financial QA, though significant headroom remains for deeper reasoning tasks.

The authors maintain a modest stance, acknowledging limitations such as the constrained compute resources (limiting the number of models tested and the size of the fine-tuning evaluation set) and the potential for linguistic ambiguity in the generated questions despite the mathematical correctness of the answers. They position V-FiLLM as a foundational tool for future work in verifier-guided training and cross-document reasoning.