🤖 AI

AgentFinVQA: A Deployable Multi-Agent Pipeline for Auditable Financial Chart QA

AgentFinVQA is a deployable, multi-agent pipeline for auditable financial chart question answering that achieves state-of-the-art accuracy on the FinMME benchmark while ensuring data residency and providing traceable verification for regulated environments.

Original authors: Aravind Narayanan, Shaina Raza

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aravind Narayanan, Shaina Raza

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a financial analyst trying to read a complex chart to answer a question like, "Which years had sales between $10k and $25k?"

If you ask a standard AI (a "zero-shot" model) to do this, it's like asking a brilliant but impatient intern to glance at the chart and shout out an answer. They might get it right, but they might also guess, hallucinate numbers, or miss a tiny detail. Worse, you have no idea how they got the answer, and if they are wrong, you can't prove it. In the financial world, where rules are strict and data is sensitive, you can't just trust a "black box" answer. You need to know the steps, and you can't send your private client data to a stranger's computer.

AgentFinVQA is the solution the authors built. Think of it not as a single genius, but as a specialized factory assembly line for answering chart questions. Instead of one person doing everything, the work is broken down into a team of specialists, each with a specific job, and every single step is written down in a permanent "receipt" (called a Model Evaluation Packet) so you can audit exactly what happened.

Here is how their "factory" works, step-by-step:

The Planner (The Architect): Before anyone looks at the chart, this text-only agent reads the question and the multiple-choice options. It doesn't see the image yet. It creates a checklist: "Okay, we need to check the colors, look at the years, and compare the bars." It tells the team exactly what to look for.
The OCR Reader (The Scribe): This agent looks at the chart and reads only the text (titles, axis labels, numbers). It writes this down in a neat list. This prevents the next steps from misreading a blurry label.
The Legend Grounder (The Translator): Charts often use colors (red line, blue bar) to represent different things. This agent matches the colors to their names (e.g., "Red = 2023 Sales"). It creates a map so no one confuses the colors later.
The Colour-Area Tool (The Measuring Tape): For tricky charts like pie charts or stacked bars, guessing sizes by eye is hard. This isn't a smart AI; it's a simple, dumb calculator. It literally counts the pixels of a specific color to give an exact measurement. It's like using a ruler instead of guessing with your eyes.
The Vision Agent (The Inspector): Now, the main agent looks at the chart with the checklist, the text list, the color map, and the pixel measurements. It puts it all together to draft an answer.
The Verifier (The Quality Control Manager): This is the most important step for trust. A second, independent agent looks at the draft answer and the chart again. It asks, "Does this actually match the data?"
- If it agrees, it says "Confirmed."
- If it disagrees, it says "Revise."
- Crucially, the system records a "confidence score." If the manager is unsure, it flags the answer for a human to check. This lets companies focus their human reviewers only on the answers that are likely to be wrong, saving time.

Why is this a big deal?

It's Auditable: Because every step is recorded in that "receipt" (the MEP), you can look back and say, "Ah, the system failed because it confused the red and blue lines." You know exactly why.
It's Private: You can run this entire factory on your own computers (on-premise) using open-source software. You never have to send your secret client charts to a big tech company's cloud.
It's Accurate: The authors tested this on a tough financial chart dataset called FinMME.
- When using a top-tier commercial AI (Gemini-3), their factory got 7.68% more correct answers than just asking the AI directly.
- When using a free, open-source AI (Qwen) running on their own servers, they still got 4.84% more correct answers.
- The "Quality Control Manager" (Verifier) was very good at spotting errors: when it said "Confirmed," the answer was right 68.2% of the time. When it said "Revise," the original answer was often wrong.

What didn't work perfectly?

The authors admit the system isn't magic. About two-thirds of the mistakes happened because:

The AI misunderstood the question (e.g., thinking "highest" meant "lowest").
It got confused by the legend (mixing up which color meant which year).
It read a number wrong even though it found the right spot on the chart.

Interestingly, the "Quality Control Manager" wasn't very good at spotting these specific types of mistakes. This tells the authors exactly what they need to fix next.

In short: AgentFinVQA turns a risky, opaque AI guess into a transparent, step-by-step factory process. It proves you can have high accuracy, full privacy, and a clear paper trail all at once, which is exactly what financial institutions need to trust AI with their data.

Technical Summary: AgentFinVQA

Problem Statement

Financial chart question answering (QA) in regulated environments faces a triad of challenges that existing solutions fail to address simultaneously:

Auditability: Practitioners require visibility into reasoning to trust answers, yet current agents are often opaque "black boxes."
Data Residency: Many financial institutions cannot transmit sensitive client data to external API providers, necessitating on-premise deployment.
Accuracy vs. Transparency: Existing systems either prioritize raw accuracy using proprietary APIs (lacking audit trails) or rely on local fine-tuning and segmentation models that are difficult to deploy and audit.

Current agentic frameworks often rely on local computer-vision tools to manipulate images or require heavy fine-tuning on annotated datasets, creating barriers to entry for regulated sectors. There is a gap for a system that is prompting-only (no local segmentation or task-specific fine-tuning), fully auditable, and deployable on open-weight models without significant accuracy loss.

Methodology: The AgentFinVQA Pipeline

The authors propose AgentFinVQA, a multi-agent pipeline that decomposes chart QA into sequential, traceable stages. The system operates on a financial chart image ( $c$ ), a natural language question ( $q$ ), and optional multiple-choice options ( $O$ ).

Core Architecture

The pipeline executes stages in a fixed order: Plan $\rightarrow$ OCR $\rightarrow$ Ground $\rightarrow$ Colour-Area $\rightarrow$ Inspect $\rightarrow$ Verify. Each stage writes inputs, outputs, and tool traces to a Model Evaluation Packet (MEP), a portable JSON artifact enabling full auditability and post-hoc error attribution.

Planner: A text-only LLM analyzes the question and MCQ options (without seeing the image) to generate a structured JSON inspection plan. It classifies the question type and instructs the subsequent vision agent on how to handle single-select vs. multi-select queries.
OCR Reader: A focused Vision Language Model (VLM) call transcribes all visible text (axis labels, ticks, legends, annotations) into structured metadata. This prevents the vision agent from misreading text later.
Legend Grounder: A targeted VLM call maps legend entries to visual properties (color descriptions, approximate RGB, line style). This map is injected into the vision prompt as explicit evidence, with a compliance check ensuring the vision agent references these labels.
Colour-Area Tool: A deterministic, rule-based pixel-counting stage (using OpenCV) that activates for bar/pie/donut charts with comparison keywords. It applies HSV color masks based on the legend map to count pixels per series, providing a "dominant-label hint" to the vision agent. This stage requires no GPU or trained models.
Vision Agent: An orchestrated agent (using CrewAI) executes the inspection plan using the chart image, OCR metadata, legend map, and color-area hints. It produces a draft answer and per-choice confidence scores. A "forced-choice retry" mechanism handles MCQ over-refusal.
Verifier: An independent VLM call audits the draft answer against the chart image. It issues a CONFIRM or REVISE verdict with a self-reported confidence score. A confidence gate (threshold 0.75) prevents low-confidence revisions from overriding high-quality vision outputs.

Deployment Flexibility

The system uses a configurable backend abstraction. It supports proprietary APIs (e.g., Gemini-3 Flash) and locally served open-weight models (e.g., Qwen3.6-27B-FP8 via vLLM). This allows institutions to run the entire pipeline on-premise on a single A100 GPU, ensuring data never leaves their infrastructure.

Key Contributions

AgentFinVQA Pipeline: A prompting-only, multi-agent framework that decomposes chart QA into specialized steps, recording every action in a traceable MEP for full auditability.
Open-Weight Viability: Empirical evidence that the accuracy gains of agentic decomposition transfer from proprietary backbones to locally served open-weight models, supporting in-house deployment with a modest accuracy trade-off.
Confidence-Based Routing: The verifier's verdict serves as a practical confidence signal. "Revised" answers are flagged for human review, while "Confirmed" answers are treated as lower priority, optimizing the human-in-the-loop workflow.

Results

Evaluated on the FinMME benchmark (~11,000 samples), AgentFinVQA demonstrates significant improvements over model-matched zero-shot baselines:

Proprietary Backbone (Gemini-3 Flash):
- Mean Accuracy: 71.24% vs. 63.56% (Zero-shot).
- Improvement: +7.68 percentage points (pp).
- MCQ Accuracy: +8.1 pp.
- Statistical Significance: McNemar $p \approx 1.1 \times 10^{-16}$ .
Open-Weight Backbone (Qwen3.6-27B-FP8, Local):
- Mean Accuracy: 66.52% vs. 61.68% (Zero-shot).
- Improvement: +4.84 pp.
- MCQ Accuracy: +5.3 pp.
- Statistical Significance: McNemar $p \approx 3.0 \times 10^{-6}$ .
- Note: The open-weight model showed no reliable gain on open-ended questions (-1.0 pp), suggesting a modest accuracy cost for the hardest question types in on-premise settings.

Error Analysis & Verifier Utility

Failure Modes: Approximately two-thirds of failures stem from question misunderstanding, legend confusion, and extraction errors.
Verifier Performance: The verifier effectively identifies errors in some categories but is least effective at detecting question misunderstandings and legend confusion.
Routing Signal: Answers marked as "Confirmed" by the verifier achieved 68.2% exact accuracy, compared to 55.6% for "Revised" answers. This suggests the verifier can successfully route the ~19% of outputs most likely to be wrong to human analysts, while the remaining ~81% can be trusted with higher confidence.

Significance and Claims

The paper claims that auditable, on-premise financial chart QA is practical. It demonstrates that:

Agentic decomposition improves accuracy significantly over zero-shot baselines without requiring local computer-vision models or task-specific fine-tuning.
These gains are not dependent on proprietary APIs; they persist with open-weight models served locally, enabling full data residency.
The integration of a verifier provides a mechanism for "confidence-based human review routing," offering a pragmatic trade-off between automation and auditability suitable for regulated financial settings.

The authors acknowledge limitations, including the lack of cross-dataset generalization testing (only FinMME used), the modest accuracy cost on open-ended questions for open-weight models, and the fact that the verifier's confidence is self-reported rather than calibrated. They note that the colour-area tool's primary value is architectural (deterministic grounding and audit trace) rather than aggregate accuracy lift, as it activates on only 5% of the dataset.