💰 quantitative finance

PyFi: Toward Pyramid-like Financial Image Understanding for VLMs via Adversarial Agents

This paper introduces PyFi, a framework that utilizes a multi-agent adversarial mechanism to synthesize a 600K-scale, pyramid-structured financial dataset, enabling Vision Language Models to achieve significant accuracy improvements in complex financial reasoning by progressively decomposing questions from basic perception to expert-level analysis.

Original authors: Yuqun Zhang, Yuxuan Zhao, Sijia Chen

Published 2026-04-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuqun Zhang, Yuxuan Zhao, Sijia Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to be a financial expert. You show it a chart of a country's economy and ask, "What should we do to fix this?"

Most robots today are like smart but impatient tourists. They can read the signs on the map (the chart), but if you ask them to navigate a complex city with traffic jams and detours (a complex financial decision), they often get lost, guess wildly, or hallucinate a solution that sounds good but is wrong.

This paper introduces PyFi, a new way to train these robots so they don't just guess—they think.

Here is the breakdown of how it works, using simple analogies:

1. The Problem: The "Leap" is Too Big

Currently, if you ask a robot a hard financial question, it tries to jump straight to the answer. It's like asking a child to solve a calculus problem without teaching them how to add or subtract first. They might get lucky once, but they will fail most of the time because they lack the step-by-step foundation.

2. The Solution: The "Financial Pyramid"

The authors built a massive training dataset called PyFi-600K. Think of this not as a pile of random questions, but as a 6-story pyramid.

Level 1 (The Base): "What color is this line?" (Basic perception).
Level 2: "What number is written there?" (Data extraction).
Level 3: "If you subtract these two numbers, what do you get?" (Calculation).
Level 4: "Does this line go up or down over time?" (Pattern recognition).
Level 5: "Why did it go down? What does that mean?" (Logical reasoning).
Level 6 (The Top): "Based on all that, what policy should the government adopt?" (Decision making).

The magic is that the robot cannot answer Level 6 until it has correctly answered Levels 1 through 5. It forces the robot to build a ladder of logic before it can climb to the top.

3. The Teacher: The "Adversarial Game Show"

How did they create 600,000 of these pyramid questions without hiring thousands of human experts? They invented PyFi-adv, a multi-agent system that acts like a Game Show.

Imagine two AI agents playing a game:

The Solver: Tries to answer a question.
The Challenger: Tries to trick the Solver.

Here's the loop:

The Challenger asks a simple question (Level 1).
The Solver answers it.
If the Solver gets it right, the Challenger gets angry and asks a harder follow-up question (Level 2) based on that answer.
They keep playing, getting harder and harder, until they reach a complex decision (Level 6).
If the Solver fails, the game restarts. If they succeed, the whole chain of questions is saved as a "perfect lesson."

It's like a sparring match where the opponent keeps raising the stakes until the fighter proves they are a master. This happens automatically, creating a massive library of "step-by-step" financial reasoning.

4. The Results: From "Guessing" to "Reasoning"

The authors tested this on 15 different AI models.

Before training: When faced with a Level 6 question, the robots were like a student who memorized the answer key but didn't understand the math. They got about 33% correct.
After training: When they were taught to break the problem down into the pyramid steps (using the "Question Chains"), their accuracy jumped significantly. One small model improved by nearly 20%.

5. Why This Matters: The "Black Box" is Open

The best part of PyFi is transparency.

Old Way: The robot gives an answer. If it's wrong, you have no idea why. Did it misread the chart? Did it do the math wrong? Did it misunderstand the logic?
PyFi Way: Because the robot is forced to answer the small steps first, we can trace its path. If it fails the final decision, we can look back and say, "Ah, it failed at Level 3 (Calculation)." It's like having a GPS that tells you exactly where you took a wrong turn, rather than just saying "You are lost."

Summary

PyFi is a new training method that teaches financial AI to stop guessing and start climbing a ladder. Instead of jumping to the conclusion, it learns to:

See the data.
Extract the numbers.
Do the math.
Spot the trends.
Understand the logic.
Make the decision.

By using a "game show" of AI agents to generate these lessons automatically, they created a massive, high-quality textbook that makes financial AI smarter, more reliable, and easier to trust.

1. Problem Statement

Vision Language Models (VLMs) have shown promise in general domains but struggle significantly in the financial domain, particularly when interpreting complex financial images (charts, graphs, tables) to perform multi-step reasoning and decision-making.

Data Scarcity & Cost: Existing financial benchmarks rely heavily on manual annotation by domain experts, which is prohibitively expensive, time-consuming, and difficult to scale.
Quality & Reliability: Automated methods that simply prompt LLMs to generate questions often result in factual errors, hallucinations, or a lack of deep domain expertise.
Evaluation Limitations: Current benchmarks typically provide a single scalar score for a final answer, failing to diagnose where in the reasoning chain a model fails (e.g., did it fail at data extraction, calculation, or logical inference?).
Reasoning Gap: Financial decision-making requires a progression from basic visual perception to complex strategic decisions, a capability current VLMs lack.

2. Methodology

The authors propose PyFi, a comprehensive framework consisting of a novel dataset (PyFi-600K) and a scalable data synthesis mechanism (PyFi-adv).

A. The Pyramid-Structured Dataset (PyFi-600K)

Instead of flat question-answer pairs, the dataset is organized into a 6-level cognitive pyramid:

Perception (PP): Basic visual recognition (e.g., identifying chart axes, colors).
Data Extraction (DE): Converting visual elements into quantitative data.
Calculation Analysis (CA): Performing arithmetic operations on extracted data.
Pattern Recognition (PR): Identifying trends, cycles, or correlations.
Logical Reasoning (LR): Inferring causal relationships or implications.
Decision Support (DS): Formulating strategic financial recommendations.

Structure: The dataset contains 603,859 samples organized into 62,660 question chains. Each chain starts with simple Level 1 questions and progressively builds up to a Level 6 decision, demonstrating how lower-level insights support higher-level reasoning.
Coverage: It spans 17 financial themes (e.g., Macroeconomics, Risk Management) and 11 image types (e.g., Line charts, Candlestick charts).

B. Adversarial Data Synthesis (PyFi-adv)

To generate this massive dataset without human annotation, the authors introduce a Multi-Agent Adversarial Mechanism operating under a Monte Carlo Tree Search (MCTS) paradigm:

Agents:
- Challenger Agent ( $\psi$ ): Generates increasingly difficult follow-up questions to probe the limits of the solver.
- Solver Agent ( $\phi$ ): Attempts to answer the questions using financial knowledge and visual reasoning.
Process:
1. The Challenger starts with a simple question (Level 1).
2. The Solver answers.
3. Based on the answer, the Challenger formulates a harder question (Level 2, then 3, etc.), creating a "question chain."
4. This continues until a Level 6 decision is reached or a depth limit is hit.
5. Backpropagation: The correctness of the final decision is backpropagated to assign reward scores to every step in the chain, enabling Process Reward Modeling (PRM).
Diversity: The system uses Bernoulli sampling and UCT (Upper Confidence Bound applied to Trees) to balance exploration (new question paths) and exploitation (refining existing paths), preventing homogenization.

3. Key Contributions

PyFi-600K Dataset: The first large-scale, automatically synthesized financial image dataset with hierarchical, step-wise annotations. It enables fine-grained evaluation of VLM capabilities from perception to decision-making.
PyFi-adv Framework: A novel adversarial multi-agent MCTS approach for synthesizing high-quality, domain-specific reasoning chains without human intervention. This solves the scalability and cost issues of manual annotation.
Process Supervision: By assigning reward scores to individual reasoning steps, the framework supports the training of Process Reward Models (PRMs), allowing for error tracing and verification of intermediate steps.
Comprehensive Evaluation & Insights: The paper provides a granular analysis of 15 state-of-the-art VLMs, revealing specific bottlenecks (e.g., Calculation Analysis) and demonstrating that Chain-of-Thought (CoT) fine-tuning on question chains significantly boosts performance.

4. Experimental Results

The authors evaluated 15 pre-trained VLMs (including GPT-4.1, Claude-opus, GLM-4.5V, and various Qwen models) and fine-tuned Qwen2.5-VL models on PyFi-600K.

Baseline Performance:
- There is a sharp decline in accuracy as capability levels increase. Average accuracy drops from 71.80% at Level 1 (Perception) to 32.95% at Level 6 (Decision Support).
- Even top-tier models like GLM-4.5V struggle with Level 6 tasks (46.15% accuracy).
- Key Bottleneck: Calculation Analysis (CA) was identified as the primary failure point for most models, causing downstream errors in final decisions.
Fine-Tuning Results:
- Fine-tuning Qwen2.5-VL-3B and Qwen2.5-VL-7B on the question chains (specifically with CoT) yielded significant improvements.
- Qwen2.5-VL-3B: Improved by 19.52% (reaching ~40% accuracy on the filtered test set).
- Qwen2.5-VL-7B: Improved by 8.06%.
- The smaller 3B model, when trained with CoT, even surpassed the accuracy of the much larger 72B model in specific reasoning tasks (Pattern Recognition and Logical Reasoning).
Error Tracing: The pyramid structure allowed the authors to trace errors backward. For instance, a failure in a Level 6 decision was often traced back to an error in Level 3 (Calculation) or Level 2 (Data Extraction), proving the utility of the chain structure for interpretability.

5. Significance

Scalability in Finance: PyFi-adv demonstrates that high-quality, expert-level financial datasets can be generated automatically, overcoming the bottleneck of manual expert annotation.
Interpretability: The framework shifts the focus from "black-box" final answers to traceable reasoning paths. This is crucial for financial applications where explainability and auditability are mandatory.
Training Paradigm: The results suggest that decomposing complex financial problems into a hierarchy of sub-questions is a highly effective training strategy. It allows smaller models to learn complex reasoning by mastering foundational steps first.
Future Directions: The dataset and method pave the way for training robust Process Reward Models (PRMs) specifically for financial visual reasoning, potentially leading to more reliable AI agents for financial analysis and decision support.

In summary, PyFi addresses the critical gap in financial VLMs by providing a scalable, hierarchical dataset and a novel adversarial synthesis method, proving that structured, step-by-step reasoning training is essential for mastering complex financial image understanding.