💬 NLP

Self-Ensembling Vision-Language Models for Chart Data Extraction

This paper proposes a self-ensembling vision-language model method that aggregates multiple sampled tabular outputs to improve chart-to-table data extraction accuracy and reliability, validated by a new complex benchmark (WB-ChartExtract) showing significant performance gains over single-pass approaches.

Original authors: Thomas Berkane, Qianyi Wang, Maimuna S. Majumder

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Thomas Berkane, Qianyi Wang, Maimuna S. Majumder

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: Charts are "Locked" Images

Imagine you have a beautiful, colorful chart in a PDF report or on a website. It shows sales numbers, temperature changes, or population growth. To a human, it's easy to glance at and understand the trend. But to a computer, that chart is just a picture (a "rasterized image"). The actual numbers are trapped inside the pixels.

If you want to use those numbers to do math, compare them with other data, or build a new model, you can't just copy-paste them. You have to manually type them out, which is slow, boring, and full of typos.

The Current Solution (and its Flaw)

Recently, we've built "Vision-Language Models" (VLMs)—AI brains that can look at a picture and "read" it. You can ask an AI, "What are the numbers in this chart?" and it will try to write them down in a table.

However, these AIs are a bit like a nervous student taking a test. If you ask the same student the same question twice, they might give you two slightly different answers.

Run 1: "The sales were 100, 105, and 102."
Run 2: "The sales were 100, 104, and 103."

Sometimes the AI misses a number entirely; other times, it invents a number that isn't there. Current methods usually just take the first answer the AI gives and hope for the best. This is risky because that single guess might be wrong.

The New Idea: The "Committee of Experts"

The authors of this paper propose a clever fix: Don't trust one guess; trust the average of many guesses.

They created a method called Self-Ensembling. Think of it like this:

The Poll: Instead of asking the AI once, you ask it the same chart question 20 times.
The Crowd: You get 20 different tables back. Some have 100, some have 101, some have 99.
The Consensus: The system lines up all 20 tables. For every single number in the table, it looks at all 20 versions and picks the median (the middle value).
- If 19 people say "100" and 1 person says "1000" (a mistake), the system ignores the outlier and settles on 100.
- If the AI is confused and gives different numbers, the "middle ground" is usually much more accurate than any single guess.

Bonus Features: Knowing When to Stop and How Sure You Are

The paper adds two smart tools to this process:

The "Stop Sign" (Convergence Detection): Asking the AI 20 times costs time and money. The system checks after every few guesses: "Are the answers stabilizing?" If the last few guesses are all basically the same, the system says, "Okay, we have a solid answer, let's stop." This saves money on easy charts.
The "Confidence Meter" (Uncertainty Estimation): If the AI's 20 guesses are all very close to each other, the system knows it's confident. If the guesses are all over the place (some say 50, some say 90), the system flags that specific number as "Unreliable." This helps users know which parts of the data they can trust and which parts need a human to double-check.

The New Test: "The Hard Mode"

The authors realized that the standard tests used to check these AIs (like ChartQA) were too easy. They were like a driving test on an empty parking lot. The charts were simple, and the numbers were often printed right on the bars, making it easy for the AI to just "read" the text rather than actually understanding the chart.

To fix this, they built a new, much harder test called WB-ChartExtract.

Source: They used real data from the World Bank.
Difficulty: These charts are crowded, complex, and have no numbers printed on them. The AI has to actually measure the height of a bar or the position of a line to guess the number.
Variety: They used four different chart types and four different software tools to draw them, ensuring the AI can't just memorize one style.
Scale: On average, these charts have 7 times more data points than the old tests.

The Results

When they ran their "Committee of Experts" method on both the easy tests and the new hard tests:

It worked everywhere: The method improved accuracy for almost every AI model they tried.
Big gains on hard charts: On the difficult new test, the method improved accuracy by up to 23% compared to just asking the AI once.
Cost vs. Reward: While asking the AI more times costs a bit more, the system stops early for easy charts, keeping the total cost very low (often under $15 for a whole dataset).

Summary

The paper says: "AI is great at reading charts, but it's inconsistent. If you ask it the same question many times and take the middle answer, you get a much more accurate result. We also built a harder test to prove this works on real-world, messy data, and we added a 'confidence meter' so you know when to trust the AI."

Important Note: The authors explicitly state that their method is limited by the AI's own mistakes. If the AI consistently misunderstands a specific type of chart, asking it 20 times won't fix that. Also, their new test is synthetic (computer-generated), so while it mimics real data, it might not catch every weird artifact found in a scanned, low-quality document.

Technical Summary: Self-Ensembling Vision-Language Models for Chart Data Extraction

Problem Statement
Charts are a primary medium for communicating quantitative information in scientific and policy contexts, yet the underlying data is often locked in rasterized image formats, rendering it inaccessible for search, joining, or downstream analysis. While manual digitization is possible, it is slow and error-prone. Automatic chart-to-table extraction has emerged as a solution, with recent approaches leveraging Vision-Language Models (VLMs) to directly "read" charts and emit textual table representations. However, existing VLM-based methods exhibit brittleness: a single forward pass often yields inconsistent results, such as dropping datapoints, hallucinating values, or misreading axes. Furthermore, current benchmarks (e.g., ChartQA) often feature simple plots with limited stylistic variation and printed value labels, leading to saturated performance metrics that do not reflect the difficulty of real-world, complex figures.

Methodology
The authors propose a model-agnostic self-ensembling method that exploits the stochasticity of VLMs to improve extraction accuracy and reliability. The core workflow involves:

Iterative Sampling: For a given chart image, the base VLM is queried repeatedly (at a specific sampling temperature) to generate multiple candidate tabular outputs in TSV format.
Parsing and Normalization: Raw outputs are parsed into TSV, with rows/columns repaired if necessary. Cells are converted to numeric values, stripping artifacts like currency symbols, while non-numeric strings are treated as missing.
Row and Column Alignment: Since label spelling and ordering may vary across samples, the method clusters row and column labels independently using Average Normalized Levenshtein Similarity (ANLS). Labels are grouped into canonical clusters if they meet a similarity threshold and do not originate from the same sampled table (avoiding intra-table duplicates). Spurious clusters appearing in fewer than 20% of tables are pruned.
Cell-Wise Aggregation: For each aligned cell (row, column), the method collects all numeric predictions from the sampled tables. A consensus value is computed as the median of these values.
Convergence Detection: The sampling process includes an early-stopping mechanism. Sampling halts if, for a set number of consecutive updates, at least 95% of aggregated cell values change by no more than a 1% relative tolerance. This balances computational cost with marginal accuracy gains.
Uncertainty Estimation: To assess reliability, the method calculates the Relative Median Absolute Deviation (MAD) for each cell. High dispersion across samples indicates low extraction confidence, providing a signal for users to verify results or request more samples.

Key Contributions

Self-Ensembling Framework: A model-agnostic approach that aggregates multiple VLM outputs at the cell level to produce a more accurate consensus table, suppressing outlier predictions and hallucinations.
Convergence Detection: A mechanism to identify when additional samples are unlikely to alter the aggregated result, enabling practical early stopping.
Uncertainty Quantification: An ensemble-based uncertainty estimate (Relative MAD) that correlates inversely with extraction accuracy, helping users gauge reliability.
WB-ChartExtract Benchmark: A new benchmark constructed from World Bank time-series data, featuring 1,000 synthetic charts. Unlike existing datasets, it includes:
- Higher Complexity: Charts contain on average 7× more datapoints than those in ChartQA.
- Stylistic Diversity: Charts are rendered using four libraries (Matplotlib, Seaborn, Plotly, Bokeh) across four chart types (line, area, grouped bar, stacked bar) with randomized fonts, colors, and layouts.
- No Printed Labels: Values are not printed directly on the chart, forcing models to infer data from axes and geometry rather than relying on OCR shortcuts.

Results
The method was evaluated on both the standard ChartQA and the new WB-ChartExtract benchmarks across various base models, including specialized chart extractors (OneChart, TinyChart, DePlot) and general-purpose open-source and closed-source VLMs (Qwen3-VL, Llama 4 Scout, Seed 1.6 Flash, GPT-5.1, Claude Opus 4.6, Gemini 2.5 Pro).

Accuracy Improvements: Self-ensembling consistently improved extraction accuracy (measured by Relative Mapping Similarity, RMSF1) over single-pass baselines.
- On WB-ChartExtract, relative improvements ranged from 2.7% (TinyChart) to 23.1% (Seed 1.6 Flash).
- On ChartQA, improvements were observed for general-purpose models (e.g., Qwen3-VL improved from 91.43 to 93.21), though gains were minimal for TinyChart, which was already saturated on its in-domain training distribution.
Error Analysis: The primary failure modes corrected by ensembling were Value Errors (numeric mismatches) and Missing datapoints. Label errors were negligible. The method was particularly effective at recovering missing structure in complex charts (e.g., area and line charts).
Cost-Efficiency: While ensembling increases API costs (roughly 4–16× depending on the model), the convergence detection mechanism significantly mitigates this by stopping early for easier samples. On WB-ChartExtract, early stopping retained 99% of the ensemble's accuracy boost while using 1.24× fewer samples on average.
Uncertainty Correlation: The proposed uncertainty metric (Relative MAD) showed a significant negative correlation with RMSF1 (Spearman $\rho \approx -0.34$ to $-0.37$), confirming that higher ensemble disagreement reliably predicts lower extraction accuracy.

Significance and Claims
The paper claims that this approach moves chart-to-table extraction closer to real-world deployment by addressing the brittleness of single-pass VLM generation and the limitations of current benchmarks. By introducing WB-ChartExtract, the authors provide a more rigorous testbed that exposes genuine differences in visual reasoning capabilities across models, particularly regarding data density and stylistic variation. The self-ensembling method offers a practical, model-agnostic strategy to unlock tabular data previously siloed in chart images, enabling downstream analysis with quantified reliability.

Limitations
The authors acknowledge that the method is bounded by the failure modes of the base VLM; systematic errors may be reinforced rather than corrected. The convergence criterion focuses on numeric stability and may miss semantic inconsistencies in table structure. Additionally, the uncertainty estimate is heuristic and not a calibrated probability of correctness. Finally, while WB-ChartExtract is diverse, it is synthetically rendered and may not capture all real-world artifacts like low-resolution scans or non-standard layouts.