💬 NLP

EpiCurveBench: Evaluating VLMs on Epidemic Curve Digitization

This paper introduces EpiCurveBench, a benchmark of 1,000 real-world epidemic curve images, and EpiCurveSimilarity (ECS), a novel evaluation metric that accounts for temporal structure and local shifts, to demonstrate that current vision-language models struggle with epidemic curve digitization and that ECS provides a more discriminative and epidemiologically relevant assessment than existing key-value metrics.

Original authors: Thomas Berkane, Maimuna S. Majumder

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Thomas Berkane, Maimuna S. Majumder

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a library of old, dusty public health reports. Inside these reports are colorful charts (called "epicurves") that show how many people got sick, were hospitalized, or died during various outbreaks over the last 175 years. The problem? These charts are just pictures. To use this data to predict future outbreaks, computers need the numbers written out in a list, not hidden inside an image.

This paper introduces a new way to test if Artificial Intelligence (AI) can read these charts and turn them into lists of numbers accurately. Here is the breakdown in simple terms:

1. The Problem: The AI is "Blind" to Time

Previously, researchers tested AI on chart-reading using simple, clean, computer-generated charts. The AI was doing great on those tests (scoring over 89%). But those tests were like a driving test on an empty, straight highway. Real-world epidemic charts are like driving through a chaotic, rainy city with blurry signs, overlapping lines, and tiny text.

The old way of grading the AI was like a teacher checking a student's homework by looking at the answers in random order.

The Flaw: If the AI got the numbers right but wrote them down one day late (a "temporal shift"), the old grading system gave it a zero. It treated a "right answer, wrong day" the same as a "completely wrong answer." This is unfair because, in real life, knowing the trend is right is often more important than being off by a single day.

2. The Solution: A New Test and a New Grader

The authors created two things to fix this:

A. The Test: EpiCurveBench
They gathered 1,000 real-world epidemic charts from public health agencies around the world. These aren't clean, perfect images. They are messy:

Some are scanned from old, blurry paper.
Some have lines that cross over each other.
Some have text that is rotated or tiny.
Some have hundreds of data points (like a dense forest of dots).
They cover diseases from 1849 to 2025.

B. The Grader: EpiCurveSimilarity (ECS)
They invented a new scoring system called ECS. Think of it like a rubber ruler instead of a rigid metal one.

If the AI gets the shape of the curve right but is slightly shifted in time, the rubber ruler stretches to match it and gives partial credit.
If the AI misses a chunk of the data (like cutting off the end of the chart), the ruler shrinks and penalizes the score heavily.
This metric is designed to care about the story the data tells (the trend), not just if every single number is in the exact right box.

3. The Results: The AI Still Has a Long Way to Go

The authors tested six different AI models (three famous "closed" ones, one open-source one, and two specialized chart-readers) on this new, hard test.

The Score: Even the best AI model only scored 52.3% on this new test. This means that even the smartest AI currently available is still making significant mistakes when trying to digitize real-world public health data.
The Specialized vs. General: The models built specifically for charts (like "OneChart") actually did worse than the general-purpose AI models. They got confused by the messy, real-world styles.
The "Reasoning" Trap: Telling the AI to "think harder" or use a calculator tool didn't always help. Sometimes, it made the AI crop the image wrong or get confused, lowering its score.

4. Why This New Grader Matters

The paper proves that the old grading system (RMS) was hiding the truth.

The Old Grader: Put all the AI models in a tiny, crowded room where they all looked about the same (scores within a 5-point range). It couldn't tell who was actually better.
The New Grader (ECS): Spread the models out into a large field (a 25-point range), clearly showing which ones were better at understanding the shape and timing of the data.

Most importantly, the authors checked if a higher ECS score actually meant better results for real-world math. They found that yes, it did.

If an AI had a high ECS score, the total number of cases it calculated was closer to the truth.
It was better at predicting when the peak of the outbreak happened.
It was better at predicting how fast the disease was spreading.

The Bottom Line

We have a treasure trove of historical disease data trapped in pictures. We have AI that is getting better at reading them, but it's not ready for prime time yet. The authors built a harder test and a fairer grading system that shows us exactly where the AI is failing (like missing data points or getting numbers slightly wrong) so engineers can fix it. Until the AI can score higher on this new test, we can't fully trust it to unlock decades of vital health data.

Technical Summary: EpiCurveBench and EpiCurveSimilarity

Problem Statement

Vision-language models (VLMs) are increasingly deployed to extract structured numerical data from chart images. However, current evaluation frameworks are insufficient for real-world, dense time-series charts, particularly epidemic curves ("epicurves"). Existing benchmarks like ChartQA rely on simple, synthetic infographics where frontier VLMs already exceed 89% accuracy, offering diminishing headroom for progress. Furthermore, standard metrics such as Relative Mapping Similarity (RMS) and SCRM treat extracted data as unordered key-value pairs. This approach ignores the temporal structure of time series and catastrophically penalizes minor temporal shifts or header mismatches, conflating value-correct-but-shifted extractions with completely wrong ones. Consequently, the community lacks a rigorous stress-test benchmark for dense time-series extraction and a metric that reflects the downstream utility of extracted data for epidemiological analysis.

Methodology

1. EpiCurveBench: A Real-World Benchmark

The authors introduce EpiCurveBench, a dataset of 1,000 real-world, openly licensed epidemic curve images spanning 14 diseases, 37 countries, and the years 1849–2025. The dataset is designed to be significantly more challenging than existing benchmarks, featuring dense datapoints (median 156 per chart, up to 8,592), overlapping/stacked series, rotated text, low-resolution scans, and thin bars.

The benchmark consists of two sets:

Set 1 (900 charts): Curated from sources that provide machine-readable tabular ground truth (e.g., CDC reports). A 20% random sample was audited by human annotators, confirming 100% alignment between images and source tables.
Set 2 (100 charts): Manually annotated using WebPlotDigitizer to ensure stylistic diversity and test generalization where no tabular data exists. These were curated via a three-stage protocol involving visual dissimilarity selection, DINOv2 embedding clustering to remove duplicates, and manual verification.

2. EpiCurveSimilarity (ECS): A Temporal-Aware Metric

To address the limitations of key-value metrics, the authors propose EpiCurveSimilarity (ECS), an evaluation metric based on Edit Distance with Real Penalty (ERP). ECS aligns predicted and ground-truth series via dynamic programming and incorporates three key extensions:

Label-Similarity Matching: Uses Normalized Levenshtein Similarity (NLS) to match series based on label text, accommodating multi-series charts where order or naming may differ.
Per-Series Y-Axis Normalization: Substitution penalties are normalized by the specific y-axis range of each series, ensuring comparability across different scales.
Tolerance Threshold and Gap Penalty: The metric tolerates local temporal shifts and gaps (insertions/deletions) but penalizes them proportionally. A gap penalty ( $\lambda = 1$ ) is applied for unmatched points, and substitution costs are clipped if they exceed a threshold ( $\theta = 0.01$ ) of the y-axis range.

The final ECS score for a series is $1 - \frac{C(M, N)}{\text{#matches} + \text{#gaps}}$ , where $C(M, N)$ is the alignment cost.

3. Experimental Setup

The authors evaluated six methods on EpiCurveBench:

Three Frontier Closed VLMs: GPT-5.2, Claude Opus 4.5, and Gemini 2.5 Pro.
One Open VLM: Qwen3-VL-235B-A22B-Instruct.
Two Specialized Systems: OneChart and TinyChart.

Models were tested under three configurations: minimal reasoning, high reasoning, and high reasoning with Python code execution (for image cropping/zooming).

Key Results

Model Performance

Even the strongest model, Gemini 2.5 Pro (high reasoning effort), achieved only 52.3% ECS on the full benchmark. Specialized chart-extraction systems performed poorly (OneChart: 9.4%, TinyChart: 11.4%), likely due to distribution shifts from synthetic training data and lower model capacity. Among general-purpose VLMs, Gemini 2.5 Pro led, followed by GPT-5.2 and Claude Opus 4.5, with Qwen3-VL (open-weight) achieving 27.5% at high effort.

Metric Discrimination

Key-Value Metrics (RMS/SCRM): Compressed the four general-purpose VLMs into a narrow ~5-point band (e.g., RMS spread: 4.8 points) and incorrectly identified Claude Opus 4.5 as the top performer. They failed to distinguish between models that made temporal shifts versus those that made value errors.
Temporal Metrics (ECS/DTW): Spread the four VLMs over a 25-point range (ECS spread: 24.8 points; DTW spread: 20.7 points) and correctly identified Gemini 2.5 Pro as the leader.
ECS vs. DTW: While both temporal metrics discriminated well, ECS correlated 1.5–3.6× more strongly than DTW with four downstream epidemiological statistics: total counts, peak timing, peak magnitude, and growth-rate fidelity. This is attributed to ECS's explicit gap penalty, which prevents truncated predictions from receiving high scores by freely warping to match a subset of ground truth.

Error Analysis

For general-purpose VLMs, the largest sources of error were numerical imprecision (20–30% of loss) and missed datapoints. Specialized models failed primarily due to missed datapoints (often extracting at coarser temporal resolutions) and series-label mismatches. Tool use (code execution) yielded inconsistent results, sometimes degrading performance due to inaccurate cropping of chart regions.

Significance and Claims

The paper claims that EpiCurveBench and EpiCurveSimilarity address critical gaps in evaluating VLMs for time-series chart extraction:

Real-World Stress Test: The benchmark reveals that even leading VLMs fall well short of usable extraction quality on real-world public-health charts, highlighting a need for improved robustness in dense, noisy visual environments.
Downstream Relevance: ECS is validated as a metric that correlates strongly with the accuracy of derived epidemiological statistics. Unlike gap-free metrics like DTW, ECS penalizes truncation and missing data, which are critical failures in outbreak monitoring.
General Applicability: While developed for epidemic curves, the benchmark and metric are applicable to any structured time-series chart-extraction setting, offering a more nuanced evaluation framework than current key-value approaches.

The authors conclude that current models require significant advancement to unlock decades of outbreak data trapped in published figures, and that future progress must address numerical precision on dense series and the handling of long, truncated time series.