PlotPick: AI-powered batch extraction of numerical data from scientific figures

Technical Summary: PlotPick – AI-powered Batch Extraction of Numerical Data from Scientific Figures

Problem Statement
Systematic reviews and meta-analyses in biomedical sciences frequently encounter a bottleneck: critical numerical data is often reported exclusively within scientific figures rather than in machine-readable tables. While existing software tools (e.g., WebPlotDigitizer) offer improvements over manual estimation, they remain inefficient for large-scale studies because they require manual point-clicking for every data series and fail to produce structured, labelled output suitable for downstream statistical analysis. Furthermore, dedicated chart-to-table models (such as DePlot and TinyChart) exhibit poor generalization to the diverse chart types found in biomedical literature, particularly those absent from their specific training distributions.

Methodology
The authors present PlotPick, an open-source, single-file Streamlit application designed to automate the extraction of structured tabular data from scientific figures without requiring model training or manual annotation. The software architecture operates as follows:

1. PDF Parsing: Uploaded PDFs are processed using PyMuPDF to extract text blocks and their bounding boxes.
2. Figure Detection: The system identifies figure captions (e.g., "Figure 1") using regular expressions to anchor crop regions. These regions are expanded to encompass nearby raster images and vector graphics on the same page.
3. VLM Extraction: Each cropped figure image is sent to a Vision-Language Model (VLM) via a minimal, backend-agnostic prompt: "Extract the data from this chart as a tab-separated table."
4. Output: Results are displayed as editable tables with per-row confidence scores and can be exported in multiple formats (Excel, CSV, LaTeX, JSON, and R scripts) to facilitate integration into statistical workflows.

The system is designed to be backend-agnostic, defaulting to Claude Haiku but supporting any VLM accepting image input. It supports batch processing, allowing multiple figures to be handled in a single session.

Key Contributions

• Tool Development: The creation of PlotPick, a tool that combines automatic figure detection from PDFs with VLM-based extraction, eliminating the need for manual point-clicking or prompt engineering.
• Empirical Evaluation: A comprehensive evaluation of six VLMs from three providers (Anthropic, Google, OpenAI) against the dedicated chart-to-table model DePlot on two established benchmarks: ChartX and PlotQA.
• Benchmarking Insights: The study demonstrates that general-purpose VLMs outperform specialized models across all tested chart types, including those not present in the specialized models' training data.

Results
The evaluation utilized two primary metrics: Recall (fraction of ground-truth values recovered within a 5% relative tolerance) and RMSF1 (Relative Mapping Similarity F1, the harmonic mean of precision and recall).

• ChartX Benchmark (n=300): Restricted to bar charts, line charts, box plots, and histograms, all six VLMs outperformed DePlot.
  • Recall: VLMs achieved 88–96%, compared to 71% for DePlot. Even the smallest evaluated VLM (Claude Haiku 4.5) exceeded DePlot by 18 percentage points.
  • Generalization Gap: The performance gap was most pronounced on chart types absent from DePlot's training data. On box plots, DePlot achieved only 24% RMSF1, whereas VLMs achieved 83–97%.
• PlotQA Benchmark (n=529): Despite DePlot being specifically trained on this dataset, PlotPick's VLMs outperformed it.
  • RMSF1: VLMs achieved 86–99%, compared to 94% for DePlot.
• Prompt Engineering: The study found that a simple prompt yielded results within 1–3 percentage points of a detailed, multi-rule prompt, indicating that accuracy is driven primarily by the VLM's inherent vision capabilities rather than complex prompt engineering.
• Limitations Observed: Accuracy remains dependent on chart type and model size. Charts requiring axis-scale reading are less precise, with smaller models occasionally misinterpreting magnitudes (e.g., reading "2.3 million" as "2300"). Stacked and grouped bar charts remain systematically difficult due to the need to decompose overlapping visual elements.

Significance and Claims
The paper claims that general-purpose VLMs, accessed through simple prompts and without specialized training, represent a superior solution for extracting data from scientific figures compared to dedicated chart-to-table models. This superiority is attributed to three factors:

1. Training Data Scale: VLMs are trained on orders of magnitude more data than specialized models, providing broader visual understanding.
2. Chart-Type Coverage: Unlike specialized models trained primarily on bar and line charts, VLMs handle diverse chart types (e.g., box plots, histograms) without specific training.
3. Labelled Data: Extraction accuracy is highest when numeric values appear directly in the image (≥98% accuracy), suggesting that the presence of explicit labels is a stronger predictor of success than model size or prompt design.

The authors position PlotPick as a tool to accelerate data extraction for systematic reviews and meta-analyses in the biomedical sciences. The tool is currently in use at the Copenhagen Research Centre for Biological and Precision Psychiatry to support ongoing evidence synthesis in psychiatry research. The software is released as open source under the MIT license.