PlotChain: Deterministic Checkpointed Evaluation of Multimodal LLMs on Engineering Plot Reading

1. Problem Statement

Engineering practice relies heavily on interpreting quantitative plots (e.g., Bode plots, stress-strain curves, FFT spectra) to validate designs and troubleshoot systems. While Multimodal Large Language Models (MLLMs) have advanced in general chart understanding, existing benchmarks suffer from three critical limitations when applied to engineering:

• Lack of Specificity: Most benchmarks focus on generic infographics or captioning rather than canonical engineering plot families requiring precise quantitative extraction.
• Noisy Ground Truth: Many datasets rely on human annotations or data tables that do not perfectly align with the rendered pixel data, introducing annotation noise.
• Black-Box Evaluation: Standard benchmarks evaluate only the final answer, making it impossible to distinguish between visual reading errors (e.g., misreading an axis) and downstream reasoning errors (e.g., incorrect arithmetic).

The paper argues that there is a need for a benchmark that provides exact, auditable ground truth, targets specific engineering domains, and supports diagnostic evaluation to isolate failure modes.

2. Methodology: PlotChain Benchmark

PlotChain is a deterministic, generator-based benchmark designed to evaluate MLLMs on recovering quantitative values from engineering plots.

A. Dataset Generation

• Structure: The dataset contains 450 items across 15 canonical engineering plot families (30 items per family).
• Families: Includes step responses, Bode magnitude/phase, bandpass responses, FFT spectra, I-V curves (resistor/diode), stress-strain, pump curves, and S-N fatigue curves.
• Deterministic Ground Truth: Unlike datasets using OCR or human labeling, PlotChain generates plots from known parameters. The ground truth is computed directly from these generation parameters using analytic baselines, ensuring exact verifiability.
• Difficulty Levels: Items are categorized into three difficulty tiers:
  • Clean: Standard visual aids (ticks, grids).
  • Moderate: Mild noise or perturbations.
  • Edge: Reduced visual aids (tighter windows, sparse ticks) to stress human-readable limits.
• Quantization: Values are quantized to realistic decimal precision (e.g., integer Hz, 0.1 dB) to reflect human reading capabilities rather than sub-pixel reconstruction.

B. Checkpoint-Based Diagnostic Evaluation

A core novelty of PlotChain is the inclusion of intermediate "checkpoint" fields (prefixed cp_).

• Final Fields: The headline numeric answers (e.g., cutoff_hz, settling_time).
• Checkpoint Fields: Intermediate reads required to derive the final answer (e.g., cp_mag_at_fc_db, cp_peak_value).
• Purpose: This allows researchers to isolate whether a model failed at perception (reading the graph) or reasoning (calculating the derived value).

C. Evaluation Protocol

• Models Evaluated: Four state-of-the-art MLLMs: Google Gemini 2.5 Pro, OpenAI GPT-4.1, OpenAI GPT-4o, and Anthropic Claude Sonnet 4.5.
• Constraints:
  • Deterministic Decoding: Temperature set to 0.
  • Output Schema: Strict JSON-only numeric output.
  • Scoring: Uses tolerance-based scoring (absolute and relative error) rather than exact string matching, reflecting realistic human plot-reading precision.
• Reproducibility: The release includes the generator code, dataset, raw model outputs, and checksums, enabling retrospective rescoring under different tolerance policies.

3. Key Contributions

1. Deterministic Engineering Benchmark: A generator-based dataset with exact ground truth derived from parameters, eliminating annotation noise.
2. Checkpoint Diagnostics: A novel evaluation framework using intermediate cp_* fields to localize failures between visual extraction and logical derivation.
3. Standardized Protocol: A unified evaluation setup with strict JSON output and tolerance-based scoring aligned with human engineering practices.
4. Open Artifact Release: Full transparency with code, data, and raw outputs to support reproducible research and retrospective analysis.

4. Results

The study evaluated four models on the 450-item dataset using the "plotread" tolerance policy.

A. Overall Performance

• Top Performers (Item-Level Strict All-Pass):
  • Gemini 2.5 Pro: 72.0%
  • GPT-4.1: 68.4%
  • Claude Sonnet 4.5: 61.3%
  • GPT-4o: 32.4%
• Field-Level Pass Rates: The top three models clustered closely (78–80%), while GPT-4o lagged significantly (61.6%).
• Statistical Significance: The difference between the top two models (Gemini 2.5 Pro vs. GPT-4.1) was not statistically significant after Holm correction. However, all three frontier models significantly outperformed GPT-4o.

B. Family-Level Insights

• Solved Tasks: Simple families like I-V Resistor curves were solved at 100% by all models.
• Persistent Bottlenecks:
  • Frequency-Domain Tasks: Bandpass Response remained extremely difficult (7–23% pass rate).
  • FFT Spectra: Remained challenging (25–50% pass rate).
  • Derived Quantities: Tasks requiring multiple intermediate reads to compute a final value (e.g., bandwidth from two cutoff points) showed compounding errors, causing strict all-pass rates to drop significantly even when intermediate fields were correct.

C. Checkpoint vs. Final Field Analysis

• Failure Localization:
  • GPT-4o & Claude: Tended to score higher on checkpoint fields than final fields, suggesting they can read intermediate cues but fail at multi-step derivations.
  • Gemini 2.5 Pro & GPT-4.1: Showed stronger performance on final fields, indicating better handling of compounding errors.
• Latency Trade-off: GPT-4.1 achieved high accuracy with the lowest latency (1.97s), while Gemini 2.5 Pro was the most accurate but significantly slower (20.66s).

5. Significance and Impact

• Diagnostic Precision: PlotChain moves beyond "black box" scoring, enabling researchers to pinpoint whether a model's failure is due to visual perception (axis calibration) or logical reasoning (arithmetic/derivation).
• Realistic Engineering Simulation: By using tolerance-based scoring and realistic difficulty tiers, it better simulates the actual workflow of an engineer reading a plot compared to exact-match benchmarks.
• Reproducibility Standard: The release of the generator and raw outputs sets a new standard for transparent ML evaluation, allowing the community to re-score results as models or tolerance policies evolve.
• Future Directions: The paper highlights that while frontier models are approaching human-level performance on simple plots, frequency-domain and derived-quantity tasks remain a significant challenge, identifying a clear area for future model improvement.