Assessing Y-Axis Influence: Bias in Multimodal Language Models on Chart-to-Table Translation

1. Problem Statement

Multimodal Language Models (MLMs) are increasingly used to translate chart images into structured tabular data (Chart-to-Table translation). While previous research has investigated biases in MLMs related to chart complexity, color schemes, and font sizes, there is a significant gap in understanding how y-axis specific information affects model performance.

The authors identify that public datasets (e.g., PlotQA, ChartQA) suffer from severe imbalances regarding y-axis characteristics, such as:

• Digit length: The number of digits in the integer part of the maximum y-axis value.
• Tick configurations: The number of major ticks and the range of values.
• Value formats: The use of abbreviations (K, M, B), scientific notation, or commas.

These imbalances lead to unintended biases where models perform well on specific y-axis configurations but fail to generalize to others. Furthermore, existing evaluation metrics (like RMSF1) often fail to capture visually significant errors, such as deviations larger than a minor tick interval, leading to unreliable assessments.

2. Methodology: The FairChart2Table Framework

To address these gaps, the authors propose FairChart2Table, a bias-controlled framework consisting of a new benchmark dataset and novel evaluation metrics.

A. Benchmark Construction

The framework generates a synthetic dataset using the Bokeh plotting library to ensure strict control over variables. The dataset is divided into four subsets (Parts A–D) to isolate specific biases:

• Part A (Digit Length): Generates 3,060 images varying the digit length of y-axis values (0–16) and the number of entities (1–6).
• Part B (Major Ticks): Compares performance with 3, 6, and 11 major ticks.
• Part C (Value Range): Tests three range scenarios:
  • Positive (Pos): Origin shifted up (all values positive).
  • Negative (Neg): Origin shifted down (all values negative).
  • Extended (Ext): Data range doubled, pushing values to the bottom.
• Part D (Number Formats): Tests plain numbers, commas, abbreviations (K, M, B, T), and scientific notation.

All other visual elements (titles, fonts, colors, x-axis) are standardized to prevent confounding factors.

B. New Evaluation Metrics

The authors critique the standard RMSF1 metric for not adequately penalizing visually noticeable errors. They propose Tick-Based Error (TBE) metrics:

1. $DTBE$ (Tick-Based Distance): Normalizes the absolute error ($|g - p|$) by the minor tick interval ($t$), rather than the ground truth value.
   $$DTBE(g, p) = \min\left(1, \frac{|g - p|}{t}\right)$$
2. $RMSTBE_{F1}$: An F1-score variant using $DTBE$ for numerical alignment, removing header penalties to focus on data accuracy.
3. $RMSTBE_{F1}$-Sig: Focuses only on "significant" deviations where the error exceeds the minor tick interval.
4. SES (Swapping Error Score): Measures structural errors where values are correctly extracted but assigned to the wrong entity (common in crossing lines), calculated as the difference between a minimal cost matching score and the standard $RMSTBE_{F1}$.

3. Experimental Setup

The framework was used to evaluate five state-of-the-art models:

• Closed-Source: GPT-4o, Gemini-2.0-flash.
• Open-Source: Pixtral-12B.
• Specialized/Supervised: DePlot (chart-to-table specific), ChartGemma, and TinyChart (general chart understanding).

Experiments involved paired statistical tests (Wilcoxon signed-rank) to compare performance across different bias configurations.

4. Key Results and Findings

RQ1: Impact of Y-Axis Information

• Digit Length: Performance generally degrades as digit length increases.
  • DePlot and TinyChart show significant drops at higher digit lengths (e.g., 11, 15, 16).
  • GPT-4o performs the worst overall and exhibits strong bias against high digit lengths.
  • Gemini is the most robust but still shows a downward trend.
• Major Ticks: All models perform worst with 3 major ticks. Closed-source models (Gemini, GPT-4o) improve with 11 ticks, while open-source models show no uniform trend.
• Value Range:
  • DePlot struggles with negative origins and extended ranges.
  • TinyChart excels when the axis crosses at zero but fails with displaced origins.
  • Gemini remains consistent across most ranges but drops significantly with extended ranges.
• Number Formats:
  • Abbreviations (K, M, B, T) cause dramatic performance drops for almost all models (e.g., TinyChart drops from ~80% to ~26%).
  • Scientific Notation also negatively impacts many models, though DePlot and TinyChart handle it better than abbreviations.

RQ2: Other Factors (Entities and Chart Types)

• Number of Entities: As the number of entities (1–6) increases, performance generally declines due to an increased probability of line crossings.
  • SES Scores: Models like DePlot and TinyChart show high swapping errors (confusing entities) with 5–6 entities.
  • Gemini and Pixtral are more robust against entity count increases.
• Chart Types: When underlying data is identical, performance differences between Line, Dot, and Bar charts are statistically significant but less pronounced than y-axis biases.

RQ3: Impact of Prompting with Y-Axis Hints

• Providing explicit y-axis tick values as part of the prompt significantly improves performance for some models.
• Pixtral saw gains ranging from 0.94% to 26.60%.
• Gemini showed a dramatic improvement (from 46.54% to 61.32%) specifically when handling abbreviated formats, suggesting the hint helps decode non-standard formats.

5. Significance and Contributions

1. Identification of Hidden Biases: The paper systematically reveals that y-axis characteristics (digit length, tick count, formats) are major, previously unexamined sources of bias in MLMs.
2. FairChart2Table Framework: A reproducible benchmark and methodology to generate bias-controlled datasets, allowing researchers to isolate specific variables.
3. Improved Metrics: The introduction of TBE-based metrics ($RMSTBE_{F1}$) provides a more visually accurate assessment of chart-to-table translation errors compared to traditional text-based metrics.
4. Practical Insights: The study demonstrates that prompt engineering (providing y-axis hints) can mitigate specific biases, particularly for handling abbreviations and complex digit lengths.

6. Limitations

• The dataset is synthetic (generated via Bokeh) and does not include real-world, noisy chart images.
• All text (charts and legends) is in English.
• The study focuses exclusively on y-axis biases, not other chart elements.

Conclusion

The authors conclude that current MLMs are highly sensitive to y-axis configurations, often failing on data with high digit lengths or non-standard formats. The FairChart2Table framework provides a necessary tool for developing more robust, unbiased models for chart understanding, and suggests that explicit y-axis context can be a powerful lever for improving model accuracy.