Technical Summary: From Data to Insights: Exploring Program-of-Thoughts Prompting for Chart Summarization

Problem Statement

Chart summarization is a critical task for conveying numerical data insights through structured visual representations. However, existing Visual Language Models (VLMs) struggle with accurate chart description due to two primary challenges:

1. Semantic and Numerical Reasoning Gaps: Models often fail to parse text within charts or demarcate numerical values accurately, leading to factual inconsistencies and hallucinations, particularly in Level 2/Level 3 content (e.g., statistical relations like min/max, trends).
2. Computational and Structural Limitations: Current state-of-the-art approaches often rely on fine-tuning or pre-training on specific chart instruction data. While this improves generalization, it introduces significant computational overhead, making these models resource-intensive. Furthermore, existing methods struggle to capture the structural interplay between chart elements and perform robust statistical reasoning without relying on provided data tables, which are often unavailable in real-world scenarios.

Methodology

The paper proposes a zero-shot, training-free pipeline that integrates Program-of-Thoughts (PoT) prompting to enhance the statistical reasoning capabilities of lightweight VLMs. The methodology consists of three sequential stages:

1. Chart-to-Dictionary Representation:
   Instead of relying on traditional chart-to-table methods which may lose finer details, the pipeline introduces a novel chart-to-dictionary auxiliary task. A lightweight VLM (specifically InternVL-2.5-4B in a zero-shot setting) is prompted to convert the visual chart into a valid, executable Python dictionary. This representation offers a more flexible structure than markdown tables for downstream processing. If the initial generation fails (e.g., syntax errors), a secondary model (ChatGPT-4o-mini) is used to correct the dictionary.

2. Statistical Analysis via PoT:
   The generated Python dictionary serves as the input for a code-specialized Large Language Model (Qwen-2.5-Coder-14B). The LLM is instructed to generate a Python program that performs statistical analysis (e.g., calculating totals, minimums, maximums, and averages) on the dictionary data. This step acts as an intermediate reasoning process, disentangling computation from reasoning to ensure numerical accuracy. The generated code is constrained to use only Python's built-in libraries.

3. Program Execution and Summary Generation:
   The generated Python program is executed using a Python interpreter to produce a dictionary of accurate summary statistics. These calculated statistics, along with the original chart dictionary and the chart title, are then fed back into the VLM to generate the final natural language summary. This ensures the summary is grounded in verified numerical data rather than model hallucination.

Key Contributions

• PoT-Integrated Pipeline: The authors propose a training-free pipeline that augments lightweight VLMs for chart summarization in a zero-shot setting by leveraging Python programs as intermediaries for statistical reasoning.
• Chart-to-Dictionary Auxiliary Task: A novel representation method is introduced where charts are converted into Python dictionaries. This offers greater flexibility compared to chart-to-table approaches and is specifically designed to integrate with the PoT strategy.
• Comprehensive Evaluation: The study conducts extensive evaluations across lexical (BLEU, CIDEr, ROUGE, BERTScore), semantic, and factual (UniEval, AlignScore) dimensions, comparing the PoT approach against Direct prompting and Multimodal Chain-of-Thought (MCoT) strategies across diverse VLMs and chart types.

Results

The experimental evaluation was conducted on the Pew and VisText datasets, covering various chart types (Area, Bar, Line, Pie, Scatter) and complexity levels.

• Performance Variance by Model: The efficacy of the PoT strategy is highly dependent on the underlying VLM architecture.
  • InternVL-2.5: Demonstrated significant improvements with PoT, particularly in factual correctness (AlignScore) and semantic coherence, often outperforming MCoT and approaching Direct prompting performance.
  • Qwen2.5-VL-3B: Showed consistent outperformance over MCoT, suggesting PoT is a more effective reasoning strategy for this model, though it sometimes lagged behind Direct prompting.
  • DeepSeek-vl2-tiny: The PoT strategy resulted in considerably lower scores compared to Direct and MCoT methods, indicating that for this specific architecture, the additional textual information from PoT may introduce noise or interfere with the model's native processing.
• Chart Type Sensitivity: Models generally performed better on simpler charts (Bar, Pie) compared to complex ones (Line, Scatter). The PoT strategy showed particular effectiveness in enhancing information collection for Area charts with limited data.
• Factual Accuracy: While BLEU scores (surface-level matching) varied, the PoT approach often yielded higher scores in factual evaluation metrics (AlignScore, UniEval) compared to baseline methods, indicating a reduction in hallucinations regarding numerical data.
• Ablation Studies: The inclusion of PoT-generated statistics and dictionaries generally improved performance over using titles alone, particularly for the InternVL model. However, the study noted that poorly extracted data could sometimes negatively impact the pipeline, highlighting the sensitivity of the method to the quality of the initial dictionary generation.

Significance and Claims

The paper claims that the proposed PoT strategy offers a viable, resource-efficient alternative to heavy fine-tuning for chart summarization, particularly for lightweight VLMs. By delegating complex numerical calculations to a code interpreter, the method mitigates the hallucination errors common in direct VLM generation.

The authors emphasize that the chart-to-dictionary task is a promising auxiliary objective that captures semantic nuances better than rigid table formats, facilitating robust integration with reasoning frameworks like PoT. However, the paper maintains a modest tone regarding its conclusions, acknowledging that:

• The benefits of PoT are not universal across all VLM architectures; model design and pre-training data significantly influence outcomes.
• The current implementation relies on lightweight models, and results might differ with larger parameter models.
• Complex chart types remain challenging, suggesting a need for future work on normalizing data structures to bridge complexity gaps.

Ultimately, the work demonstrates that combining visual understanding with executable code generation can enhance the factual grounding of chart summaries without the computational cost of extensive model retraining.