Technical Summary: ChartAct: A Benchmark for Dynamic Chart Understanding

Problem Statement

Existing benchmarks for chart understanding primarily focus on static charts, where all information is presented within a single, unchanging image. However, real-world data visualization often involves dynamic and interactive charts embedded in webpages or dashboards. In these scenarios, critical information (e.g., precise data values, specific series details) is frequently hidden and only becomes visible after user interactions such as hovering, clicking, zooming, or dragging.

Current Multimodal Large Language Models (MLLMs) and GUI agents struggle with this dynamic nature because:

1. Interaction Requirement: They must not only perceive visual content but also select appropriate interface actions to reveal hidden evidence.
2. State Evolution: They must reason over continuously changing chart states, integrating observations from multiple interaction steps.
3. Localization: They must accurately locate interactive elements (e.g., tooltips, legends, sliders) amidst complex page layouts, which is difficult when charts are embedded in broader dashboard contexts.

Existing static benchmarks fail to evaluate these capabilities, as they do not require models to perform sequential actions or observe state changes.

Methodology

1. The ChartAct Benchmark

The authors propose ChartAct, an interactive benchmark designed to evaluate dynamic chart understanding. The benchmark consists of:

• Data Collection: 673 dynamic charts were crawled from 8 real-world charting libraries (e.g., Apache ECharts, Highcharts, Plotly, AntV G2). These charts were manually filtered to ensure clear data semantics, rich interactive behaviors, and analytical value.
• Question Construction: 1,440 high-quality question-answer pairs were constructed. Questions were generated using GPT-5 and rigorously reviewed to ensure they:
  • Cannot be answered solely from the initial static view (requiring interaction).
  • Do not rely on hidden HTML/data that cannot be accessed through observable UI interactions.
  • Cover diverse tasks: visible-state QA, interaction-revealed information, cross-state comparison, and multi-chart evidence integration.
• Evaluation Environments: Each sample is instantiated in two distinct environments to test different levels of difficulty:
  • Dynamic Chart (DC): A clean environment where the chart is displayed directly on the page.
  • Dashboard Chart (DB): The same chart is embedded into a generated dashboard page containing titles, metric cards, and other visual modules. This tests the model's ability to locate the target chart and ignore visual distractors.

2. Experimental Setup

The benchmark evaluates 11 advanced multimodal models and GUI agents (including proprietary models like Claude-Opus-4.7, GPT-5.5, and open-source models like Kimi-K2.5, Qwen series, and Gemma).

• Unified Framework: All models operate within a unified interactive environment based on the OSWorld framework. Models observe screenshots, execute pyautogui actions (click, hover, scroll), and submit final answers.
• Evaluation Metric: Performance is measured by Success Rate, determined by an LLM judge (DeepSeek-v4-pro) that compares the model's final answer against the ground truth. A majority voting mechanism (3 votes, 2 required) is used to ensure stability.
• Subset Construction: For high-cost models, a representative subset of 300 samples was constructed using a greedy selection algorithm based on pilot model performance to preserve difficulty distribution and chart-type coverage while reducing evaluation costs.

Key Results

1. Overall Performance

Dynamic chart understanding remains a significant challenge for current models:

• Top Performer: Claude-Opus-4.7 achieved the highest average success rate of 84.5%.
• Performance Gap: There is a clear gap between the top model and others. GPT-5.5 scored 58.3%, while most open-source models and other proprietary models scored below 60% (e.g., Qwen3.6-Plus at 39.2%, Gemma-4-31B-IT at 23.1%).
• Open-Source Limitations: Excluding the large-scale Kimi-K2.5 (66.0%), the best open-source model (Qwen3.5-122B-A10B) achieved only 41.9%, indicating substantial room for improvement in the open-source community.

2. Impact of Environment (DC vs. DB)

The Dashboard Chart (DB) environment consistently reduced performance across all models compared to the clean Dynamic Chart (DC) environment.

• Context Sensitivity: Models like Gemini-3.1-Pro and Qwen3.6-Plus suffered massive drops (30.0% and 30.8% respectively) in the DB environment, suggesting they struggle to transfer skills from clean charts to complex dashboard contexts.
• Robustness: The strongest model, Claude-Opus-4.7, showed the highest resilience, with only a 1.7% drop in the DB environment.

3. Chart Type Analysis

• Line Charts: Showed the largest performance drop when moving from DC to DB, likely due to the complexity of triggering tooltips along continuous axes and distinguishing adjacent lines in cluttered layouts.
• Pie Charts: While generally lower-performing, they showed the smallest drop in DB. However, many models failed on Pie charts by overconfidently guessing values from visible percentages without interacting to reveal precise data.
• No Fixed Order: The difficulty of chart types varied by model architecture; for instance, Claude-Opus-4.7 performed best on Pie charts, while GPT-5.5 performed best on Box plots.

4. Failure Analysis

The authors identified three primary failure modes:

1. Failure to Extract Evidence: Models answered directly from the initial screenshot without triggering necessary interactions.
2. Mis-localized Interactions: Models correctly inferred the need for interaction but targeted the wrong element (e.g., hovering on a neighboring data point).
3. Target Object Misidentification: In dashboard environments, models confused the target chart with surrounding components (e.g., metric cards or titles).

Significance and Claims

The paper claims that ChartAct provides a necessary new testbed for studying chart understanding in real interactive environments. Its significance lies in:

• Bridging the Gap: Moving beyond static image analysis to evaluate the full cycle of interaction, observation, and reasoning required in real-world data analysis.
• Revealing Limitations: Demonstrating that current state-of-the-art models, even the strongest proprietary ones, have clear limitations in dynamic chart understanding, particularly in fine-grained operations and evidence acquisition.
• Standardizing Evaluation: Offering a controllable, reproducible, and realistic environment (via the OSWorld-based framework) that preserves the complexity of real charts while allowing for systematic benchmarking.

The authors conclude that dynamic charts introduce evaluation requirements that go beyond static chart understanding, necessitating models capable of strategic interaction and state tracking.