Do Large Language Models Understand Data Visualization Rules?

This paper presents the first systematic evaluation of large language models as flexible validators for data visualization rules, revealing that while they achieve high adherence and effectively detect common violations when prompted with natural language, they significantly underperform compared to symbolic solvers on subtle perceptual rules and technical constraint formulations.

The Gist

Imagine you are a teacher grading a stack of student art projects. The students have drawn charts and graphs, but some of them are misleading or just plain wrong. For example, one student might use a rainbow of colors to show a simple list of names (which is confusing), or another might stretch a graph so a tiny change looks like a huge explosion.

For years, we've had a rulebook for this. It's a strict, mathematical guide called Draco that says, "If you do X, you must do Y." But this rulebook is written in a secret code (like a complex computer programming language) that only a few expert engineers can read. It's like having a library of laws written in ancient hieroglyphs: accurate, but hard to use for everyday people.

Recently, Large Language Models (LLMs)—the super-smart AI chatbots like the ones you talk to—have become very popular. People started asking: "Can these AI chatbots read the art projects, understand the rulebook, and tell us which students broke the rules?"

This paper is the first big test to answer that question. Here is how they did it and what they found, explained simply:

1. The Experiment: Creating the "Test"

The researchers couldn't just ask the AI to look at real charts because they wouldn't know for sure if the AI was right or wrong. So, they built a giant, controlled test lab:

• They created 2,000 fake chart instructions (called Vega-Lite specs).
• They used the strict "secret code" rulebook (Draco) to intentionally break the rules in these fake charts.
• They made sure the test was balanced, so the AI didn't just get easy questions or only hard ones.
• They translated the "secret code" rules into plain English so the AI could actually read them.

2. The Contestants: The AI Models

They pitted several different AI models against this test. Think of them as different types of students taking the exam:

• The "Big Brains" (GPT-oss, Gemma 27B): These are the massive, powerful models.
• The "Medium Students" (Gemma 4B, Llama 3.2): Smaller, faster models.
• The "Struggling Students" (Llama 3.1): Older or smaller versions.

3. The Results: Who Passed?

A. Did they follow the instructions? (Prompt Adherence)

Before checking if the AI was smart, the researchers checked if it was obedient. The test required the AI to answer in a very specific format (like a list of errors).

• The Winners: The "Big Brains" (Gemma and GPT) were perfect. They followed the format 100% of the time.
• The Losers: Some of the smaller models got confused. They would give a long paragraph of text instead of a list, or they would forget the format entirely.
• The Lesson: If an AI can't follow simple formatting instructions, it doesn't matter how smart it is; you can't trust its answers.

B. Did they spot the errors? (Accuracy)

Now, did they actually find the broken rules?

• The "Obvious" Mistakes: When the error was clear and common (like using the wrong chart type for the data), the big models were fantastic. They caught about 82% of these errors correctly.
• The "Subtle" Mistakes: When the error was about human perception (like "this color makes the data look bigger than it is"), the models struggled. Their accuracy dropped to near zero for some of these tricky rules.
• The Language Factor: When the researchers gave the AI the rules in plain English, the smaller models got much better (up to 150% improvement!). But when they tried to feed the AI the original "secret code" (the math-heavy rules), the AI was almost completely lost.

4. The Big Takeaway

The paper concludes that AI is getting good at being a "rule checker," but it's not perfect yet.

• The Good News: We can use these AI models as flexible assistants. If you ask them to check a chart for common mistakes in plain English, they are surprisingly good at it. They are like a helpful teaching assistant who knows the basics well.
• The Bad News: They aren't ready to replace the strict, mathematical "rulebook" (Draco) just yet. They miss the subtle, tricky errors that require deep human intuition. Also, if you use a smaller AI, it might get confused and give you a messy answer.

In a nutshell:
Think of the strict rulebook as a laser-guided robot that never makes a mistake but is hard to program. The AI models are like smart interns. They are great at catching the obvious mistakes and can understand you when you speak normally, but they sometimes miss the tiny details and need to be told exactly how to write their report. We should use the interns to help us, but we still need the robot for the final, perfect check.

Technical Summary

1. Problem Statement

Data visualization relies on established rules derived from decades of research in design and perception to ensure trustworthy communication. While constraint-based systems (e.g., Draco) exist to automatically validate chart specifications using logical constraints (Answer Set Programming or ASP), they require significant expert effort to maintain and lack flexibility. Conversely, while Large Language Models (LLMs) have shown promise in detecting misleading charts from images or answering visualization questions, it remains unclear whether they can reason about and enforce specific design rules directly within chart specifications (e.g., Vega-Lite).

The core research question is: Can LLMs accurately identify violations of data visualization rules in chart specifications, and how does their performance compare to symbolic solvers?

2. Methodology

The authors developed a systematic evaluation framework to test LLMs against "hard-verification" ground truth derived from symbolic solvers.

A. Dataset Generation (The 2,000-VIZ Dataset)

• Source: The team generated 2,000 synthetic chart specifications using Draco (a constraint-based system) and converted them into Vega-Lite format.
• Ground Truth: Violations were identified using Draco's ASP solver (Clingo), ensuring every data point has a precise, solver-verified label.
• Balancing Strategy: Random sampling initially produced a skewed distribution of problem types. The authors applied a Kullback–Leibler (KL) divergence filter during generation. This algorithm accepted or rejected candidate specifications based on how much they moved the problem distribution toward uniformity, ensuring a balanced dataset across different violation categories.
• Annotation: Each instance was annotated with specific rule violations (e.g., size_negative, log_scale).

B. Evaluation Framework

• Models Tested: Open-source models including Llama 3.1 (8B), Llama 3.2 (3B), Gemma 3 (4B & 27B), and GPT-oss (20B).
• Prompting Strategy:
  • Models were prompted with Vega-Lite specifications and data tables.
  • Two Rule Formulations:
    1. ASP-based: Technical constraints directly from the solver.
    2. Natural Language: Human-readable translations of the same constraints.
  • Variants: Five different prompt wording variants were used per instance to test sensitivity.
  • Repetition: Each inference was repeated 5 times to average results and reduce variance.
• Metrics:
  1. Prompt Adherence: Did the model output the response in the strictly required structured format (e.g., a JSON list of problem names)?
  2. Accuracy (F1-Score): Did the model correctly detect the specific rule violations compared to the ground truth?

3. Key Contributions

1. First Systematic Evaluation: This is the first study to evaluate LLMs against visualization rules using ground truth derived from symbolic solvers (ASP), bridging the gap between image-based detection and specification-based reasoning.
2. New Dataset: Introduction of a controlled, balanced dataset of 2,000 Vega-Lite specifications with explicit, solver-verified rule violations.
3. Dual-Metric Evaluation: The study introduces a rigorous evaluation protocol that separates reasoning capability (detection accuracy) from instruction following (prompt adherence), highlighting that format failure renders reasoning irrelevant.
4. Formulation Analysis: The paper demonstrates that translating technical constraints (ASP) into natural language significantly impacts model performance.

4. Key Results

A. Prompt Adherence

• High Adherence: Frontier models showed exceptional ability to follow formatting instructions.
  • Gemma 3 (4B & 27B): 100% adherence.
  • GPT-oss 20B: 98% adherence.
• Low Adherence: Smaller Llama variants struggled significantly.
  • Llama 3.1 8B: 65% adherence.
  • Llama 3.2 3B: 87% adherence.
• Implication: Models that fail to adhere to the output format cannot be reliably evaluated, regardless of their reasoning potential.

B. Rule Detection Accuracy (F1-Scores)

• Overall Performance: Performance varies drastically by model size and architecture.
  • GPT-oss 20B: Achieved the highest global average F1-score (0.82).
  • Gemma 3 27B: Second best (0.23 global average), with strong performance on mark and scale issues.
  • Gemma 3 4B: Outperformed Llama variants (0.14 global average).
  • Llama Variants: Struggled significantly (Global Avg F1: 0.07–0.12).
• Impact of Natural Language: Translating ASP constraints to natural language improved performance, particularly for smaller models.
  • Gemma 3 4B: Improved by 150% (from F1 0.058 to 0.145) when using natural language vs. ASP.
  • Llama 3.2 3B: Improved by 13%.
• Category Specifics:
  • Models performed well on common, structural violations (e.g., size_negative, log_scale).
  • Models performed poorly on subtler perceptual rules (e.g., specific encoding overlaps or complex data cardinality issues), with F1 scores dropping below 0.15 for several categories.

C. Comparison of Formulations

• Using raw ASP constraints resulted in near-random performance for most models.
• Natural language descriptions were essential for LLMs to access their reasoning capabilities, confirming that LLMs are not yet capable of directly interpreting formal logic constraints without translation.

5. Significance and Future Work

• Potential: LLMs, particularly larger frontier models, show promise as flexible, language-driven validators for visualization rules, offering a more accessible alternative to maintaining complex symbolic solvers.
• Limitations: Current models lack the precision of symbolic solvers for subtle perceptual rules and are highly dependent on prompt adherence. They often rely on prior knowledge rather than strict logical deduction of the provided specification.
• Future Directions:
  • Expanding the dataset to cover more diverse, real-world scenarios.
  • Investigating fine-tuning strategies to improve robustness against prompt variations.
  • Integrating these models into automated chart auditing tools and visualization recommendation systems.
  • Developing metrics that capture partial correctness and reasoning quality beyond simple F1 scores.

In conclusion, while LLMs are not yet a replacement for symbolic solvers in high-stakes verification, they represent a viable path toward flexible, automated visualization quality assurance, provided that rules are translated into natural language and output formats are strictly enforced.