Evaluating Graphical Perception Capabilities of Vision Transformers

This paper benchmarks Vision Transformers against CNNs and humans on elementary graphical perception tasks inspired by Cleveland and McGill, revealing that while ViTs excel in general vision, they exhibit significant limitations in aligning with human-like perceptual accuracy for interpreting visualizations.

The Gist

Imagine you have two very different types of students trying to learn how to read a map.

Student A (The Human) is an experienced traveler. They look at a map and instantly understand that a long line means a long road, a big circle means a large city, and a steep angle means a sharp turn. They don't need to count every single pixel; they just "get" the shape and size intuitively.

Student B (The Old Robot - CNN) is a student who learned by looking at millions of tiny, blurry photos. They are very good at recognizing patterns like "this looks like a cat" or "that looks like a car" because they've seen them a thousand times. They are good at spotting details, but they sometimes struggle to see the whole picture at once.

Student C (The New Robot - Vision Transformer or ViT) is the star student of the modern era. They are famous for being able to look at a whole room and instantly understand how the furniture relates to the walls, the windows, and the door. They are incredibly smart at connecting distant dots and understanding complex scenes.

The Big Question

This paper asks a simple but crucial question: If we give these robots a basic math test involving shapes and sizes (like reading a bar chart), will the New Robot (ViT) act more like the Human traveler, or will it still act like a robot?

In the world of data visualization (charts, graphs, maps), we need computers to "see" things the way humans do. If a computer thinks a bar chart is easy to read but a human finds it confusing, the computer might give us bad advice.

The Experiment

The researchers set up a series of "elementary school" tests for these robots. They didn't ask them to write a poem or diagnose a disease. They asked them to do basic visual judgments, just like the famous Cleveland and McGill studies from the 1980s:

• Length: Which bar is longer?
• Angle: Which slice of the pie is bigger?
• Position: Which dot is higher up?
• Area: Which shape covers more space?
• Counting: How many dots are in this cloud?

They tested three versions of the New Robot (ViT):

1. The Pure Transformer: Just the raw attention mechanism.
2. The Hybrid (CvT): A mix of the old robot's local vision and the new robot's global vision.
3. The Swin: A version that looks at the image in small "windows" first, then zooms out.

The Results: The Plot Twist

Here is where it gets interesting. You might expect the New Robot (ViT) to be perfect because it's the "state-of-the-art" technology. But the results were surprising:

1. The Human is still the Gold Standard.
In almost every test, the Human traveler was the most accurate. They could compare lengths and positions with incredible precision.

2. The New Robot (ViT) is actually worse than the Old Robot (CNN) at basic math.
This is the biggest shock. While ViTs are amazing at recognizing complex scenes (like "is this a dog running in a park?"), they were struggling with the simple stuff.

• When asked to compare the length of two bars, the ViT made more mistakes than the older CNN models.
• When asked to count dots in a cloud, the ViT got very confused.
• The ViT seemed to think some difficult tasks (like judging curved lines) were easy, and some easy tasks (like comparing lengths) were hard. It had a completely different "sense" of reality than humans.

3. The "Window" Robot (Swin) was the best of the bunch, but still not human.
The Swin Transformer performed the best among the robots, but it still couldn't match human accuracy. It was like a student who is great at writing essays but keeps failing basic arithmetic.

The Analogy: The "Zoom" Problem

Think of it this way:

• Humans look at a chart and see the meaning. We know that a bar's height represents a number.
• CNNs look at the chart and see local details. They see the edges of the bar very clearly.
• ViTs look at the chart and try to see everything at once. They are so busy looking at the relationship between the title, the legend, and the background that they sometimes lose track of the exact length of the bar. They are "distracted" by the big picture and miss the small, precise details needed for accurate measurement.

Why Does This Matter?

We are building more and more tools that use AI to read charts, summarize data, and even design new graphs for us. If we use a Vision Transformer to build a dashboard for doctors or financial analysts, and that AI "sees" the data differently than a human does, it could lead to dangerous misunderstandings.

For example, if an AI thinks a small difference in a graph is huge (because it's bad at judging length), it might tell a doctor that a patient's condition has worsened when it hasn't.

The Takeaway

The paper concludes that while Vision Transformers are powerful and exciting, they are not yet "human-like" when it comes to basic visual perception.

They are like a genius who can write a symphony but can't tell you if two sticks are the same length. To use them safely in data visualization, we need to either:

1. Train them specifically to be better at these basic tasks.
2. Be very careful about where we use them, knowing they might "hallucinate" the size or shape of things.

The researchers are calling for more work to bridge the gap between how these super-smart robots see the world and how we, as humans, see it. Until then, we should probably keep a human in the loop to double-check the math.

Technical Summary

1. Problem Statement

Vision Transformers (ViTs) have become the state-of-the-art in general computer vision, often outperforming Convolutional Neural Networks (CNNs) in tasks like classification and object detection. However, their application in data visualization and graphical perception remains under-explored.

While CNNs have been previously evaluated for their ability to perform elementary visual judgment tasks (inspired by Cleveland and McGill's foundational studies on human perception), it is unclear if ViTs align with human perceptual hierarchies. Specifically, the paper addresses the gap in understanding whether ViTs can accurately interpret low-level visual encodings (e.g., position, length, angle, area) which are the building blocks of effective data visualization. The core question is: Do ViTs perceive visual data in a way that aligns with human cognitive processing, or do they exhibit fundamental perceptual gaps that limit their utility in visualization systems?

2. Methodology

Experimental Design

The study replicates and extends the classic perceptual experiments by Cleveland and McGill [12] and Haehn et al. [26]. The researchers evaluated three representative ViT architectures against human observers and established CNN benchmarks.

Visual Tasks Evaluated:
The study focused on nine elementary perceptual encodings, grouped into specific experimental tasks:

1. Position-Angle: Estimating ratios in bar charts and pie charts.
2. Position-Length: Evaluating grouped vs. divided bar charts (assessing position on a common scale vs. length estimation).
3. Bars and Framed Rectangles: Judging length and position on non-aligned scales.
4. Point Cloud: Estimating dot quantities (10, 100, 1000) in scatter plots, subject to Weber's Law.

Models Tested:
Three distinct ViT architectures were selected to represent different design strategies:

• Vanilla ViT (vViT): Pure transformer architecture with global self-attention and no inductive biases.
• Convolutional Vision Transformer (CvT): Integrates convolutional layers for token embedding and projection to enhance local feature extraction.
• Swin Transformer: Uses hierarchical representation learning and shifted windows to capture local and global context efficiently.

Data and Training:

• Stimuli: 100,000 synthetic images per task (100x100 binary images resized to 224x224 for ViT input), generated with random parameters to prevent memorization.
• Training Protocol: Unlike many studies using pre-trained foundation models, these models were trained from scratch on the specific task datasets to ensure fair comparison and control over architectural biases.
• Metrics: Performance was measured using Mean Log Absolute Error (MLAE) to quantify perceptual accuracy. Lower MLAE indicates better alignment with ground truth.

3. Key Contributions

1. Systematic Benchmarking: The first comprehensive evaluation of ViTs on low-level graphical perception tasks, directly comparing them to human performance and CNN baselines.
2. Architectural Analysis: A comparative study of three distinct ViT families (vViT, CvT, Swin) to determine how architectural choices (e.g., global attention vs. convolutional inductive bias) impact perceptual fidelity.
3. Perceptual Gap Identification: Identification of specific visual encodings where ViTs diverge significantly from human perception, particularly in comparative reasoning tasks.
4. Ablation Studies: Investigation into the effects of image resolution, patch size, training data volume, and pre-training on ViT performance in these specific tasks.

4. Key Results

ViTs vs. Humans

• Overall Performance: Humans consistently outperformed ViTs in most low-level tasks.
• Specific Divergences:
  • Strengths: ViTs (particularly Swin) outperformed humans in Direction and Shading tasks, suggesting strong capabilities in processing fine-grained texture and orientation.
  • Weaknesses: ViTs struggled significantly with Position-Length and Point Cloud estimation. For example, in Position-Length tasks, humans achieved an MLAE of 2.01, while Swin recorded 4.72. In Point Cloud estimation, humans scored 4.95 vs. Swin's 6.37.
• Ranking Discrepancies: While humans and ViTs agreed on the difficulty of simple tasks (like Length), ViTs ranked complex tasks like Curvature and Area as significantly easier than humans do, indicating a fundamental misalignment in how they process spatial relationships.

ViTs vs. CNNs

• Error Rates: Across all tasks, ViTs exhibited higher error rates than CNNs (specifically VGG19 and ResNet-18).
  • Point Cloud: CNNs (MLAE ~3.40) significantly outperformed ViTs (MLAE ~6.37).
  • Bars/Framed Rectangles: CNNs (MLAE ~1.93) vastly outperformed ViTs (MLAE ~4.75).
• Generalization: ViTs showed poor generalization to unseen parameter variations (e.g., changes in object width or spatial alignment) compared to CNNs. The Swin Transformer was the best-performing ViT but still lagged behind CNNs in precision.

Ablation Findings

• Resolution & Patch Size: Increasing image resolution or reducing patch size (to 8x8) did not improve performance.
• Data Volume: Increasing training data size by 4x yielded no significant performance gains, challenging the assumption that ViTs simply need more data for these tasks.
• Pre-training: Pre-training on ImageNet improved performance for some models (like CvT) but did not universally solve the perceptual alignment issues.

5. Significance and Implications

• Limitations in Visualization Systems: The study concludes that while ViTs are powerful for general vision, they currently lack the perceptual fidelity required for reliable automated chart interpretation and redesign. Their inability to accurately judge relative lengths and quantities (critical for data visualization) poses a risk for automated systems that rely on human-like perception.
• Architectural Insights: The results suggest that the global attention mechanism of standard ViTs may be less effective than the local receptive fields of CNNs for specific low-level graphical encodings. The Swin Transformer's hierarchical, windowed approach performed best among ViTs, hinting that a hybrid approach (combining local and global processing) is necessary.
• Future Directions: The paper calls for a redesign of visualization AI systems to better leverage ViT strengths (e.g., texture/orientation) while mitigating weaknesses (e.g., comparative estimation). It suggests future research should explore larger models, hybrid architectures, and the integration of multimodal models to bridge the gap between machine and human perception.

In summary, the paper serves as a critical "reality check" for the visualization community, demonstrating that state-of-the-art ViTs do not yet replicate human graphical perception and that relying on them for low-level visual judgment tasks without careful validation could lead to erroneous data interpretations.