Read, Critique, or Sketch? Investigating Alternative Visualization Literacy Assessment Modalities

Technical Summary: Read, Critique, or Sketch? Investigating Alternative Visualization Literacy Assessment Modalities

Problem Statement
Visualization literacy is a multifaceted construct involving skills in decoding, constructing, and critiquing visualizations. However, existing assessments (e.g., VLAT, Mini-VLAT, CALVI) are predominantly multiple-choice formats focused on lower-order skills like chart comprehension and factual extraction. These quantitative methods often suffer from "ceiling effects," where even modestly skilled individuals score near the maximum, failing to differentiate between varying levels of expertise. Furthermore, they lack the nuance to assess higher-order skills such as applying external knowledge, formulating critiques, and designing visualizations from scratch. The authors argue that, analogous to the inclusion of essays in standardized tests like the SAT and GRE for textual literacy, the visualization community requires unconstrained, creation-focused qualitative assessments to fully evaluate the range of literacy skills.

Methodology
The authors conducted a within-subjects user study ($N=80$) to develop and validate two new qualitative assessment modalities: Sketching and Critique.

• Participants: Three distinct groups representing varying expertise levels were recruited:
  • Crowdworkers ($n=42$): Recruited via Prolific, presumed to have minimal formal visualization training.
  • Students ($n=21$): Undergraduate or graduate students who had completed a visualization course.
  • Experts ($n=17$): Visualization researchers (PhD holders or capable of reviewing research papers).
• Assessment Tasks:
  • Sketching: Participants were presented with five diverse datasets (e.g., social networks, file hierarchies, rainfall data) and asked to "Sketch a visualization that you believe best represents this data" while thinking aloud. They used a web-based canvas (TLDraw) with provenance tracking.
  • Critique: Participants were shown three visualizations with intentional design flaws (e.g., a 3D treemap with an inappropriate color scale, a line chart with misleading log-scale projections, and a map with poor binning). They were asked to think aloud and critique the visualizations.
  • Control Measures: Participants also completed established multiple-choice assessments: Mini-VLAT (comprehension) and CALVI (critical thinking).
• Grading and Analysis:
  • Rubrics: Holistic rubrics were developed for both modalities. The Sketching rubric evaluated Visualization Design/Hierarchy, Base Encoding Choice, Semantic Validity, and Guides/Text. The Critique rubric evaluated Reading, Contextualization, and Judgment. Scores ranged from "None" to "Excellent" (5-point scale).
  • Statistical Analysis: The authors utilized Bayesian Item Response Theory (IRT) using a 2PL ordinal regression model to estimate latent abilities and item discrimination. They performed Welch's t-tests to compare group means and calculated Pearson correlations between the latent ability scores of the four assessments.

Key Contributions

1. Novel Assessment Modalities: The paper introduces and validates two qualitative, open-ended assessment tasks (Sketching and Critique) designed to measure higher-order visualization literacy skills that multiple-choice tests miss.
2. Holistic Rubrics: The authors provide detailed, holistic grading rubrics for evaluating freeform sketches and think-aloud critiques, moving beyond simple correctness to evaluate design rationale and reasoning.
3. Empirical Validation: Through a comparative study with established tests, the authors demonstrate that these qualitative modalities successfully differentiate between expertise levels (crowdworkers, students, and experts) where traditional tests fail.

Results

• Differentiation of Expertise: Both Sketching and Critique assessments showed strong sensitivity to differences in expertise.
  • Sketching: Significantly differentiated between all three groups (Experts vs. Crowdworkers, Students vs. Crowdworkers, and Experts vs. Students).
  • Critique: Significantly differentiated between Experts and Crowdworkers, and Experts and Students, though the distinction between Students and Crowdworkers was less pronounced.
  • Comparison: In contrast, Mini-VLAT exhibited a strong ceiling effect, with no significant pairwise differences between groups. CALVI showed some differentiation (Experts vs. Crowdworkers) but was less sensitive than the qualitative measures.
• Item Discrimination: Bayesian IRT analysis revealed that Sketching and Critique items had high discrimination values (median ~1.92 for Critique, ~1.52 for Sketching), indicating they effectively distinguish between participants of varying ability levels.
• Correlation: The qualitative assessments (Sketching and Critique) showed moderate correlation with each other ($r \approx 0.59$) and with CALVI ($r \approx 0.52-0.57$), but weak correlation with Mini-VLAT ($r \approx 0.30$). This suggests that while related, these assessments capture distinct skill sets.
• Qualitative Observations:
  • Crowdworkers frequently expressed frustration with manual sketching and relied on familiar, often inappropriate, chart types (e.g., bar charts for all data).
  • Experts demonstrated nuanced design choices (e.g., avoiding edge crossings in node-link diagrams) and deeper contextual reasoning.
  • Crowdworkers often produced "shallow critiques" focusing on basic rules (e.g., "needs a title") rather than design empathy or data literacy issues.

Significance and Claims
The authors claim that qualitative, multimodal assessments are a promising complement to existing visualization literacy assessments, particularly when the goal is to distinguish between individuals with visualization training and those without. They argue that while multiple-choice tests are efficient for basic comprehension, they are insufficient for evaluating the construction and critique skills essential for advanced visualization work.

The paper modestly positions these tools not as a replacement for all existing tests, but as a necessary addition to the toolkit, especially for contexts requiring high-fidelity skill differentiation (e.g., selecting participants for experiments involving visualization design). The authors acknowledge the increased burden of administration and grading as a limitation but suggest that the insights gained into higher-order skills justify the effort. They also note that future work should explore the use of Large Language Models (LLMs) for grading, though they caution regarding current limitations in handling holistic rubrics and dialect bias.