Using Vision + Language Models to Predict Item Difficulty

This study demonstrates that a multimodal approach combining vision and language models (GPT-4.1-nano) to analyze both visualization images and text features significantly outperforms unimodal methods in predicting the difficulty of data literacy test items for U.S. adults, achieving a mean absolute error of 0.224.

The Gist

Imagine you are a teacher trying to write a quiz about reading charts and graphs. You want to make sure your questions aren't too easy (boring) and aren't too hard (frustrating). But how do you know how hard a question is before you actually give it to students? Usually, you have to wait, give the test, and look at the results.

This paper is about teaching a super-smart computer (an AI) to guess how hard a question will be just by looking at it, without needing to wait for students to take the test.

Here is the breakdown of how they did it, using some everyday analogies:

The Goal: The "Difficulty Crystal Ball"

The researchers wanted to build a "crystal ball" that could look at a data visualization question (which has a picture of a chart and some text) and predict: "What percentage of people will get this right?"

If the AI predicts 90% will get it right, it's an easy question. If it predicts 10%, it's a nightmare question.

The Three "Detectives"

To figure out the best way to make these predictions, they trained three different types of AI "detectives" using a powerful model called GPT-4.1-nano. Think of them as three different ways to solve a mystery:

1. The "Text Detective" (Text-Only):

   • What it does: This detective only reads the question and the answer choices. It ignores the picture entirely.
   • The Analogy: Imagine trying to guess how hard a puzzle is just by reading the instructions, without ever seeing the picture on the box.
   • Result: It was okay, but not great. It missed the visual clues that make a chart confusing.

2. The "Art Critic" (Vision-Only):

   • What it does: This detective only looks at the chart or graph. It ignores the question text.
   • The Analogy: Imagine looking at a messy, cluttered painting and trying to guess how hard it is to understand, without knowing what the artist was trying to ask you about it.
   • Result: It was better than the Text Detective, but still missed the context. A simple chart can be very hard if the question asks a tricky question about it.

3. The "Super Detective" (Multimodal):

   • What it does: This detective looks at both the picture and the text together. It sees how the question interacts with the chart.
   • The Analogy: This is like a detective who reads the instructions and looks at the puzzle pieces. They understand that a simple chart becomes a nightmare if the question asks you to find a tiny, hidden detail.
   • Result: This was the winner. By combining both views, the AI made the most accurate guesses.

The Results: Who Won the Race?

The researchers tested these detectives on a bunch of real questions. They measured "mistakes" using a score called MAE (Mean Absolute Error). Think of this as the "distance" between the AI's guess and the real answer. The lower the number, the better.

• Text Detective: Made big mistakes (Score: 0.338).
• Art Critic: Made medium mistakes (Score: 0.282).
• Super Detective: Made the smallest mistakes (Score: 0.224).

The "Super Detective" proved that to understand how hard a question is, you can't just look at the words or just look at the picture. You have to see how they work together.

The Final Test: The "Blind" Challenge

To make sure the AI wasn't just memorizing the answers, they gave the "Super Detective" a brand new set of questions it had never seen before (the "held-out test set").

Even on these new, unseen questions, the AI did a very good job. It was close enough to the real results that the researchers are confident this technology could actually be used in the real world.

Why Does This Matter? (The Big Picture)

Why should we care if a computer can guess test difficulty?

• Supercharging Test Makers: Imagine a teacher or a test creator who can write a question, show it to the AI, and the AI says, "Hey, this chart is too cluttered, or this question is too tricky. Let's fix it before we print 10,000 copies."
• Better Learning: It helps create fairer tests that actually measure what people know, rather than just how confusing the chart looks.
• Speed: Instead of waiting months to see how students perform on a new test, we can get instant feedback on how hard the questions are.

The One Catch

The AI had one small problem: It couldn't read a specific type of image file (SVG) used in some of the test questions. For those few images, the AI just guessed "50/50" (like flipping a coin). This wasn't perfect, but for the rest of the test, it worked amazingly well.

In short: By teaching an AI to look at both the picture and the words, we can now predict how hard a data question is. It's like giving test-makers a crystal ball to build better, fairer, and more effective quizzes.

Technical Summary

Here is a detailed technical summary of the paper "Using Vision + Language Models to Predict Item Difficulty" by Samin Khan.

1. Problem Statement

The paper addresses the challenge of Data Visualization Literacy (DVL) assessment. While DVL is a critical skill, developing standardized, reliable, and well-calibrated test items is difficult and time-consuming. A core component of psychometric evaluation is predicting item difficulty (defined as the proportion of test-takers answering correctly).

The research seeks to determine if Large Language Models (LLMs), specifically multimodal ones, can automate this prediction by analyzing the intrinsic features of test items. The study formulates two primary research questions:

1. Which features predict item difficulty more effectively: visual features (the chart/image) or textual features (the question and answer options)?
2. Does a multimodal approach (combining both visual and textual features) offer superior predictive power compared to unimodal approaches?

2. Methodology

Data Preparation

• Dataset: The study utilized a dataset collected by Verma and Fan (2025) containing responses from U.S. adults and college students to items from five DVL assessments (WAN, GGR, BRBF, VLAT, CALVI).
• Target Variable: Item difficulty was calculated as the mean of incorrect responses (0 = everyone correct, 1 = everyone incorrect). The study modeled easiness ($1 - \text{difficulty}$), representing the proportion of correct answers.
• Data Split: The data was split into an 80% validation set and a 20% held-out test set.
• Filtering: For the modeling phase, a subset of 154 items from the validation set was selected, specifically filtering for items with .png images to ensure compatibility with the chosen API.

Modeling Approach

The study employed GPT-4.1-nano via the OpenAI API. The authors utilized Pydantic models to enforce structured JSON output for reliable data extraction. Three distinct modeling strategies were implemented:

1. Text-Only Model: Input included only question_text and possible_responses.
   • Prompt Focus: Cognitive task type, question clarity, information integration, number of options, distractor plausibility, and format consistency.
2. Vision-Only Model: Input included only the image_url.
   • Prompt Focus: Chart type, axis clarity, data encoding, readability, clutter level, data series count, and overall visual complexity.
3. Multimodal Model: Input included the image_url, question_text, and possible_responses.
   • Prompt Focus: A comprehensive analysis of the interaction between visual elements, textual demands, and answer option quality.

Evaluation Metrics

• Validation: Mean Absolute Error (MAE) was calculated between the predicted easiness and the actual easiness for the 154-item validation subset.
• External Testing: The best-performing model was applied to a held-out test set of 46 items. Performance was evaluated via a competition platform using Mean Squared Error (MSE).

3. Key Results

Validation Performance (154 items)

The multimodal approach significantly outperformed unimodal approaches:

• Multimodal (Vision + Text): MAE = 0.2239 (Best)
• Vision-Only: MAE = 0.2819
• Text-Only: MAE = 0.3382

Distribution Analysis:

• The Vision-only model tended to overestimate easiness (peak around 0.85–0.9).
• The Text-only model showed a wide spread with a cluster around low easiness (0.25).
• The Multimodal model produced a more centered distribution, suggesting it successfully captured the interplay between clear visuals and complex questions (and vice versa).

Held-Out Test Set Performance

The multimodal model was applied to the external test set (46 items):

• Handling Limitations: 6 items were .svg files, which the API could not process. These were assigned a default prediction of 0.5 (chance performance).
• Final Metric: The model achieved a Mean Squared Error (MSE) of 0.10805 on the remaining 40 .png items.

4. Key Contributions

1. Demonstration of Multimodal Superiority: The study provides empirical evidence that combining visual and textual context is essential for accurately predicting the cognitive load and difficulty of data visualization questions. Neither modality alone is sufficient.
2. Automated Psychometrics: It validates the feasibility of using off-the-shelf multimodal LLMs (GPT-4.1-nano) for psychometric analysis, potentially replacing or augmenting traditional, labor-intensive item calibration processes.
3. Insight into Cognitive Demands: The results suggest that item difficulty arises from the interaction between the visual representation and the textual query, rather than the complexity of either in isolation.

5. Limitations

• Format Constraints: The inability to process .svg files directly forced a fallback strategy (defaulting to 0.5) for 6 test items, likely negatively impacting the final MSE.
• Proprietary Reliance: The study relies on a single, proprietary LLM, limiting generalizability across different model architectures.
• Sample Size: The validation subset (N=154) is relatively small for robust deep learning evaluation.
• Point Predictions: The model provides a single point estimate without measures of prediction uncertainty, which is crucial for practical test development.

6. Significance and Future Implications

This research demonstrates that multimodal LLMs can effectively automate the pre-calibration of test items in data visualization literacy.

• Efficiency: It can accelerate test development by predicting item difficulty before human pilot testing.
• Design Guidelines: By analyzing why an item is difficult (via the LLM's feature analysis), educators and designers can create better visualizations and clearer questions.
• Future Work: The authors suggest exploring SVG-to-PNG conversion, fine-tuning specific LLM architectures, and comparing these AI-driven methods against traditional Item Response Theory (IRT) models.

In conclusion, the paper establishes a strong baseline for using AI to understand and predict the cognitive demands of data visualization tasks, bridging the gap between computer vision, natural language processing, and educational psychometrics.