🤖 machine learning

Context-Aware Prediction of Student Quiz Performance with Multimodal Textbook Features

This paper demonstrates that incorporating lightweight linguistic and visual features from textbook review questions significantly improves the prediction of student quiz performance beyond what is achievable using prior student performance data alone.

Original authors: Samin Khan

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Samin Khan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess how well a student will do on a final test. Usually, teachers (or computer programs) look at the student's past homework scores to make that guess. It's like predicting how fast a runner will finish a race just by looking at their times in previous practice runs.

This paper asks a simple question: Does looking at the test itself help us make a better guess?

The author, Samin Khan, wanted to see if the actual words and pictures inside a quiz chapter could tell us something about how hard the test is, which might help predict a student's score even better than just looking at their past grades.

Here is the breakdown of the study using everyday analogies:

The Setup: The "CourseKata" Library

Think of the data as a giant library of online statistics textbooks (called CourseKata). Students read chapters, do practice exercises, and then take a quiz at the end.

The Data: The researcher looked at about 4,700 instances of students taking these quizzes.
The Goal: Predict the final quiz score (0 to 100%).
The Baseline: The "old way" of predicting was simply: "If a student got 80% on their practice problems, they will probably get around 80% on the quiz."

The Experiment: Adding "Context"

The researcher built a computer model to see if adding details about the quiz content improved the prediction. They treated the quiz content like a recipe:

Text Features: They analyzed the words in the questions. Did the questions use long, complicated words? Were there rare vocabulary words? Was the text very wordy? (Think of this as checking if the recipe uses fancy, hard-to-find ingredients).
Image Features: They analyzed the pictures in the textbook. They counted how many lines were in the drawings and how many distinct shapes or "regions" were visible. (Think of this as checking if the recipe has a complex diagram or just a simple sketch).

The Results: Two Different Scenarios

The researcher tested the model in two different ways, like two different game modes:

Game Mode 1: Predicting for a New Student in a Known Class

The Scenario: The computer has seen the quiz questions before (because other students took them), but it hasn't seen this specific student yet.
The Result: Adding the text and picture details helped a lot.
The Analogy: Imagine you know the menu at a restaurant perfectly. If you know a new customer usually orders spicy food, and you also know this specific menu has very spicy descriptions and pictures of hot peppers, you can predict their order much better than just knowing they like spicy food.
The Win: The model became 9.1% more accurate when it looked at the text and images. The text details were the biggest helper.

Game Mode 2: Predicting for a Brand New Chapter

The Scenario: The computer has never seen this specific chapter before. It has to guess how students will do on a totally new set of questions based only on what it learned from other chapters.
The Result: This was a mixed bag.
- Text: The wordy, complex language details still helped. It seems that if a chapter uses hard words, it's generally harder, and that rule applies even to new chapters.
- Images: The picture analysis hurt the prediction. The model got worse when it tried to use the image data.
The Analogy: Imagine you are trying to guess how hard a new puzzle is. You know that puzzles with long instructions are usually hard (Text = Good). But your method for judging pictures was just counting "lines" and "shapes." You might count a simple table as having 50 lines and a complex graph as having 5 lines. Because the "line counter" was too simple, it confused the computer, making the prediction for the new puzzle worse.

What Did the Computer Learn?

When the researcher looked at the "weights" (how much the computer cared about each factor), they found:

Word Count & Complexity: These were the most important clues. If a chapter has many words and complex vocabulary, the computer knew to expect different scores.
Images: The simple "line counting" method wasn't smart enough to understand what the pictures actually meant. A picture with many lines could be a simple chart or a confusing mess; the computer couldn't tell the difference.

The Bottom Line

This paper proves that knowing what the test looks like helps predict how students will do.

If you know the course material, looking at the words in the questions makes your predictions much sharper.
Looking at pictures helps if you are analyzing a specific, known chapter, but the current way of analyzing pictures (just counting lines) is too clumsy to help when predicting performance on new chapters.

The author concludes that to really understand student performance, we shouldn't just look at the student's history; we need to look at the "context" of the test itself, specifically the language used in the questions. However, we need smarter ways to analyze images before they can be useful for predicting performance on brand-new material.

Technical Summary: Context-Aware Prediction of Student Quiz Performance with Multimodal Textbook Features

Problem Statement
Educational data mining has traditionally focused on predicting student performance based on interaction histories and prior exercise scores. However, this approach assumes that assessment content is static or irrelevant to the prediction task. In reality, the difficulty of end-of-chapter quizzes varies significantly due to linguistic complexity, vocabulary rarity, and visual representation. This paper investigates whether incorporating lightweight, content-specific features extracted from textbook review questions can improve the prediction of student quiz scores beyond what is achievable using prior student performance alone. The study addresses two distinct generalization scenarios: predicting scores for new students within a known curriculum (Held-Out Student) and predicting scores for entirely new curriculum chapters (Held-Out Chapter).

Methodology
The study utilizes a dataset of 4,742 student-chapter observations derived from 562 class-student IDs across five college classes using the Statistics and Data Science: A Modeling Approach textbooks (ABC and ABCD versions) on the CourseKata platform. The target variable is the end-of-chapter quiz score (scaled 0–1).

The methodology involves a four-step pipeline:

Baseline Construction: A baseline predictor is established using the mean proportion of correct answers (avg_exercise_scores) from prior exercises attempted by the student within the same chapter.
Feature Extraction:
- Text Features: Extracted from chapter-review question wording using wordfreq. Metrics include average word length, mean word rarity (based on Zipf frequency), the count of complex words (low Zipf frequency), and total word count.
- Image Features: Extracted from textbook visuals using OpenCV. Metrics include Canny edge detection, the count of Hough line segments, and the count of external contours (visual regions). Chapter-level features are aggregated as averages per image and total image counts.
Modeling: The study compares four feature sets: (1) Base (prior performance only), (2) Base + Image, (3) Base + Text, and (4) Base + Image + Text. Ridge regression is employed to handle feature correlation and prevent overfitting, with regularization strength selected via cross-validation.
Evaluation:
- RQ1 (Held-Out Student): Five-fold grouped cross-validation where all chapters for a specific student are kept in the same fold. This tests generalization to new students on known content.
- RQ2 (Held-Out Chapter): Leave-chapter-out validation where the model is trained on eight chapters and tested on the ninth. This tests generalization to unseen curriculum content.

Key Results

Held-Out Student Prediction (RQ1): Adding content features significantly improves prediction accuracy. The full multimodal model (Base + Text + Image) achieves an $R^2$ of 0.4338 and a Mean Squared Error (MSE) of 0.02270, representing a 9.1% relative reduction in MSE compared to the baseline ( $R^2$ = 0.3775, MSE = 0.02496). Both text and image features contribute to this improvement, with text features showing a slightly larger individual impact (8.9% reduction) than image features (2.9% reduction).
Held-Out Chapter Prediction (RQ2): The results diverge based on feature type.
- Text Features: Incorporating text features improves performance, reducing MSE by 8.7% ( $R^2$ = 0.4119) compared to the baseline. This suggests that linguistic features capture transferable signals regarding assessment difficulty.
- Image Features: Models including image features perform worse than the baseline. The Base + Image model increases MSE by 15.5%, and the full multimodal model results in a negative $R^2$ (-0.1776) and a massive 82.9% increase in MSE.
Feature Importance: Standardized coefficient analysis of the Ridge model indicates that word count and complex-word count carry the largest weights among content features, followed by prior exercise scores. Image features (line counts, region counts) have smaller weights.

Significance and Claims
The paper claims that assessment content contains predictive signal that is distinct from a student's historical performance.

Context-Awareness: For operational settings involving known curricula, a context-aware model that integrates text and visual features provides a more accurate forecast of student performance than relying on past performance alone.
Transferability of Linguistic Signals: The study demonstrates that linguistic features (e.g., vocabulary rarity, word count) capture information about assessment difficulty that generalizes across different chapters, making them valuable for predicting performance on new content.
Limitations of Current Visual Features: The paper explicitly notes that current image features (edge counts, contour counts) are too coarse to be useful for new-content prediction. The authors argue that these features fail to encode the semantic complexity of visual relations (e.g., distinguishing between a complex statistical graphic and a simple table), suggesting that future work requires semantically meaningful representations (e.g., chart types, vision-language embeddings) to effectively model visual complexity.

Conclusion
The study concludes that student modeling should treat assessment content as an integral part of the prediction problem. While lightweight text features successfully enhance both new-student and new-chapter predictions, current simple image features do not yet offer the same benefit for generalizing to unseen curriculum. The authors emphasize that these findings are descriptive of predictive roles rather than causal estimates of question difficulty.