Technical Summary: FoundationalASSIST

Problem Statement

The deployment of Large Language Models (LLMs) for adaptive testing and personalized tutoring relies on the assumption that these models can understand student cognition. However, existing educational datasets are fundamentally incompatible with the natural language reasoning capabilities of LLMs. Traditional benchmarks, such as ASSISTments, EdNet, and Junyi Academy, represent student interactions as tuples of opaque identifiers and binary correctness labels. This format strips away the semantic content required for LLMs to reason about why a student struggles, what specific misconception they hold, or which wrong answer they selected. Consequently, it is impossible to rigorously evaluate whether LLMs can perform foundational educational tasks like knowledge tracing (predicting student performance) or pedagogical grounding (understanding assessment properties) without a dataset that preserves the full textual richness of the learning process.

Methodology

The authors introduce FoundationalASSIST, the first English-language educational dataset designed to bridge this gap. The dataset is derived from the ASSISTments platform, covering interactions from January 2019 to July 2024.

Dataset Construction

FoundationalASSIST comprises 1.7 million interactions from 5,000 unique students across 3,395 mathematics problems. Unlike prior datasets, it provides:

1. Complete Question Text: Full problem statements with mathematical notation converted to readable text.
2. Actual Student Responses: Instead of binary labels, the dataset records the specific option selected (for multiple-choice) or the exact numerical/textual answer written (for fill-in-the-blank).
3. Distractor Selection Data: Records of which specific wrong answers students chose, enabling the analysis of misconception patterns.
4. Curriculum Alignment: All problems are mapped to Common Core State Standards (K-12), providing structured knowledge component codes and descriptions.

The dataset includes 64.4% fill-in-the-blank problems and 23.4% single-select multiple-choice problems, with an overall correctness rate of 61.5% (balanced to 51.3% for the evaluation set).

Benchmark Tasks

The paper defines six zero-shot evaluation tasks divided into two tracks:

1. Knowledge Tracing:
   • Question-Level Prediction: Predicting whether a student will answer a specific question correctly based on their history.
   • Cognitive Student Modeling: Predicting the exact response (option letter or numerical value) a student will generate.
2. Pedagogical Grounding:
   • Difficulty Comparison: Identifying which of two problems is more difficult.
   • Discrimination Comparison: Identifying which problem better distinguishes between high- and low-ability students (based on Item Response Theory parameters).
   • Most Common Distractor Prediction: Predicting which wrong answer students select most frequently.
   • Least Common Distractor Prediction: Predicting which wrong answer students select least frequently.

Experimental Setup

The authors evaluated four frontier LLMs: GPT-OSS-120B, Llama-3.3-70B-Instruct, Qwen3-Next-80B-Instruct, and Qwen3-Next-80B-Thinking. All experiments utilized zero-shot prompting with structured inputs containing student history and target questions. The evaluation focused on inherent capabilities derived from pretraining, excluding fine-tuning or few-shot learning.

Key Results

Knowledge Tracing Performance

• Marginal Improvement over Baseline: All models barely exceeded a trivial baseline of 51.3% (always predicting "correct"). The best model, GPT-OSS-120B, achieved 56.2% accuracy, a gain of only ~5 percentage points.
• Optimistic Bias: Models exhibited a systematic bias toward predicting correctness. For instance, Llama-3.3-70B achieved 85.4% accuracy when students were correct but only 12.6% when they were incorrect. This suggests models fail to recognize struggling students.
• Cognitive Modeling: While predicting exact responses is harder, models showed some capability, particularly for fill-in-the-blank problems (40–50% accuracy), where random chance is near zero. However, performance dropped significantly for incorrect answers.
• Context Length: Increasing the history length from 50 to 400 interactions did not improve performance, indicating LLMs are not effectively leveraging long-term interaction sequences to build student models.

Pedagogical Grounding Performance

• Difficulty Estimation: Models demonstrated genuine competence, with GPT-OSS-120B reaching 68.6% accuracy (18.6% above random). Performance improved with larger difficulty gaps (80.0% for large gaps).
• Discrimination Failure: Every model performed below random chance (50%) on discrimination comparison. The best performer, Qwen3-Next-80B-Thinking, achieved 46.9%. This indicates a fundamental failure to understand what makes an item diagnostic.
• Distractor Prediction: Models showed partial success in identifying the most common distractor (up to 47.9% accuracy vs. 35.8% baseline) but failed significantly at identifying the least common distractor (dropping to 20.5% for the Thinking variant).
• Reasoning Effects: The "Thinking" variant of Qwen3 showed mixed results: it slightly improved discrimination accuracy but significantly degraded distractor prediction accuracy, suggesting extended reasoning may introduce overthinking on concrete pattern-matching tasks.

Significance and Claims

The paper claims that FoundationalASSIST establishes a necessary foundation for evaluating LLMs in education, moving beyond the limitations of ID-based datasets. The authors assert that:

1. Current LLMs are insufficient for personalized learning: Despite their reasoning capabilities, frontier models cannot reliably predict student performance or identify struggling learners without significant fine-tuning or architectural changes.
2. A specific gap exists in pedagogical understanding: While LLMs can intuit problem difficulty from surface features, they lack the metacognitive reasoning required to understand item discrimination (how an item separates ability levels) and the distribution of student errors.
3. Data scarcity is a primary bottleneck: The lack of datasets containing actual student responses and error patterns in natural language has hindered progress. FoundationalASSIST provides the first resource to enable research into cognitive student modeling and misconception analysis.
4. Caution in deployment: The systematic optimistic bias and failure in discrimination tasks suggest that deploying current LLMs for high-stakes adaptive testing or personalized tutoring without human oversight is premature and potentially harmful.

The authors position FoundationalASSIST not as a solution to these problems, but as a diagnostic tool to identify specific failure modes (such as the inability to predict incorrect answers or understand discrimination) that future research must address to make LLMs viable for educational applications.