Detecting Knowledge Gaps from Conversational AI Interactions Using Curriculum Prerequisite Graphs

Technical Summary: Detecting Knowledge Gaps from Conversational AI Interactions Using Curriculum Prerequisite Graphs

Problem Statement
Large-scale online courses generate vast interaction logs from conversational AI teaching assistants, yet these data remain largely untapped for individualized diagnostic purposes. While existing knowledge gap detection methods rely on structured item responses (e.g., correct/incorrect answers to assessments) aligned with skill taxonomies, they cannot process free-form conversational questions. These unstructured questions are difficult to analyze because they lack explicit mapping to curriculum topics, and the diagnostic signal (student struggle) is implicit. The challenge lies in transforming raw student questions into curriculum-grounded signals that reflect genuine topic-level knowledge gaps without requiring a pre-existing item bank or per-question correctness assessment.

Methodology
The authors propose a five-step pipeline that bridges conversational logs and curriculum structure using a few-shot text classifier and a prerequisite knowledge graph.

1. Knowledge Graph Construction:

   • Source: 26 lesson PDFs from a graduate-level AI course.
   • Process: A domain-expert instructor used GPT-4 in a three-stage protocol to extract 54 distinct teachable topics and 47 directed prerequisite edges (e.g., Semantic Networks → Frames).
   • Validation: Ambiguous or bidirectional relationships were discarded to ensure a defensible, instructor-auditable structure. This graph defines the closed label space for the pipeline.

2. Question Classification (Hypothesis H1):

   • Labeling: GPT-4 was used to generate training labels for 1,046 unique student questions against the 54-topic space (plus an "unknown" class).
   • Model: A FastFit classifier (a few-shot text classification model using batch contrastive learning on sentence embeddings) was trained on the LLM-labeled data.
   • Architecture: The model uses a frozen all-mpnet-base-v2 sentence-embedding backbone.
   • Data Handling: To address class imbalance (median 5 samples per topic, max 89), the "unknown" class was downsampled, and low-frequency topics were augmented with paraphrases derived from their knowledge graph definitions. The final training pool consisted of 737 items across 43 effective classes (42 topics + 1 unknown).

3. Survey Validation (Hypothesis H2):

   • Metric: The pipeline compares the volume of assistant questions per topic against self-reported difficulty from a mid-semester Qualtrics survey ($n=280$).
   • Alignment: 28 topics were manually matched between the survey list and the knowledge graph nodes.
   • Analysis: Spearman's rank correlation ($\rho$) was computed to test if high question volume correlates with high self-reported difficulty.

Key Results

• Classification Accuracy (H1): The FastFit classifier achieved 80.0% accuracy and a macro F1 of 0.669 on a 70-question human-annotated gold set (43 classes). This performance slightly exceeded using GPT-4 directly as an inference-time classifier (78.6% accuracy).
  • Error Analysis: Misclassifications clustered into three patterns: surface-keyword traps, confusion between adjacent reasoning topics (e.g., Analogical Reasoning vs. Problem Reduction), and vague questions triggering the abstention class.
• Survey Alignment (H2): A statistically significant positive correlation was found between topic-level question volume and self-reported difficulty ($\rho = 0.491, p = 0.008$) across the 28 matched topics.
  • Observation: Topics with the highest question volume (e.g., Analogical Reasoning, Generalization) generally aligned with topics students reported as most difficult.

Significance and Claims
The paper claims that conversational AI interaction logs, when mapped onto a curriculum prerequisite knowledge graph, provide actionable signals about topic-level knowledge gaps. The significance of this work lies in:

1. Actionable Granularity: By treating topics (rather than individual questions) as the analytic unit, the system produces signals that match the granularity at which instructors can intervene.
2. Curriculum Grounding: The approach validates student struggle against an independent, instructor-authored curriculum structure rather than relying solely on engagement proxies or retention metrics.
3. Feasibility without Item Banks: The pipeline demonstrates that diagnostic signals can be extracted from free-form text using few-shot learning and LLM-extracted graphs, bypassing the need for carefully designed item banks or Bayesian knowledge tracing models.

Limitations and Future Directions
The authors explicitly note three limitations:

1. The 20% classification error rate introduces label noise, likely attenuating the measured correlation.
2. The evaluation is limited to a single graduate-level AI course; replication across disciplines is required.
3. The current results are aggregate (cohort-level) and do not yet identify specific individual students struggling with specific topics.

Future work aims to extend this pipeline to:

• Per-student gap detection: Flagging students whose question distributions diverge anomalously from the cohort.
• Prerequisite propagation: Using the graph edges to trace downstream difficulties back to upstream root-cause prerequisites for individual students.
• Intervention studies: Testing a closed-loop dashboard where curriculum-grounded remediation prompts are surfaced to instructors or students.