Technical Summary: Towards Just-in-Time Adaptive Feedback via Knowledge-Grounded LLMs

Problem Statement

In STEM education, particularly in physics, students frequently engage in "Recursive Plug-and-Chug" (or "Formula Hunting"), where they focus on surface features of problems rather than deep logical structures. While "strategy writing"—asking students to articulate their reasoning without numbers or formulas—is a proven intervention to counteract this, it faces a scalability bottleneck. Providing high-quality, formative, Just-in-Time (JiT) feedback to thousands of students in real-time is prohibitively time-consuming for human instructors. Although Large Language Models (LLMs) offer a pathway to scale, existing applications often lack domain-specific grounding or fail to move beyond reactive chatbot interactions that do not effectively disrupt novice problem-solving heuristics.

Methodology

The authors propose a framework that integrates domain-specific expert knowledge into LLMs to provide adaptive, JiT feedback based on student strategy essays. The methodology consists of four key phases:

1. Data Collection & Expert Annotation:

   • The system operates within a large-scale calculus-based physics course ($N > 1,000$).
   • Students submit written strategy essays (minimum 50 words, no formulas) for quiz problems.
   • Human experts (faculty and instructors) manually annotate a small representative set (50 examples) of these essays. They link specific reasoning patterns to error types (e.g., direction errors, position errors, or combined errors) and write tailored feedback. Crucially, experts identify missing concepts even when the student's final answer is correct.

2. Knowledge Grounding & Few-Shot Learning:

   • The LLM is grounded using the expert-annotated data as few-shot examples.
   • Instead of simple classification, the system employs a Chain-of-Thought (CoT) reasoning approach. It prompts the LLM to identify both a primary and a secondary most likely error type (e.g., "correct," "direction," "position," "position-direction").
   • This expanded label space allows the LLM to generate more robust feedback that addresses potential misconceptions regardless of the final answer's correctness.

3. Adaptive Intervention Design:

   • Non-Intrusive Feedback: The system is explicitly constrained not to provide direct solutions or correct answers. Instead, it directs attention to underlying concepts and reasoning logic.
   • JiT vs. Post-Hoc: While post-hoc feedback was used to survey student preferences, the core intervention is JiT, occurring while students are solving the quiz.
   • Preference Findings: A survey revealed that regardless of their actual knowledge level (novice vs. advanced), students overwhelmingly preferred feedback targeted at the "novice" level. This suggests that LLMs currently struggle to generate pedagogically effective "advanced" feedback, and increasing vocabulary complexity may hinder learning.

4. Deployment:

   • The framework was deployed in a Spring 2026 semester quiz. Students interacted with the LLM via an integrated interface, receiving feedback based on their initial strategy essays. They were encouraged to iteratively revise their essays and engage in conversations with the LLM to refine their understanding.

Key Contributions

The paper introduces three novel contributions:

1. Proactive Reasoning Analysis: Unlike reactive chatbots that may succumb to student "formula-hunting," this system proactively analyzes the reasoning logic within strategy essays to guide students toward deep structural understanding.
2. Expert-Grounded LLMs: The framework grounds LLMs with carefully annotated, domain-specific knowledge from human experts, offering a significant advantage over generic persona-based prompt engineering.
3. Scalable Pedagogical Intervention: The system provides a robust, automated intervention that maintains pedagogical soundness at a massive scale ($N > 1,000$), bridging the gap between the benefits of strategy writing and the reality of large enrollment courses.

Experimental Results

The study evaluated the framework's impact on a specific quiz question over four consecutive semesters:

• Performance Improvement: The deployment of the JiT feedback framework resulted in an 80% improvement in student performance compared to previous semesters. The error rate dropped from over 50% in prior semesters to less than 10% with the intervention.
• Error Prediction Accuracy: While the LLM's accuracy in predicting error types from essays alone was modest (60.61% accuracy with few-shot and secondary labels), the feedback remained highly effective. This is attributed to the domain grounding, which allowed the LLM to address missing concepts even if the specific error classification was imperfect.
• Student Engagement: Approximately 77.6% of students reported the feedback was helpful. Qualitative analysis showed students valued the feedback for both concept verification and clarifying overlooked aspects.
• Iterative Learning: About 20% of students voluntarily engaged in iterative conversations with the LLM. In these cases, the proportion of essays classified as "correct" rose from 42.79% (initial) to 71.64% (final). Ultimately, 91.43% of students who engaged in this iterative process solved the quiz correctly.
• Revision vs. Time: Analysis of conversational data revealed that the degree of revision (change in word count) was a statistically significant predictor of essay correctness ($r=0.18, p<0.001$), whereas the time spent (reflective latency) showed negligible correlation. This suggests that active reformulation of reasoning is more critical than passive time spent.

Significance and Claims

The authors claim that this work demonstrates the potential of LLMs to serve as sophisticated pedagogical tools that facilitate a shift from misconceptions to correct understanding. The significance lies in:

• Validating JiT Interventions: Proving that knowledge-grounded LLMs can effectively disrupt "formula-hunting" heuristics by forcing students to reflect on deep structures.
• Scalability: Demonstrating that high-quality, adaptive feedback can be delivered to over 1,000 students simultaneously, a feat previously impossible with human instructors.
• Pedagogical Utility: Showing that even with imperfect error classification, the combination of expert-grounded knowledge and iterative student-LLM conversation leads to substantial learning gains.

The paper remains modest regarding its limitations, acknowledging that error type prediction from essays is non-trivial, the intervention was limited to a single quiz, and the system currently lacks memory for efficient multi-turn conversations. It also notes that the LLM struggles to generate feedback suitable for advanced learners, who paradoxically prefer novice-level scaffolding.