Temporal Stability and Few-Shot Prompting in Math Task Assessment

Technical Summary: Temporal Stability and Few-Shot Prompting in Math Task Assessment

Problem Statement
As AI tools are increasingly integrated into educational contexts, there is a critical need to understand their temporal stability and responsiveness to prompt engineering. While Large Language Models (LLMs) evolve rapidly, their performance on specialized educational tasks—specifically the classification of mathematical tasks by cognitive demand—may not improve or may even degrade over time due to "temporal drift" and optimization trade-offs. Furthermore, it remains unclear whether prompt engineering techniques, such as few-shot prompting, can compensate for these instabilities or outperform passive reliance on model version updates. This study addresses the gap in knowledge regarding how general-purpose and education-specific AI tools perform over time and whether few-shot prompting can enhance their ability to classify math tasks using the Task Analysis Guide (TAG).

Methodology
This longitudinal study evaluated two AI tools: a general-purpose model (Google Gemini) and an education-specific model (Coteach). The assessment focused on the tools' ability to classify 12 pre-classified middle-school mathematics tasks into four cognitive demand categories defined by the TAG: Memorization, Procedures without Connections, Procedures with Connections, and Doing Mathematics.

The study employed a three-phase design:

1. Time 1 (Baseline): Both models classified the 12 tasks using zero-shot prompting (standard instructions and the TAG framework).
2. Time 2a (7 Weeks Later): The models were re-tested with the same zero-shot prompts. During this interval, Gemini underwent a major version update (from 2.5 Flash to 3 Flash), while Coteach received regular, unannounced updates based on its underlying model (Anthropic's Claude).
3. Time 2b (Few-Shot Prompting): Both models were re-tested using few-shot prompting. This condition included two exemplar tasks for each of the four cognitive demand categories (8 exemplars total), formatted in LaTeX to reduce processing strain. The exemplars were semantically similar to the test set but distinct in wording.

Key Results

• Temporal Stability: The models exhibited distinct patterns of instability over the seven-week interval.
  • Gemini: Overall accuracy remained stable at 58% (7/12 tasks). However, the specific tasks classified correctly shifted; three previously correct classifications became incorrect, and three previously incorrect ones became correct. This indicates "prompt drift" where the model's behavior changes despite stable aggregate metrics.
  • Coteach: Overall accuracy declined significantly from 75% to 50% (a 25-percentage-point drop), representing a loss of capability rather than a redistribution of errors.
• Impact of Few-Shot Prompting: Few-shot prompting improved performance for both models, demonstrating that prompt optimization can be more effective than model updates.
  • Gemini: Accuracy increased from 58% to 67% (+8 percentage points).
  • Coteach: Accuracy recovered from 50% back to 75% (+25 percentage points), fully offsetting the temporal degradation observed in Time 2a.
• Category-Specific Performance: Accuracy varied by cognitive demand category. "Procedures without Connections" was the most accurately classified category (avg. 94%), while "Doing Mathematics" was the least (avg. 33%). Few-shot prompting did not uniformly improve all categories; "Doing Mathematics" tasks remained difficult to classify correctly, suggesting a limitation in how LLMs handle non-algorithmic, complex thinking tasks.

Key Contributions

• Evidence of Temporal Instability: The study provides empirical evidence that AI performance on educational tasks is not static. It distinguishes between "visible degradation" (overall accuracy drops, as seen in Coteach) and "invisible drift" (task-level shifts with stable overall accuracy, as seen in Gemini), the latter of which may go undetected by standard benchmark reports.
• Efficacy of Prompt Engineering: The findings demonstrate that few-shot prompting is a viable and potent strategy for mitigating temporal drift. In the case of Coteach, prompt engineering completely reversed performance losses attributed to model updates.
• Tool Comparison: The study highlights different response patterns between general-purpose and education-specific tools. While the education-specific tool showed greater vulnerability to temporal drift, it also demonstrated higher sensitivity and responsiveness to domain-specific prompting techniques compared to the general-purpose model.

Significance and Claims
The paper claims that for specialized educational tasks, active prompt optimization is a more reliable and effective strategy than passive reliance on model improvements. The results challenge the assumption that newer AI model versions inherently offer better performance for specific educational applications. Instead, the study suggests that educators and researchers must account for temporal instability and prioritize prompt engineering (specifically few-shot prompting with relevant exemplars) to ensure consistent and accurate AI tool performance. The authors note that while few-shot prompting improved overall accuracy, it did not resolve fundamental challenges in classifying high-cognitive-demand "Doing Mathematics" tasks, indicating that current LLM architectures may still struggle with tasks requiring complex, non-algorithmic reasoning.