Should There be a Teacher In-the-Loop? A Study of Generative AI Personalized Tasks Middle School

1. Problem Statement

The study addresses the tension between the potential of Generative AI (GenAI) to personalize instruction at a fine-grained level and the practical constraints of teacher time and control. While GenAI (specifically Large Language Models or LLMs) can theoretically tailor educational tasks to individual student interests (context personalization), it is unclear how efficient and effective this process is when a teacher remains "in-the-loop" to oversee, modify, or generate these tasks.

Key research gaps identified include:

• Efficiency: Commercial tools claim to save teacher time, but empirical evidence on the actual time cost of co-designing personalized tasks with LLMs is lacking.
• Grain Size: There is a conflict between the "broad grain size" personalization a teacher can manage for a whole class versus the "small grain size" (specific to individual student interests) that students prefer.
• Depth vs. Surface: LLMs often generate superficial connections to student interests. It is unknown how effectively teachers can monitor and correct for "depth" (realism of the scenario and mathematical validity) when using LLMs.
• Iterative Growth: Little is known about how teachers' prompting strategies and efficiency evolve as they gain experience with LLMs in a real classroom setting.

2. Methodology

Participants and Setting:

• Teachers: 7 middle school mathematics teachers (mean experience = 11.9 years) from 6 diverse US schools.
• Students: 521 7th-grade students.
• Curriculum: The study utilized the Illustrative Mathematics (IM) curriculum, a free, open-source K-12 math program.
• Tools: Teachers used ChatGPT (GPT-4o) to modify existing IM problems.

Procedure:

1. Training: Teachers underwent a 2-hour training on personalization theory, bias, and ChatGPT prompting.
2. Task Generation: Teachers selected problems from Unit 4 (Proportional Relationships) and Unit 6 (Expressions, Equations) of the IM curriculum.
3. Co-Design Sessions: Teachers participated in 1.5-hour Zoom sessions where they used ChatGPT to rewrite problems to match specific student interests (e.g., sports, food, pop culture). Sessions were video-recorded, and interaction logs were captured.
4. Deployment: Personalized problems were assigned to students via the ASSISTments platform. Students could choose between different interest-based versions of the same problem.
5. Feedback Loop: After Unit 4, teachers reviewed student feedback and ratings before repeating the process for Unit 6.

Data Collection & Analysis:

• Quantitative:
  • Teacher Metrics: Time spent per problem, number of prompts used, and prompt categorization (inductive coding).
  • Student Metrics: 5-point Likert scales for "Interestingness" and "Closeness to interests" (n=52 problems rated).
  • Statistical Models: Mixed-effects linear regression models were used to analyze the relationship between prompts/time and student ratings, accounting for random intercepts (Teacher, Student, Problem).
• Qualitative:
  • Prompt Analysis: Coded for themes regarding grain size (broad vs. specific), ownership (teacher vs. AI decision), and depth (realism).
  • Student Feedback: Open-ended survey responses analyzed for reasons regarding interest levels and suggestions for improvement.

3. Key Contributions

• Empirical Time Costs: Provides rare empirical data on the actual time investment required for teachers to co-design personalized tasks with LLMs, challenging the "time-saving" narrative of commercial AI tools.
• The "Goldilocks Zone" of Prompting: Identifies a specific range of teacher interaction (3–5 prompts) that correlates with optimal student satisfaction, offering a quantifiable metric for effective prompting.
• Teacher Growth Trajectory: Documents that teacher efficiency decreased (time and prompts increased) from Unit 4 to Unit 6, contrary to typical learning curves, because teachers used reflection on student data to intentionally increase the quality and specificity of the personalization.
• Grain Size Mismatch: Highlights a critical disconnect: Teachers default to broad grain sizes (e.g., "sports") due to time constraints, while students strongly prefer small grain sizes (e.g., "Taylor Swift" or "Roblox").

4. Key Results

RQ1: Teacher Prompting Strategies

• Teachers made 415 additional prompts beyond the initial request (avg. 2.9 per problem).
• Common Moves:
  • Specific References: Teachers frequently asked the AI to swap generic terms for specific brands or celebrities (e.g., "Change Doritos to Flaming Hot Cheetos").
  • Realism Checks: A significant portion of prompts (139 total) focused on correcting depth issues, such as unrealistic numbers (e.g., half a coin) or implausible scenarios (e.g., buying curtain rods).
  • Ownership: Teachers oscillated between letting the AI decide on references and explicitly dictating them based on their knowledge of students.

RQ2: Student Feedback

• Interest Drivers: Students rated problems as interesting primarily when they contained specific popular culture references they personally liked.
• Disinterest Drivers: Students disliked problems when the specific reference was something they did not like (e.g., "I don't play Roblox").
• Depth Perception: Students rarely noticed or commented on mathematical or contextual inaccuracies (depth), focusing almost exclusively on surface-level interest (grain size).
• Preference: Students overwhelmingly requested changes to specific references rather than broad topic categories, indicating a desire for smaller grain sizes than teachers provided.

RQ3: Efficiency and Effectiveness

• Time Cost: Creating one personalized problem took an average of 4 minutes (246 seconds). A standard 10-problem set would take a teacher ~40 minutes, leaving no time to create multiple versions for different student interests.
• Prompt Efficiency: There is a non-linear relationship between prompts and ratings.
  • 3–5 Prompts: This range yielded the highest student ratings for interestingness and closeness.
  • <3 or >5 Prompts: Ratings declined. Accepting the first output (0–2 prompts) was suboptimal, while excessive prompting (>5) did not yield better results and wasted time.

RQ4: Growth from Unit 4 to Unit 6

• Decreased Efficiency: Contrary to expectations, teachers became less efficient in Unit 6.
  • Prompts increased from 3.2 to 4.6 per problem.
  • Time increased from 204s to 286s per problem.
• Increased Quality: Despite the time increase, student ratings for both interestingness and closeness significantly improved in Unit 6.
• Interpretation: Teachers used the reflection period between units to intentionally invest more time to address student feedback, prioritizing quality over speed.

5. Significance and Conclusion

The "Teacher In-the-Loop" Dilemma:
The study concludes that while a teacher in-the-loop is necessary for ethical oversight (checking for bias, inappropriate content, and mathematical errors) and professional learning, it creates a bottleneck.

• The Trade-off: The teacher-in-the-loop model forces a broad grain size personalization (class-level interests) because creating individual, fine-grained tasks for 500+ students is time-prohibitive.
• Student Impact: This broad approach fails to meet the specific preferences of many students, who desire highly specific cultural connections.

Implications for AI in Education:

1. Time Efficiency Myth: GenAI does not currently save teachers time when high-quality, personalized task creation is the goal. The "4-minute per problem" metric suggests that scaling this to full curricula is currently unfeasible without significant workflow changes.
2. Professional Development: Teachers need training not just on using AI, but on prompting strategies (specifically the 3–5 prompt "sweet spot") and how to balance depth vs. grain size.
3. Future Directions:
   • Multi-Agent Systems: The authors propose using autonomous AI agents to handle the "heavy lifting" of checking realism, bias, and cultural relevance, allowing the teacher to perform a final "quick check."
   • Student-Facing AI: For fine-grained personalization, systems where the AI interacts directly with the student (e.g., Khanmigo) may be more effective than teacher-mediated generation, provided safety and ethical safeguards are robust.

Final Verdict: Teachers should remain in-the-loop to ensure pedagogical soundness and ethical safety, but the current workflow is too inefficient to support the level of personalization students desire. Future research must focus on hybrid models (multi-agent systems) that preserve teacher agency while automating the granular details of personalization.