Exploring Teacher-Chatbot Interaction and Affect in Block-Based Programming

This study examines middle school teachers' perspectives on using LLM-based chatbots in block-based programming, identifying three distinct interaction profiles and highlighting both the potential benefits and risks to propose design recommendations for effective scaffolding.

The Gist

Imagine a classroom where every student has a super-smart, instant-answer robot friend sitting right next to them. This robot can write code, explain science, and solve puzzles in seconds. Sounds like a dream, right? But what happens when the teachers try to use this robot to help their students learn? Do they feel like they've found a magic wand, or do they feel like they've been handed a tangled ball of yarn?

This paper is a story about 11 teams of middle school teachers who tried out just such a robot (an AI chatbot) while learning to teach computer programming using "block-based" coding (like digital LEGO bricks). The researchers wanted to see how the teachers felt and what they thought about letting this AI into their classrooms.

Here is the breakdown of their adventure, explained simply:

1. The Setup: The "Robot Tutor"

The teachers were given a special coding environment called Stax.fun. It had a built-in chatbot with four different "hats" it could wear:

• The Code Writer: You type "make a cat jump," and it writes the code.
• The Q&A Expert: You ask, "Why isn't my light wave reflecting?" and it explains.
• The Coach: It gives you step-by-step hints if you're stuck.
• The Prompt Polisher: It helps you ask better questions.

The teachers had to use this tool to solve science puzzles involving light and sound waves.

2. The Three Types of Teachers (The "Personas")

Just like people react differently to a new video game, the teachers fell into three distinct groups based on how they interacted with the robot:

• The Explorers (The "Tinkerers"):
  • Analogy: Imagine a kid who gets a new Lego set and immediately starts building a spaceship, a castle, and a dragon, just to see what happens.
  • Behavior: These teachers were excited! They clicked every button, tried weird prompts, and loved seeing the robot's immediate results. They felt confident and curious. They didn't need much help.
• The Frustrated (The "Stuck in Traffic" Group):
  • Analogy: Imagine trying to drive a car where the steering wheel is sticky, the GPS speaks a language you don't know, and the map keeps disappearing.
  • Behavior: These teachers felt the robot was fighting them. The chat box covered their screen, buttons didn't work right, or the robot gave answers that didn't make sense. They felt confused, annoyed, and helpless. They kept asking the human researchers for help.
• The Mixed (The "Rollercoaster Riders"):
  • Analogy: Imagine a day where the weather is sunny, then rainy, then sunny again.
  • Behavior: These teachers had a mix of feelings. Sometimes they were happy when the robot worked; other times they were stuck and frustrated. They bounced back and forth between following instructions and trying to figure things out on their own.

3. The Big Debate: The "Cheat Code" vs. The "Training Wheels"

After the activity, the teachers had a deep conversation about whether they would let their students use this robot. Their opinions were split, like a debate between two coaches:

The "Yes, Let's Use It!" Team (The Benefits):

• Confidence Booster: They thought the robot would help shy students feel brave. "Hey, I can do this!"
• The "Prompting" Skill: They realized that in the future, knowing how to ask the robot the right question (prompting) is a superpower, just like knowing how to type was a superpower in the past.
• Teacher's Helper: It would act like a second teacher, helping students who are stuck so the human teacher isn't overwhelmed.

The "Wait a Minute..." Team (The Risks):

• The "Cheat Code" Fear: They worried students would just copy the robot's answers without learning how to build the code themselves. It's like letting a student use a calculator for every single math problem; they might get the right answer, but they won't learn the math.
• Loss of "Productive Struggle": Learning to code is hard. You have to get stuck, think, and struggle to solve it. That struggle is where the brain grows. If the robot fixes everything instantly, students miss out on that "aha!" moment.
• The "Black Box" Problem: If the robot writes the code, how does the teacher know if the student actually understands it?

4. What the Teachers Wanted (The "Design Wish List")

The teachers didn't say "ban the robot." Instead, they said, "We need to be the boss of the robot." They suggested three main changes:

1. The "Training Wheels" Mode: Teachers want to be able to turn off the robot's ability to write full code for beginners. They want the robot to only give hints or answer questions until the student learns the basics.
2. The "Differentiation" Dial: Teachers want to adjust the robot for different kids. For a student who reads slowly, the robot should speak the answer out loud. For an advanced student, the robot should give harder challenges.
3. The "Clear View" Window: The robot shouldn't cover the screen or hide the code. It needs to be transparent so the teacher and student can see exactly what is happening.

The Bottom Line

This study is like a test drive for a new car. The teachers realized that AI chatbots are powerful tools, but they aren't magic.

If you just hand a student a robot that does all the work, they might get the answer but lose the skill. But if you use the robot as a coach that guides them, helps them when they're stuck, and lets them drive the car themselves, it can be amazing.

The key takeaway? Teachers need control. They need to be able to decide when the robot helps and how much it helps, ensuring it builds up the students' brains rather than replacing them.

Technical Summary

1. Problem Statement

The rapid integration of Large Language Models (LLMs) and AI chatbots in K-12 education has created a tension between potential pedagogical acceleration and the risk of counterproductive learning outcomes. Key concerns include:

• Cognitive Offloading: Unrestricted use may bypass the "productive struggle" necessary for deep learning, particularly in programming where debugging and problem-solving are critical.
• Lack of Teacher Control: Most AI interactions are student-directed and occur outside teacher visibility, leading to uncertainty about plagiarism and the depth of student understanding.
• Design Gaps: Current "one-size-fits-all" AI tools often lack the scaffolding and instructor control mechanisms required for diverse classroom settings.
• Teacher Uncertainty: There is limited empirical evidence on how teachers emotionally and cognitively interact with LLMs in block-based programming environments, and how these interactions influence their intention to adopt such tools.

2. Methodology

The study employed a mixed-methods approach combining behavioral tracking, sentiment analysis, and thematic qualitative analysis.

• Participants: 25 middle-school science teachers (8 experienced lead teachers, 17 novice participant teachers) organized into 11 pairs.
• Environment: The study utilized Stax.fun, a block-based programming environment (Scratch-based) integrated with an LLM-powered chatbot. The chatbot offered four modes:
  • Code: Generate blocks from natural language.
  • Q&A: Answer conceptual questions.
  • Coach: Provide step-by-step guidance.
  • Prompts: Refine user prompts.
• Procedure:
  • Phase 1 (Day 1): Introduction to block-based programming (Snap!) without the chatbot.
  • Phase 2 (Day 2): A 30-minute "think-aloud" study where pairs completed wave-interaction coding tasks (light/sound waves) using Stax.fun.
  • Data Collection: Screen recordings, audio transcripts, and chatbot interaction logs (prompts, revisions, tab switches).
  • Post-Activity: 15-minute group discussions regarding perceptions, risks, and pedagogical strategies.
• Analysis Techniques:
  • Quantitative/Behavioral: Time-stamped annotation of tasks, emotion tracking (Positive, Neutral, Negative), and interaction patterns.
  • Qualitative: Hybrid thematic analysis (deductive and inductive) of discussion transcripts.
  • Theoretical Framework: Findings were mapped to Technology Acceptance Models (TAM) and Theory of Planned Behavior (TPB) to interpret adoption intentions.

3. Key Contributions

• Teacher Interaction Personas: The identification of three distinct teacher profiles based on affect and behavior:
  1. Explorer/Task-Focused: High positive affect, high curiosity, low help-seeking, and efficient task completion.
  2. Mixed: Fluctuating emotions, alternating between instruction-following and exploration, occasional frustration.
  3. Frustrated: High negative affect, repeated retries without progress, heavy reliance on facilitators, and feelings of confusion.
• Multi-Lens Emotional Analysis: A novel framework categorizing emotional triggers into three lenses:
  • L1 (UI/Interaction): System-level issues (e.g., prompts disappearing, chatbox obscuring code).
  • L2 (CT/Science): Cognitive challenges in understanding wave physics or programming logic.
  • L3 (Instruction-Following): Confusion regarding task framing or step-by-step guidance.
• Integrated Theoretical Model: A conceptual model combining TAM and TPB to explain how emotional states (affect) mediate the relationship between system design and teachers' behavioral intentions to adopt AI.

4. Key Results

A. Affective Responses and Behaviors

• Positive Emotions: Driven by immediate feedback, successful code generation, and the ability to experiment with multiple variants (e.g., "Excited," "Curious").
• Negative Emotions: Primarily triggered by L1 (UI) issues (e.g., chatbox taking up screen real estate, inability to resize) and L2/L3 (confusion over scientific concepts or unclear instructions).
• Correlation: Groups with high positive affect (Explorers) spent more time refining prompts and switching tabs, whereas Frustrated groups spent significant time on retries and seeking human help.

B. Teacher Perceptions: Benefits vs. Risks

• Benefits:
  • Student Self-Efficacy: Chatbots can boost confidence, especially for novices who feel stuck.
  • Prompting Skills: Teachers valued the opportunity to teach students how to effectively prompt AI, a critical future skill.
  • Differentiation: The tool can support diverse learners (e.g., English Language Learners, advanced students) by providing tailored explanations or challenges.
  • Teacher Support: Reduces teacher workload by acting as a "second pair of eyes" for debugging and providing an "answer key" for diverse student code.
• Risks:
  • Loss of Productive Struggle: Fear that students will skip the cognitive effort required for deep learning.
  • Superficial Understanding: Concerns that students will copy code without understanding the underlying logic.
  • Shift in Focus: The learning goal might shift from "how to code" to "how to prompt," potentially undermining foundational programming skills.

C. Pedagogical Preferences

• Scaffolding: Teachers overwhelmingly agreed that foundational programming skills must be taught before introducing the chatbot.
• Control Mechanisms: A primary request was the ability for teachers to toggle specific features (e.g., disabling the "Code Generation" tab while keeping "Q&A" active) to control the level of assistance.
• Transparency: Teachers expressed a need for the chatbot to explain why it generated specific code, not just what it generated.

5. Significance and Design Implications

This study provides critical insights for the design of AI tools in education, moving beyond generic adoption to specific, context-aware implementation:

1. Situational Awareness (SA) Design: Systems must improve "Level 2 SA" (Comprehension) by automatically commenting generated code, highlighting diffs, and explaining the rationale behind outputs to prevent confusion.
2. Instrumental Scaffolding: Chatbots should move beyond performative empathy (e.g., "I understand you are frustrated") to instrumental repair (e.g., "Let's rephrase your prompt" or "Here is a hint on the logic").
3. Teacher Control as a Feature: To mitigate risks of over-reliance, chatbots must include "pedagogical switches" allowing teachers to disable code generation or limit prompt counts based on the lesson's learning objectives.
4. Differentiation Support: Future tools should adapt to learner levels (e.g., visual/auditory outputs for accessibility) and allow teachers to calibrate support based on student proficiency.
5. Emotional Design: Recognizing that negative affect often stems from UI friction rather than content difficulty suggests that improving interface usability is as crucial as improving the AI's intelligence for teacher adoption.

Conclusion: The paper argues that for LLMs to be effective in block-based programming, they must be designed as collaborative partners that augment teacher expertise rather than replace it. Success depends on balancing automation with human control, ensuring transparency, and providing the scaffolding necessary to maintain "productive struggle" for students.