💻 computer science

Analyzing Middle School Students' Dialogue and Behaviors during Collaborative AI Chatbot Development Using Ordered Network Analysis

This study employs Ordered Network Analysis to reveal that middle school students' collaborative AI chatbot development is characterized by integrated sequences of explanation, testing, and refinement, which are positively associated with both higher-quality chatbot artifacts and stronger AI knowledge outcomes.

Original authors: Shan Zhang, Andres Felipe Zambrano, Xiaoyi Tian, Yukyeong Song, Anthony F. Botelho, Kristy Elizabeth Boyer, Maya Israel, Shiyan Jiang

Published 2026-07-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shan Zhang, Andres Felipe Zambrano, Xiaoyi Tian, Yukyeong Song, Anthony F. Botelho, Kristy Elizabeth Boyer, Maya Israel, Shiyan Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a group of friends try to build a robot that can talk. They aren't just coding in silence; they are talking, arguing, testing, and fixing things together. This is the world of Collaborative Learning, where the magic happens not just in the final product, but in the messy, back-and-forth conversation between the builders. In this specific corner of science, researchers are looking at AI Literacy—teaching kids how to understand, use, and even create Artificial Intelligence. It's like teaching someone not just how to drive a car, but how the engine works so they can build their own. The big question here is: When students work together to build an AI chatbot, what does their conversation look like? Does the way they talk and tinker with the code actually change how well they learn, or does it just matter if they get the right answer at the end?

This paper dives into that question by watching middle school students as they built their own chatbots using a web tool called "AI Made By You" (AMBY). The researchers didn't just look at the final chatbots to see if they were good or bad; they acted like detectives, analyzing the students' dialogue and actions second-by-second. They used a special method called Ordered Network Analysis, which is like mapping the rhythm of a dance. Instead of just counting how many times a student said "good job" or "try this," they looked at the sequence: Did they explain an idea, then test it, then fix it? Or did they just keep guessing without thinking?

The study found that the groups who built the highest-quality chatbots had a very specific rhythm. They were constantly looping through a cycle of explaining their reasoning, giving directives (telling each other what to do next), and repeating or confirming what they just did. It was a tight loop: "Here's why I think this will work," "Okay, let's try it," "Wait, it didn't work," "Let's fix it." They also argued, but in a good way—they disagreed while giving reasons, which helped them refine their ideas. In contrast, the groups with lower-quality chatbots were more focused on just asking procedural questions ("What button do I click?") or staring at the chatbot's output without deeply understanding why it behaved that way. They got stuck in isolated steps rather than flowing through the whole process.

Interestingly, the researchers checked if the "winning" groups were just the ones who already knew how to code. They weren't. Some students in the high-performing groups had never written a program before, while some in the low-performing groups had experience. This suggests that how they worked together mattered much more than what they already knew.

The paper also looked at how these patterns related to what the students actually learned about AI. The data suggests that students who engaged in those tight loops of explanation, testing, and refinement ended up with stronger knowledge of AI concepts. On the flip side, groups that spent a lot of time just giving orders or making suggestions without deep reasoning seemed to learn less. The researchers used statistical tests to show that these patterns weren't just random luck; the connection between the "explain-test-fix" rhythm and better learning scores was strong and meaningful.

So, the takeaway isn't that you need to be a coding genius to build a great AI. It's that the best learning happens when you and your partner are constantly talking through your ideas, testing them out, and fixing them together. It's the difference between two people just pushing buttons and two people building a machine while explaining to each other exactly how the gears turn. The study suggests that if we want students to really understand AI, we should encourage them to keep that conversation going, even when things get messy.

Technical Summary: Analyzing Middle School Students' Dialogue and Behaviors during Collaborative AI Chatbot Development Using Ordered Network Analysis

Problem Statement
As Artificial Intelligence (AI) education becomes integral to K–12 curricula, instructional practices increasingly involve students designing and developing conversational agents. While prior research has established the efficacy of these activities for learning outcomes and artifact quality, there is a significant gap in understanding the collaborative processes through which learning unfolds. Existing literature often relies on post-hoc assessments or final artifact evaluations, offering limited insight into the moment-by-moment interactional mechanisms—specifically how dialogue and system-oriented development actions are jointly organized over time. Furthermore, AI artifacts exhibit probabilistic and emergent behaviors that require learners to interpret unexpected outputs and negotiate shared explanations, a dynamic distinct from other collaborative domains like standard programming. This study addresses the need to characterize how collaborative interaction patterns during AI chatbot development relate simultaneously to design quality and AI knowledge outcomes.

Methodology
The study employed a mixed-methods approach grounded in the analysis of collaborative interaction data from 100 middle school students (47 pairs) in a public school in the southeastern United States.

Procedure: Over ten 50-minute sessions, students used "AI Made By You" (AMBY), a web-based environment, to collaboratively design scientific chatbots using a pair-programming approach (alternating "driver" and "navigator" roles).
Data Collection: Researchers collected audio/video recordings of 47 dyads, resulting in a corpus of 32,976 utterances. Data also included pre-survey data on prior programming experience, a post-assessment of AI knowledge (15 items), and researcher-evaluated chatbot artifacts using a 10-dimension rubric.
Coding Schemes: Two parallel coding schemes were applied to the video and transcript data:
1. Collaborative Dialogue: Adapted from Mercer's exploratory talk and Zakaria et al.'s taxonomy, capturing acts such as explanation, directive, disagreement, and sense-making.
2. Chatbot Development Behaviors: Captured specific actions within the AMBY interface, such as testing, adding training phrases, developing responses, and debugging.
Analysis Technique: The core analytical method was Ordered Network Analysis (ONA). ONA utilizes a moving window (size 4) to identify sequential connections among constructs (dialogue acts and development behaviors). It distinguishes between the strength of transitions (e.g., A $\to$ $\to$ B vs. B $\to$ $\to$ A) and self-transitions (repetition).
- Groups were stratified by chatbot quality (high vs. low performance) using a median split.
- Statistical comparisons (Mann–Whitney U tests) were conducted on network structures.
- Spearman's rank-order correlations, corrected via Benjamini–Hochberg and validated with Monte Carlo analysis, were used to link interaction patterns to AI knowledge outcomes.

Key Results
The analysis yielded three primary findings corresponding to the research questions:

Temporal Organization of Interaction (RQ1): Collaborative AI chatbot development is characterized by non-linear, iterative cycles rather than linear progression. Across all groups, activity was structured around recurring sequences linking explanation, directive coordination, and engagement with chatbot behavior.
Differentiation by Chatbot Quality (RQ2):
- High-Performing Groups: Exhibited more densely connected networks with stronger self-loops and bidirectional transitions between explanation and directive acts. They showed stronger sequences linking justified disagreement to explanation. In development behaviors, they demonstrated frequent transitions between testing and refining responses (specifically for main intents).
- Low-Performing Groups: Interaction patterns were more segmented, centering on procedural questioning, suggestions, and orientation toward system output. Their development behaviors showed extended engagement in setup tasks (e.g., identity setup, follow-up intent configuration) with fewer transitions to testing and refinement.
- Prior Experience: Differences in performance were not explained by prior programming experience, suggesting that interactional coordination is a more critical factor than individual background.
Association with AI Knowledge (RQ3):
- Positive Associations: AI knowledge gains were positively correlated with interaction sequences involving explanation, repetition (of prior contributions), and bidirectional connections between explanation and chatbot responses. In development, stronger testing self-loops and transitions between developing responses and testing were linked to higher knowledge scores.
- Negative Associations: Knowledge outcomes were negatively associated with sequences dominated by directive and suggestion acts, as well as extensive focus on configuring follow-up intents rather than core logic.

Significance and Contributions
The paper claims to extend the Artificial Intelligence in Education (AIED) and Computer-Supported Collaborative Learning (CSCL) literature in several specific ways:

Process-Oriented Account: It provides a granular, process-oriented account of collaborative AI chatbot development, moving beyond outcome-based evaluations to model how learning unfolds through temporally organized interaction.
Integration of Modalities: By using ONA, the study demonstrates how multiple streams of activity (dialogue and artifact development) are coordinated over time, offering a more holistic view of collaborative learning than single-stream analyses.
Contextual Specificity: The findings highlight that successful collaboration in AI education contexts requires specific interactional patterns—namely, the integration of articulated reasoning, repeated testing, and iterative refinement—to navigate the probabilistic nature of AI systems.
Design Implications: The results suggest that instructional supports should foster interactional cycles that integrate explanation, testing, and revision, rather than focusing solely on procedural task completion.

The authors maintain a modest stance, acknowledging that the observed associations are correlational and do not establish causality. They note limitations regarding generalizability to other age groups or tools and the exclusion of non-verbal modalities like gesture. However, the study establishes a foundational framework for analyzing the temporal dynamics of collaborative AI learning.

Technical Summary: Analyzing Middle School Students' Dialogue and Behaviors during Collaborative AI Chatbot Development Using Ordered Network Analysis

More like this