📄 social_science

How Undergraduates Interact with a Generative AI Peer: Eight Participation Roles and the Conditions for Effective Collaboration in University Mathematics

This study identifies eight distinct undergraduate participation roles in human–GenAI collaborative problem solving, demonstrating that students who actively sustain shared cognitive states and externalize reasoning achieve significantly better outcomes than those who delegate or disengage, thereby highlighting the critical need for deliberate task design and feedback loops to foster effective AI-peer collaboration.

Original authors: Matias Rojas, Nicolás Guerra, Jonathan Frez, Brayan Díaz, Xiaoming Zhai

Published 2026-09-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Matias Rojas, Nicolás Guerra, Jonathan Frez, Brayan Díaz, Xiaoming Zhai

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern classroom, a new kind of partner has arrived, one that does not sit at a desk but lives within the software students use to learn. This partner is a generative artificial intelligence, a system capable of holding a conversation, answering questions, and helping solve problems. For years, educators have wondered how students actually work with these tools. Do they treat the computer as a smart tutor to be guided, or as a search engine to be queried? Do they think deeply alongside it, or do they simply ask for the answer and move on? The question matters because the way a student interacts with a machine might determine whether they truly learn the material or merely complete the assignment. Researchers are now trying to map the landscape of this human-machine partnership, looking for the specific behaviors that lead to real understanding and those that lead to shallow results.

A team of researchers set out to explore this dynamic in the high-stakes environment of university mathematics. They created a digital space where thirty undergraduate students worked with an artificial intelligence peer to solve complex calculus problems. The setup was designed so that neither the student nor the computer could solve the problem alone. The student held some of the necessary information, and the computer held the rest. To find the solution, they had to talk, share what they knew, and coordinate their steps. This forced a genuine collaboration rather than a simple transaction where the student asks a question and the machine provides an answer. The researchers recorded every message and action, using a sophisticated system to analyze the conversation and determine how well the pair was working together.

The study revealed that students did not all interact with the AI in the same way. Instead of a single pattern of behavior, the researchers identified eight distinct roles that students played during these sessions. At one end of the spectrum were students who acted as "Co-Regulators." These individuals treated the AI as a true thinking partner. They explained their own reasoning, checked their understanding against the computer's suggestions, and worked together to build a shared picture of the problem. They externalized their thoughts, making their mental process visible to the machine so it could respond meaningfully. At the other end were students who acted as "Early Disengagers" or "Low-Effort Guessers." These students barely engaged with the collaborative process. They might have asked for the answer, accepted it without question, or given up quickly when the task became difficult. Between these extremes lay other roles, such as the "Delegating Director," who stayed active but handed over the hard work to the computer, or the "Misaligned Follower," who tried to follow the AI but failed to build a correct shared understanding of the problem.

The results showed a clear link between how a student interacted with the AI and whether they succeeded. Students who maintained a high level of interaction, sharing information and reasoning through the steps together, were far more likely to solve both problems correctly. In fact, the odds of a student in this high-interaction group solving the tasks were about eleven times higher than for those in the low-interaction group. However, the researchers found that simply talking more was not the whole story. Some students who solved the problems did so with very little visible collaboration, often by bypassing the AI's help or working around it. Conversely, some students who talked a great deal with the AI were merely performing arithmetic or guessing numbers without truly understanding the underlying logic. The quality of the conversation mattered more than the quantity. The most productive interactions were those where the student kept their reasoning visible, maintained a shared understanding of the problem state, and distributed the mental work fairly between themselves and the machine.

To make sense of these complex conversations, the researchers developed a new way to measure collaboration automatically. They used a system of artificial intelligence agents to read the chat logs and score the interactions based on established frameworks for teamwork. This system could identify whether a student was planning, monitoring, or sharing information, providing a detailed map of the collaboration process. The automated scoring showed promise, agreeing with human experts on the general type of behavior about 60% of the time, and even better when looking at broader categories of interaction. This suggests that it is possible to analyze human-AI teamwork at a scale that was previously impossible, moving beyond simple test scores to understand the actual process of learning.

The study concludes that the relationship between a student and an AI peer is not a simple matter of using a tool. It is a complex social interaction where the student's approach determines the outcome. If a student treats the AI as a partner to think with, externalizing their own reasoning and negotiating steps, they are likely to learn deeply. If they treat it as a shortcut to an answer, they may finish the task but miss the learning. The researchers suggest that future educational tools should be designed to recognize these different roles. Instead of offering the same help to everyone, an AI system could detect when a student is delegating too much work or disengaging, and then adjust its prompts to encourage deeper thinking. By understanding the specific ways students interact with these new partners, educators can design better learning experiences that turn artificial intelligence into a genuine catalyst for human understanding.

Technical Summary: How Undergraduates Interact with a Generative AI Peer

Problem Statement
While generative AI (GenAI) is rapidly entering higher education, there is a critical lack of understanding regarding how individual students interact with AI as a peer during open-ended collaborative problem solving (CPS). Existing frameworks often treat human-AI interaction as monolithic or rely on scripted agents that suppress authentic negotiation and emergent coordination. Furthermore, measuring CPS in open-ended dialogue has historically been prohibitively expensive due to the need for intensive human coding. This study addresses the gap in understanding the specific interaction patterns, participation roles, and conditions that distinguish educationally productive human-GenAI collaboration from unproductive or transactional exchanges in university mathematics.

Methodology
The study employed a mixed-methods, within-subjects pilot design involving 30 undergraduate students enrolled in Calculus I at a private Chilean university.

System Design: A web-based platform paired each student with a Large Language Model (LLM) peer. The system utilized a "jigsaw" information-splitting structure, where the student and the LLM held complementary, non-overlapping information required to solve optimization problems. This design enforced structural interdependence, making collaboration necessary rather than optional.
Task Structure: Students completed two calculus-based optimization tasks. Task 1 involved multi-zone optimization with partial constraints, while Task 2 required inferring an unknown function. The LLM peer was prompt-engineered to act as a collaborator (avoiding direct solutions) and included a "Provocateur" agent to elicit information sharing and monitoring without providing mathematical content.
Data Collection & Automated Scoring: The system captured fine-grained dialogue and interaction traces (clicks, time-on-task). A novel three-agent automated scoring pipeline, grounded in the PISA 2015 CPS framework, was used to code student turns. This pipeline employed a mixture-of-experts approach: a Social expert assigned collaborative competencies, a Cognitive expert assigned cognitive processes, and a Judge agent synthesized these into a final code (A1–D3) and quality rating.
Validation: The automated scorer was validated against a gold-standard subset of 76 messages coded by two human raters. Agreement was measured at the 12-code level, as well as broader cognitive-process and collaborative-competence levels.
Analysis: Quantitative analysis included k-means clustering to identify interaction profiles based on trace features (actions, messages, length, time). Qualitative analysis involved a thematic case study of eight purposively selected conversations to interpret the quantitative profiles.

Key Contributions

Participation Typology: The study identifies eight distinct participation roles that students enact when collaborating with an AI peer: The Co-Regulator, The Recalibrating Collaborator, The Low-Effort Guesser, The Delegating Director, The Misaligned Follower, The Low-Engagement Answer-Seeker, The Efficient Knowledge Seeker, and The Early Disengager.
Automated Scoring Pipeline: The paper presents and validates a multi-agent LLM pipeline for scoring open-ended CPS dialogue. It demonstrates that while fine-grained 12-code agreement with humans is moderate ( $\kappa \approx .60$ ), agreement is substantially stronger at the broader collaborative-competence and cognitive-process levels ( $\kappa \approx .78$ and $.88$, respectively).
Interaction Profiles: The research distinguishes between simple interaction volume and the quality of collaboration, showing that high interaction does not automatically equate to high-quality CPS (e.g., a student may be active but rely on arithmetic trial-and-error rather than reasoning).

Results

Quantitative Findings: Clustering revealed two primary interaction profiles: a "higher-interaction" cluster (71% solved both tasks) and a "lower-interaction" cluster (19% solved both tasks). Students in the higher-interaction profile had odds of solving both tasks approximately 10.83 times higher than those in the lower-interaction profile.
CPS Metrics: While a global CPS index correlated with achievement, nested logistic models indicated that specific subdimensions—particularly the share of "take appropriate action" and "monitor and reflect" moves—were more predictive of success than the global score alone.
Qualitative Findings: The case analysis revealed that successful task completion did not guarantee deep collaboration (e.g., some students solved tasks with minimal visible CPS by bypassing the AI's scaffolding). Conversely, high engagement did not guarantee high-quality collaboration (e.g., some students delegated procedural work entirely). The most productive interactions involved students who externalized reasoning, maintained a shared cognitive state, and distributed epistemic responsibility.
Perceptions: Students perceived the AI feedback as useful (86.7%) and the peer as easy to use (80.0%), with no significant differences in perception between high and low achievers.

Significance and Claims
The paper claims that human-GenAI collaboration is not a uniform phenomenon but a complex spectrum of behaviors that significantly impacts learning outcomes. The study argues that effective AI-peer collaboration requires the deliberate design of task structures (like information splitting) and feedback loops that encourage specific regulatory behaviors.

The authors posit that their findings provide a theoretically grounded typology for interpreting student-AI logs, moving beyond binary metrics of "usage" or "success." They suggest that for AI-peer tools to be educationally productive, they must be designed to elicit and support specific participation roles (such as co-regulation) rather than allowing students to default to transactional or delegating roles. The study concludes that process-based measurement, enabled by automated scoring, is essential for distinguishing between students who solve problems with an AI peer versus those who solve around it. The authors remain modest, noting that the current evidence is from a pilot study and that the automated scorer is best suited for detecting broad collaborative moves rather than making definitive psychometric claims on every fine-grained code boundary.

More like this