💻 computer science

Beyond the AI Tutor: Social Learning with LLM Agents

This paper presents two controlled experiments demonstrating that multi-agent LLM configurations, which simulate social learning through diverse peer and tutor interactions, significantly enhance learning outcomes and idea diversity compared to traditional one-on-one AI tutoring.

Original authors: Harsh Kumar (Jace), Zi Kang (Jace), Mu, Jonathan Vincentius, Ashton Anderson

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Harsh Kumar (Jace), Zi Kang (Jace), Mu, Jonathan Vincentius, Ashton Anderson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to learn something new, like solving a tricky math problem or writing a creative story. For a long time, the "gold standard" for AI education has been the One-on-One Tutor: a single, super-smart robot teacher who sits with you, corrects your mistakes, and guides you to the right answer. It's like having a personal trainer who never gets tired and always knows the perfect form.

But this paper asks a big question: What if we stopped treating AI like a single tutor and started treating it like a whole classroom?

The researchers from the University of Toronto wanted to see if having multiple AI agents interacting with you—some acting as teachers, others acting as "peers" who make mistakes—could help you learn better than just having one perfect robot teacher.

Here is the breakdown of their two big experiments, explained with some everyday analogies.

Experiment 1: The Math Class (Convergent Learning)

The Goal: Solve SAT-level math problems where there is only one correct answer.

The Setup:
They put students into four different groups to see who learned the most:

The Solo Group: No help at all.
The "Peer" Group: Two AI students (Alice and Charlie) helped. But here's the twist: they were designed to make different kinds of mistakes.
- Alice knew the math concepts perfectly but kept messing up the simple arithmetic (like adding 2+2=5).
- Charlie was great at the math but misunderstood the core concepts (using the wrong formula).
The "Tutor" Group: A perfect AI teacher who gave hints and corrections.
The "Classroom" Group: The perfect teacher plus the two mistake-prone AI peers.

The Analogy:
Imagine you are trying to fix a leaky pipe.

The Tutor is the master plumber who shows you exactly how to do it.
The Peers are two apprentices. One is strong but drops his wrench (arithmetic error); the other is careful but uses the wrong wrench entirely (conceptual error).
The Classroom has the master plumber watching the apprentices struggle and fix their own mistakes while you watch.

The Result:
The group with both the Teacher and the Peers got the highest scores on the final test.

Why? Watching the "peers" struggle and make mistakes forced the students to think harder. They had to spot why Alice's math was wrong or why Charlie's logic was flawed. This "productive struggle" made the concepts stick better than just being told the right answer by a perfect teacher.
Bonus: Even the group with only the mistake-prone peers did better than the group with no help at all. Seeing others struggle made the students feel more confident in their own abilities.

Experiment 2: The Writing Workshop (Divergent Learning)

The Goal: Write essays (creative and argumentative) where there is no single right answer, but many possible good ideas.

The Setup:
They tested three groups:

The Solo Writer: No AI help.
The Single AI: One writing assistant (either ChatGPT or Claude) helping them.
The Duo: Two different AI assistants (ChatGPT and Claude) helping at the same time, each with a different "personality."
- One focused on structure and logic.
- The other focused on creativity and emotion.

The Analogy:
Imagine you are painting a picture.

The Single AI is like a single art critic who gives you great advice. Your painting gets better, but everyone who listens to that same critic ends up painting in the exact same style. The ideas start to look the same (homogenized).
The Duo is like having two critics with different tastes. One says, "Add more blue!" while the other says, "Make the lines bolder!" You get the best of both worlds: a high-quality painting that still feels unique to you.

The Result:

Quality: Both the Single AI and the Duo group wrote better essays than the Solo group.
Diversity: This is the big win. The Single AI group started to sound very similar to each other; their ideas became "cookie-cutter." The Duo group, however, kept their ideas diverse and unique, just like the group with no AI help.
The Catch: Having two AIs talking at once was a bit overwhelming. Some students felt distracted, like they were trying to listen to two people talking over each other in a noisy room.

The Big Takeaways

1. Mistakes are actually good (if you watch them).
In math, seeing an AI peer make a mistake and then fix it was more valuable than just being told the answer. It's like learning to drive by watching a friend fumble the parking spot and then correct it, rather than just being told "turn left here."

2. One AI makes us all think alike; two AIs keep us unique.
If you use one AI to write your essay, you might get a great grade, but you might lose your unique voice. If you use two different AIs with different strengths, you get the quality boost without losing your originality.

3. It's not just about the answer; it's about how you feel.
Students felt more confident and less intimidated when they saw "peers" (even AI ones) struggling alongside them. It made the hard stuff feel more manageable.

The Bottom Line

We are moving away from the idea of AI as a single, perfect tutor. The future of AI education might look more like a dynamic classroom where you have a wise teacher, a few friends who make funny mistakes, and maybe two different creative assistants arguing over your draft.

It's not just about getting the right answer faster; it's about building a learning environment that feels more human, keeps your ideas fresh, and helps you think for yourself.

1. Problem Statement

Current AI-assisted education is dominated by a one-on-one tutoring paradigm, where a single Large Language Model (LLM) acts as a personalized tutor for a single learner. While this approach has shown learning gains, it ignores decades of learning science research indicating that multi-party interaction (peer modeling, co-construction, and exposure to diverse perspectives) yields unique benefits that dyadic tutoring cannot replicate.

The paper addresses a critical gap: How do multi-agent LLM configurations affect learning outcomes compared to single-agent tutoring? Specifically, it investigates whether deploying multiple agents (e.g., a tutor plus peers, or specialized collaborators) can:

Enhance learning in convergent tasks (problems with a single correct answer) through error observation and schema conflict.
Improve divergent tasks (open-ended composition) by mitigating the "homogenization" of ideas often caused by reliance on a single model.

2. Methodology

The authors conducted two controlled experiments spanning distinct learning contexts (convergent vs. divergent) using a between-subjects design.

Experiment 1: Convergent Math Problem Solving

Task: Participants solved SAT-level math problems.
Design: A $2 \times 2$ $2 \times 2$ factorial design varying the presence of an LLM Tutor (Bob) and LLM Peers (Alice and Charlie).
- Control: No AI assistance.
- Peers Only: Two peers interacted with the participant.
- Tutor Only: A Socratic tutor provided feedback.
- Tutor + Peers: All three agents interacted.
Agent Configuration:
- Bob (Tutor): Provided hints and feedback without direct solutions.
- Alice (Peer): Strong conceptual understanding but prone to arithmetic errors.
- Charlie (Peer): Accurate arithmetic but prone to conceptual errors.
Procedure: Participants engaged in a 5-minute lesson phase (solving a problem with agent interaction), followed by a distractor task, and then an unassisted test phase using isomorphic variants of the problems.
Participants: $N=315$ (after filtering for attention/authenticity from an initial pool of 614 Prolific workers).

Experiment 2: Divergent Composition (Writing)

Task: Writing argumentative and creative essays based on New York Times prompts.
Design: Three conditions:
- Control: No AI assistance.
- Single: Assistance from one LLM (randomly assigned ChatGPT or Claude).
- Duo: Simultaneous assistance from both ChatGPT and Claude with role specialization.
  - Roles: One agent focused on structure/craft; the other on ideas/argumentation (roles randomized).
Procedure: Participants wrote two essays (one argumentative, one creative) with a 5-minute time limit per essay, interacting via free-form text chat.
Metrics:
- Quality: Rated by an LLM judge (validated against human raters) on a rubric covering originality, coherence, evidence, etc.
- Idea Diversity: Measured via pairwise cosine similarity of extracted key points across essays within a condition (lower similarity = higher diversity).
Participants: $N=247$ (from an initial pool of 419).

3. Key Contributions

Empirical Evidence for Multi-Agent Learning: One of the first controlled studies demonstrating that multi-agent LLM configurations can produce learning benefits distinct from single-agent tutoring.
Task-Contingent Benefits: The paper establishes that the utility of multi-agent systems depends on the task type:
- Convergent tasks benefit from diagnostic diversity (observing different types of errors).
- Divergent tasks benefit from architectural diversity (using distinct models to prevent idea homogenization).
Homogenization Mitigation: Demonstrates that the "idea-level homogenization" associated with AI writing assistance is not an inherent cost of AI but a function of deployment; using two distinct, role-specialized models can preserve the diversity of unassisted writing.
Self-Efficacy vs. Performance Trade-offs: Reveals that conditions maximizing objective performance (e.g., Tutor-only in math) do not always maximize learner self-efficacy or confidence, suggesting a need for adaptive systems.

4. Key Results

Experiment 1: Math Results

Test Accuracy: The Tutor + Peers condition achieved the highest unassisted test accuracy (~~65%), significantly outperforming Control (~~42%) and Tutor Only (~59%).
Independent Value of Peers: The Peers Only condition (~48%) outperformed Control, suggesting that observing imperfect peers has independent learning value even without an expert.
Error Observability: Participants rated the peer with arithmetic errors (Alice) more harshly than the peer with conceptual errors (Charlie), despite both being error-prone. This suggests learners calibrate trust based on the observability of errors (calculation errors are immediately verifiable) rather than severity.
Confusion: Agent conditions reported higher confusion, aligning with "desirable difficulties" theory, where productive confusion correlates with deeper engagement.

Experiment 2: Writing Results

Essay Quality: Both Single and Duo conditions significantly improved essay quality compared to Control. There was no significant difference in quality between Single and Duo.
Idea Diversity:
- Single LLM: Significantly increased idea homogenization (higher cosine similarity) compared to Control.
- Duo LLM: Preserved idea diversity at baseline levels (statistically indistinguishable from Control), effectively counteracting the homogenization effect of the Single condition.
Perceptions:
- Confusion/Overwhelm: The Duo condition reported the highest levels of feeling overwhelmed or distracted, indicating a cognitive cost to managing multiple agents.
- Self-Efficacy: Participants in the Duo condition reported higher confidence in independent writing than those in the Single condition, despite similar essay quality.

5. Significance and Implications

Pedagogical Shift: The findings argue for a shift from the "Bloom's Two-Sigma" goal of a single perfect tutor toward richer, multi-agent ecosystems that leverage peer modeling and role specialization.
Design Principles:
- Error Engineering: In math, exposing learners to specific, observable error types (arithmetic vs. conceptual) is a powerful design lever.
- Architectural Diversity: In writing, using distinct models with complementary roles prevents the "echo chamber" effect of single-model assistance.
Cognitive Costs: Multi-agent interactions introduce cognitive load (confusion, distraction). Designers must balance the benefits of diversity against the risk of overwhelming the learner.
Future of EdTech: As learners increasingly interact with multiple AI agents (e.g., using ChatGPT, Claude, and Gemini in one session), understanding these dynamics is crucial. The paper suggests that adaptive systems are needed to select the optimal agent configuration based on the learner's immediate needs (e.g., motivation vs. knowledge acquisition).

In conclusion, the paper demonstrates that moving beyond the single-tutor paradigm is not just a technical possibility but a pedagogically meaningful strategy that can unlock collaborative and observational benefits long documented in human social learning, provided the agent roles and interaction structures are carefully engineered.