The StudyChat Dataset: Analyzing Student Dialogues With ChatGPT in an Artificial Intelligence Course

This paper introduces StudyChat, a publicly available dataset of 16,851 annotated student interactions with an LLM-powered tutoring chatbot in an AI course, revealing that using the tool for conceptual understanding and coding assistance correlates with better academic performance, whereas using it to bypass learning objectives leads to lower exam scores.

The Gist

Imagine a classroom where every student has a super-smart, 24/7 personal tutor named "ChatGPT." This tutor can write code, explain complex math, and even help draft essays. But here's the big question: Is this tutor helping students learn, or is it just doing the homework for them?

This paper, titled "The StudyChat Dataset," is like a giant, transparent window into that classroom. Researchers at the University of Massachusetts Amherst set up a special experiment to watch exactly how students used this AI tutor during a real university-level Artificial Intelligence course.

Here is the story of what they found, explained simply:

🕵️‍♂️ The Experiment: A "Glass House" Classroom

Instead of banning AI (which is hard to enforce) or letting it run wild without tracking, the professors built their own version of ChatGPT right inside the course website.

• The Setup: They told students, "Use this tool as much as you want for your coding assignments. It won't hurt your grade, and we promise to keep your identity secret."
• The Catch: They recorded every single word the students typed and every answer the AI gave.
• The Result: They collected over 16,000 conversations from 203 students. It's like having a diary of 203 students' entire semester of thinking and struggling.

🏷️ The "Traffic Cop" System: Labeling the Chats

You can't just read 16,000 chats to find patterns; your brain would melt! So, the researchers created a "traffic cop" system called Dialogue Acts. They sorted every student message into categories, like:

• "The Learner": "Can you explain how this Python loop works?" (Asking for concepts).
• "The Doer": "Write this code for me." (Asking for the solution).
• "The Editor": "Fix this error message." (Checking work).
• "The Ghostwriter": "Write this report for me." (Trying to bypass the work).

They used a second AI to do the sorting, which they double-checked with human helpers to make sure it was accurate.

🔍 What Did They Discover?

The researchers looked at these labels and compared them to the students' actual grades. Here are the four big takeaways:

1. The "Conceptual" vs. "Copy-Paste" Divide 🧠 vs. 📄

• The Winners: Students who used the AI like a tutor (asking "Why does this work?" or "How do I fix this bug?") tended to get higher grades on both their assignments and the final exams. They were using the tool to build their own knowledge.
• The Losers: Students who used the AI like a ghostwriter (asking "Write this whole report" or "Give me the full solution") tended to get lower grades on the exams. It's like eating a meal someone else cooked; you get full, but you don't learn how to cook. When the exam came around (where no AI was allowed), they were hungry and unprepared.

2. The "Confusion" Signal 🚨

The researchers noticed something interesting: When students asked very specific questions about the current assignment (like "What does this specific error mean?"), it often correlated with lower scores.

• The Metaphor: Imagine a student staring at a broken car engine, asking the mechanic, "Why is this specific bolt loose?" over and over. It suggests the student is confused and stuck.
• The Twist: However, asking about general concepts (like "How do neural networks work?") was a sign of a curious, high-achieving student.

3. The "Super-Users" are the Most Consistent 📈

Some students barely used the AI, while others chatted with it hundreds of times.

• The Surprising Finding: The "Super-Users" (those who chatted the most) didn't necessarily get the highest average scores, but they were the most consistent. Their grades didn't swing wildly; they stayed in a good, safe range.
• The Analogy: Think of the AI as a safety net. The students who used it the most were like tightrope walkers who held onto the net the whole time. They didn't fall as hard as the others, even if they didn't jump higher.

4. The "Code-Only" vs. "Report-Writer" Clusters 🤖

The researchers grouped students by their "personality" with the AI:

• Group A (The Coders): These students asked the AI to help write code or explain logic. They scored high on exams.
• Group B (The Report Writers): These students asked the AI to write their English reports and summaries. They scored lower on exams.
• The Lesson: If you use AI to do the thinking part of the assignment, you learn. If you use it to do the writing part, you might be cheating yourself out of the learning experience.

🎓 The Big Picture

This paper isn't just about data; it's a warning and a guide for the future of education.

• The Warning: If we let students use AI to "do the work" (write reports, solve problems instantly), they might pass the assignment but fail the real test of knowledge.
• The Opportunity: If we encourage students to use AI as a Socratic tutor (someone who asks questions and explains concepts), it can actually boost their learning and stabilize their performance.

In short: The StudyChat dataset shows us that how you use the tool matters more than how much you use it. Using AI to learn is a superpower; using it to skip the learning process is a trap.

Technical Summary

Here is a detailed technical summary of the paper "The StudyChat Dataset: Analyzing Student Dialogues With ChatGPT in an Artificial Intelligence Course."

1. Problem Statement

The rapid proliferation of Large Language Models (LLMs) like ChatGPT in education presents a dichotomy: they offer opportunities for personalized, on-demand tutoring but pose significant risks regarding academic integrity and over-reliance. While educators are concerned about students using LLMs to circumvent learning objectives (e.g., generating code or reports without understanding), there is a lack of empirical, large-scale data on how students actually interact with these tools in real-world university settings. Existing studies are often small-scale, rely on self-reported surveys, or lack granular dialogue analysis. The authors identify a need to understand specific student interaction patterns (dialogue acts) and correlate them with learning outcomes to inform the development of better Intelligent Tutoring Systems (ITS) and assessment strategies.

2. Methodology

A. Dataset Collection (StudyChat)

• Context: The study was conducted over two semesters (Fall 2024, Spring 2025) in an upper-division Artificial Intelligence course at the University of Massachusetts Amherst.
• Participants: 203 consenting students (out of 295 enrolled) from 3rd/4th-year CS backgrounds.
• Tool Deployment: The researchers developed a web application mirroring the ChatGPT interface, powered by the gpt-4o-mini model. Students were encouraged to use this tool freely for all programming assignments without restrictions or grade penalties.
• Data Volume:
  • Conversations: 2,214 unique student-LLM conversations.
  • Utterances: 16,851 total student utterances.
  • Assignments: 7 unique graded Python programming assignments (involving search algorithms, deep learning, and NLP).
  • Submissions: 924 graded assignment submissions from 158 students.
• Privacy: A systematic process using regular expressions removed Personally Identifiable Information (PII), such as directory paths and GitHub usernames, from 6,413 cases.

B. Dialogue Act (DA) Annotation

To analyze the nature of interactions, the authors created a two-level Dialogue Act labeling schema:

• Structure: 8 Broad Categories and 31 Specific Labels.
• Categories: Include Writing (Write Code, Write English), Editing, Contextual Questions (Assignment Clarification, Code Explanation), Conceptual Questions (Python Library, Math, CS concepts), Verification, Context, Off Topic, and Misc.
• Annotation Process:
  1. Human Validation: Initial human annotation achieved high inter-rater agreement (Cohen's $\kappa$ = 0.910 for broad labels).
  2. LLM Scaling: To label the full dataset, the authors used an LLM-prompting approach (GPT-4.1) with detailed instructions and few-shot examples.
  3. Validation: Human-LLM agreement was moderate ($\kappa \approx 0.58$), comparable to human-human agreement on held-out sets, validating the scalability of the automated approach.

C. Analysis Techniques

• Regression Analysis: Linear regression models were used to predict assessment scores (assignments and exams) based on:
  • Baseline: Prior performance.
  • Features: Total utterance counts, broad DA counts, and specific DA counts.
• Clustering: K-means clustering (with $k=4$) was applied to normalized DA feature vectors to group students by behavioral profiles. Principal Component Analysis (PCA) was used for visualization.
• Usage Segmentation: Students were categorized into Low, Medium, and High interaction groups based on total utterance counts.

3. Key Contributions

1. The StudyChat Dataset: A publicly available, large-scale dataset of 16,851 real-world student-LLM interactions in a university CS course, including raw conversations, dialogue act annotations, and corresponding assignment grades.
2. Scalable Annotation Framework: A validated methodology for using LLMs to perform Dialogue Act classification on educational dialogues at scale, achieving agreement levels comparable to human annotators.
3. Behavioral Taxonomy: A refined schema distinguishing between learning-focused interactions (conceptual questions, code verification) and potential misuse (direct code generation, circumventing learning objectives).

4. Key Results and Findings

A. Dialogue Acts as Predictors

• Conceptual vs. Contextual: Students who used LLMs for Conceptual Questions (general knowledge, Python libraries) and Code Writing assistance tended to perform better on assignments and exams.
• Negative Correlation: Contextual Questions (questions specific to the immediate assignment details or error messages) were negatively correlated with assignment scores. This suggests that students asking for specific "how-to-fix-this" help may be confused or relying on the LLM to bypass the learning process.
• Mathematical Derivations: In a specific assignment requiring $n$-gram language model derivation, heavy reliance on LLMs for mathematical explanations correlated with lower scores, indicating LLMs (specifically gpt-4o-mini) struggle with procedural math derivations.

B. Usage Frequency and Outcomes

• High Engagement Stability: Students with High Usage (top 10%) showed higher minimum scores and lower variance in outcomes compared to low/medium users. While average scores were similar across groups, high engagement appeared to stabilize performance and prevent low outliers.
• Low Usage Variance: Low and medium users exhibited a wider spread of scores, including more low outliers.

C. Behavioral Clustering

Four distinct student clusters were identified:

1. Code Writers: Focused on writing code and English reports.
2. Coding Questioners: Focused on conceptual questions about Python and CS tools.
3. General Questioners: Asked a mix of conceptual and contextual questions.
4. Report Writers: Heavily relied on LLMs for synthesizing results and writing reports.

• Performance Gap: The Coding Questioners (Cluster 1) achieved the highest average exam scores ($89 \pm 7.3$). Conversely, **Report Writers** (Cluster 3) had the lowest average exam scores ($83.8 \pm 11.7$), suggesting that using LLMs to generate reports circumvents the learning objectives, leading to poorer performance on closed-book exams.

5. Significance and Future Work

• Pedagogical Insight: The study provides evidence that how students use LLMs matters more than how much they use them. Using LLMs as a "tutor" for concepts correlates with success, while using them as a "substitute" for writing or specific problem-solving correlates with failure.
• Intervention Strategies: Educators can use these insights to design prompts or tools that encourage conceptual questioning while discouraging direct answer-seeking for specific assignment tasks.
• System Design: Future ITS can be designed to detect "over-reliance" patterns (e.g., excessive contextual questioning) and flag them to instructors or adapt the tutoring strategy in real-time.
• Limitations: The study is limited to a single course and upper-division students. Additionally, the "Hawthorne Effect" (students knowing they are being recorded) may have altered natural behavior.

Conclusion: The StudyChat dataset and analysis demonstrate that LLMs are a double-edged sword in education. When used to augment conceptual understanding, they support learning; when used to bypass cognitive effort, they degrade performance. The dataset serves as a critical resource for developing AI-driven educational tools that foster productive learning behaviors.