Analyzing Undergraduate Problem-Solving in Physics Through Interaction With an AI Chatbot

Imagine physics class as a giant, noisy gym where students are trying to learn how to juggle heavy chainsaws. For decades, teachers have known that the experts (the pros) and the beginners (the novices) don't just juggle differently; they think differently. The pros have a secret map in their heads that helps them plan their moves, check their balance, and adjust mid-air. But in a traditional classroom, the teacher can only see the final result: did the chainsaw hit the floor or not? The messy, invisible thinking process that happened before the drop remains a mystery. This is a big problem because if you can't see how a student is thinking, you can't help them fix their map.

Enter the new coach: Artificial Intelligence (AI). Think of AI not as a robot that just gives answers, but as a "Socratic" tutor—a fancy word for a coach who never tells you the answer but asks you the perfect questions to help you figure it out yourself. It's like a GPS that refuses to say "turn left," but instead asks, "Do you see the street sign for Main Street?" This approach is designed to build the student's own mental map. The big question for scientists is: Does this AI coach actually help students juggle better, and can we learn something about how they are thinking just by listening to the questions they ask the coach?

---

In this study, researchers at Purdue University decided to test this AI coach in a real-life physics gym. They set up a custom chatbot for 150 first-year science students taking a tough mechanics course. Instead of just handing out homework, the chatbot acted like a human teaching assistant. It presented a tricky scenario: a human cannonball being launched by a giant spring, and the student had to figure out if the performer would clear a 15-foot wall. The catch? The chatbot wouldn't just solve it for them. It would ask, "What's your plan?" or "What data do we have?" and wait for the student to reply.

The researchers wanted to know two things. First, how did the students feel about this AI coach? Did it make them feel smarter or more confident? Second, could the researchers learn about the students' problem-solving skills just by analyzing the questions the students asked the bot? They treated the chat logs like a treasure map, looking for clues about how the students' thinking evolved from "I'm lost" to "I've got this."

What the Students Thought
When the students finished their session, they filled out a survey. The results were pretty encouraging. On a scale of 1 to 5, the students gave the chatbot a solid 4.0 out of 5 for helping them understand the actual physics concepts and skills. They felt the bot helped them learn how to tackle problems on their own. The overall effectiveness rating was a bit lower, around 3.4 out of 5, which suggests that while the bot was great for learning, it wasn't a magic wand that made everything easy or fun for everyone. Some students still found the tasks challenging, and the bot didn't completely eliminate the struggle, but it did help them navigate it.

The Evolution of Questions
Here is where the story gets really interesting. The researchers looked at the transcripts of the conversations, treating every question the student asked as a snapshot of their brain at that moment. They categorized questions into two types: "Broad" (like "I don't know where to start" or "What do I do?") and "Specific" (like "Should I use the conservation of energy here?" or "Does the spring height matter?").

At the very beginning of the chat, most students were in the dark. Only about 10–15% of the first questions were specific; the rest were broad pleas for help. But as the conversation went on, something magical happened. The students started asking sharper, more targeted questions. By the fourth turn in the conversation, about 58% of the questions were specific. By the very end of the session, 100% of the students were asking specific, focused questions. It was as if the AI coach successfully guided them from wandering aimlessly in a fog to walking a straight, well-lit path.

The Link to Success
The researchers then asked: Does asking better questions mean you get better grades? They compared how specific a student's questions were against the grade they expected to get in the course. They found a "mild positive correlation" (a statistical way of saying they move together) with a number of 0.43.

What does this mean in plain English? Students who asked more specific, detailed questions tended to expect higher grades. For example, many students who thought they would get an "A" had over 80% of their questions classified as specific. On the flip side, students who kept asking broad, vague questions tended to expect lower grades (around a "C").

However, the researchers were careful not to call this a perfect rule. The data was a bit scattered. Some students who asked very specific questions still only expected a middle-of-the-road grade, and a few students who expected high grades asked surprisingly broad questions. This suggests that while asking good questions is a strong sign of a good problem-solver, it's not the only thing that matters. Other factors, like how much you already knew or how hard you worked, also play a huge role.

The Takeaway
This study suggests that AI chatbots can be powerful tools for teaching physics, not just by giving answers, but by guiding students to ask the right questions. The chatbot helped students shift from feeling lost to thinking like experts. While asking specific questions is a good sign of success, it's just one piece of the puzzle. The researchers admit this was a small test with just one type of problem, so they need to do more studies to see if this works for everyone and every topic. But for now, it looks like having an AI coach that asks "What's your plan?" instead of "Here's the answer" is a pretty good way to help students learn how to juggle those physics chainsaws.