🔬 physics

Students' Epistemological Beliefs and their Chatbot Preferences in AI-mediated Physics Learning

This study investigates the relationship between introductory physics students' epistemological beliefs and their preferences for AI chatbot behaviors in a wave simulation module, finding that while students favoring a "combination" of guided inquiry and direct answers initially showed more sophisticated beliefs, these differences were not statistically significant after Bonferroni correction.

Original authors: Amogh Sirnoorkar, Omkar Mamidpalliwar

Published 2026-08-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Amogh Sirnoorkar, Omkar Mamidpalliwar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to learn how to ride a bike. You have two choices for a helper: one who just hops on the back and steers for you, telling you exactly where to go, and another who runs alongside you, asking, "Which way do you think you should turn?" and only grabbing the handlebars if you're about to crash. This is the core of a fascinating corner of science called physics education research. Scientists in this field don't just care if you get the right answer on a test; they care about how your brain thinks about knowledge itself. They study your "epistemological beliefs," which is a fancy way of asking: Do you think knowledge is a fixed list of facts handed down by experts, or do you think it's something you build yourself through effort and mistakes?

Now, enter the new kid on the block: Generative AI. These are the super-smart chatbots that can solve problems instantly. As students start using these tools for homework, a big question pops up: Do students who want the AI to just "give them the answer" think about learning differently than students who want the AI to "guide them through the problem"? If we can figure this out, we might be able to design AI tutors that don't just do the work for students, but actually help them become better thinkers.

This paper, written by researchers at Purdue University, dives right into that question. They set up a digital playground for students taking a calculus-based physics class. The students were asked to explore a topic called "waves" using a custom-made online module. This module had cool simulations and a built-in chatbot. Before they even started, the students were asked to pick their favorite style of chatbot interaction from three options:

Direct Answers: "Just tell me the answer."
Guided Inquiry: "Ask me questions and give me hints so I figure it out myself."
Combination: "Start by asking me questions, but if I get stuck, just give me the answer."

The researchers also gave the students a standard survey called EBAPS to measure their "epistemological beliefs"—basically, how sophisticated their ideas about learning and knowledge were. They wanted to see if the chatbot a student wanted to use matched up with how they actually thought about learning.

Here is what they found, and it's a bit of a mixed bag. First, the most popular choice was the Combination style. About 52% of the students picked this "guide me first, but help me if I'm stuck" option. A smaller group, 39.6%, wanted Direct Answers, and only 7.6% wanted pure Guided Inquiry.

When the researchers compared the survey scores of these three groups, they saw an interesting pattern. The students who chose the Combination chatbot tended to have more "sophisticated" beliefs about learning than those who just wanted Direct Answers. Specifically, these "Combination" students seemed to believe more strongly that:

Knowledge isn't just a bunch of unchangeable facts; it can evolve and change as we learn more.
Being good at science isn't about being born with a special "smart gene"; it's about putting in the effort and using good strategies.

However, the researchers are very careful not to shout "Eureka!" just yet. While the numbers showed a difference, the statistical "confidence" was a bit shaky. When they applied a stricter test to make sure the results weren't just a fluke (a method called the Bonferroni adjustment), the difference between the groups disappeared.

So, what does this mean? The paper suggests that there might be a link between wanting a helpful, balanced AI tutor and having a growth mindset about learning. It hints that students who believe effort matters might naturally prefer a chatbot that challenges them a bit before giving up the answer. But because the statistical evidence wasn't rock-solid, the authors say this is more of a "suggestive hint" than a proven fact. They didn't find any strong link between the students' beliefs and those who wanted pure guided questions versus pure answers.

The takeaway for the future? If we want to build AI tutors for physics that actually help students grow, we shouldn't just let them pick the "shortcut" button. Instead, we should design bots that start by asking questions and nudging students to think, only stepping in with the answer when the student is truly stuck. This approach seems to align best with the students who have the healthiest attitudes toward learning, even if we need more research to be 100% sure.

Technical Summary: Students' Epistemological Beliefs and their Chatbot Preferences in AI-mediated Physics Learning

Problem Statement
The rapid integration of Generative AI into higher education presents a dual-edged sword: while offering immediate access to information, it risks reducing student engagement in essential cognitive processes such as modeling, argumentation, and sensemaking. Despite a growing body of Physics Education Research (PER) examining student interactions with AI, significant gaps remain. Specifically, there is a scarcity of research regarding students' preferences for specific chatbot behaviors (e.g., guided-inquiry versus direct answers) and a lack of exploration into the intersection of these preferences with students' epistemological beliefs (their views on the nature of knowledge and learning) within the context of physics. This study addresses these gaps by investigating the association between introductory physics students' chatbot behavior preferences and their epistemological beliefs.

Methodology
The study was conducted in a large-enrollment, calculus-based introductory physics course for future engineers at a midwestern R1 university in the United States. The context was an extra-credit online module focused on waves, utilizing custom-built simulations integrated with a chatbot.

Participants: Data was collected from 1,048 overlapping responses from students who participated in both the chatbot activity and the Epistemological Beliefs Assessment for Physical Sciences (EBAPS) survey during Spring 2026.
Instrumentation:
- Chatbot Preference: Students were asked to select their preferred chatbot behavior from three options: (1) Direct Answers (immediate solutions), (2) Guided Q&A (conceptual hints only), or (3) Combination (initial guided inquiry with direct answers if the student remains stuck).
- Epistemological Beliefs: The EBAPS survey, a 30-item questionnaire, was administered to assess beliefs across five dimensions: Structure of scientific knowledge, Nature of knowing and learning, Real-life applicability, Evolving knowledge, and Source of ability to learn. Scores were normalized to percentages (0–100 scale).
Data Analysis: The researchers employed the Kruskal-Wallis test to compare EBAPS scores across the three preference groups, followed by Dunn's post-hoc tests with Bonferroni correction to identify specific pairwise differences. Effect sizes were also calculated.

Key Results
The analysis revealed that 52% of students preferred the "Combination" approach, 39.6% preferred "Direct Answers," and only 7.6% preferred "Guided Q&A."

Statistical Associations: At the nominal significance level ( $\alpha = 0.05$ ), the Kruskal-Wallis test indicated significant differences in overall EBAPS scores and specifically in Axis 4 (Evolving knowledge) and Axis 5 (Source of ability to learn) among the preference groups.
Specific Findings: Post-hoc analysis showed that students preferring the "Combination" chatbot exhibited significantly higher scores than those preferring "Direct Answers" on:
- Axis 4: Balancing absolutism (knowledge as fixed) and relativism (distinguishing evidence from opinion).
- Axis 5: Viewing success in science as determined by effort and strategy rather than innate ability.
Robustness Check: When applying a Bonferroni-adjusted significance level ( $\alpha = 0.0083$ ) to account for multiple comparisons, the observed differences for the total score and Axes 4 and 5 were no longer statistically significant. The effect sizes were also noted to be small, indicating that chatbot preference accounted for only a modest proportion of the variability in epistemological scores.

Key Contributions
This study contributes to the literature by shifting the focus from whether students use AI to how they prefer AI to support their learning. Unlike previous studies that examined general AI adoption or beliefs about AI itself, this research specifically links preferences for instructional chatbot behaviors to domain-specific epistemological beliefs in physics. It provides empirical data suggesting a potential, albeit tentative, link between sophisticated epistemological beliefs and a preference for hybrid learning tools that balance inquiry with support.

Significance and Implications
The authors characterize their findings as "suggestive rather than conclusive" due to the lack of significance under conservative statistical corrections and the modest effect sizes. However, they propose several implications for the design and instructional use of chatbots in physics education:

Design Strategy: Chatbots should be designed to initially engage students in guided inquiry (reasoning through problems) but provide direct assistance when students are stuck. This approach aims to balance "productive struggle" with timely support, preventing frustration while avoiding the inhibition of sensemaking.
Instructional Reinforcement: The findings regarding Axis 5 suggest that chatbots and instructors should reinforce the idea that learning ability is developed through effort and strategy, not innate talent. Chatbots can be programmed to prompt students to revisit concepts, break problems into steps, or reflect on unsuccessful strategies, framing assistance as a tool for developing competence rather than a substitute for thinking.
Future Research: The study calls for further investigation into whether sustained use of Socratic-style chatbots can positively shift students' epistemological beliefs over time and how AI usage impacts engagement in epistemic games during problem-solving.

The paper concludes that while the observed associations warrant further investigation, they offer a preliminary basis for designing AI-mediated learning environments that align with sophisticated epistemological frameworks.

More like this