Learning by Chatting? Investigating the Impact of Generative AI on Information Seeking and Learning
A field experiment comparing 8-day informal learning via ChatGPT versus Google Search reveals that while GenAI offers convenience, it ultimately leads to worse learning outcomes—particularly in critical thinking—by diminishing user agency, increasing meta-cognitive load, and narrowing information exploration through solution-oriented biases.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you want to learn how to cook a healthy week of meals. You have two ways to do this:
- The Old Way (Google): You act like a detective. You search for "healthy meals," find a dozen different websites, read a few articles, compare the recipes, check the nutrition facts on a label, and then decide what to cook. You are the chef, and the internet is just your pantry.
- The New Way (ChatGPT): You act like a customer at a fast-food drive-thru. You tell the AI, "I need a meal plan," and it instantly hands you a finished plate. You don't see the ingredients, you don't know how the sauce was made, and you didn't have to choose the spices.
This paper is a study that put real people in a "cooking class" for eight days to see which method helped them learn better. The researchers found that while the New Way (ChatGPT) felt faster and easier, it actually made people worse learners than the Old Way (Google).
Here is the breakdown of what happened, using simple metaphors:
1. The "Passenger vs. Driver" Problem
When you use Google, you are driving the car. You choose the route, you look at the map, and you decide when to turn. You are in control.
When you used ChatGPT in this study, you became a passenger. The AI decided where to go, what to show you, and how fast to drive.
- The Result: Because the AI was doing all the "driving" (selecting the information), the participants felt they had lost control. They felt like they were just along for the ride. This made them feel frustrated and mentally tired because they were constantly trying to steer the car without having a steering wheel.
2. The "Magic 8-Ball" vs. The Library
The study found two big problems with how ChatGPT acted as a teacher:
- Problem A: The "Solution-First" Bias
ChatGPT is like a helpful friend who always jumps straight to the answer. If you ask, "How do I balance my diet?", it immediately hands you a meal plan (a finished artifact).- The Catch: It rarely explains the principles (the "why" and "how"). It's like being given a solved math problem without seeing the steps. The participants learned what to eat, but they didn't learn why it was healthy. They couldn't apply the knowledge to new situations later.
- Problem B: The "Echo Chamber" Trap
Because ChatGPT remembers your past chats and tries to be a "personal assistant," it tends to stick to what you already asked.- The Catch: If you ask about "low-carb diets," it keeps talking about low-carb. It doesn't wander off to show you the broader world of nutrition. Google, on the other hand, is like a library where you can stumble upon a book you didn't know you needed. ChatGPT kept the participants in a small, narrow room, while Google let them explore the whole building.
3. The "Mental Gym" That Wasn't Worked
Learning is like going to the gym. To get strong (knowledge), you have to lift the weights (search, compare, verify, and think).
- Google Users: They lifted the weights. They had to check if a source was trustworthy, compare two different articles, and figure out the facts themselves. Their "learning muscles" got stronger.
- ChatGPT Users: They asked the machine to lift the weights for them. The machine did the heavy lifting, so the users' learning muscles stayed weak. When they tried to answer a test question later, they struggled because they hadn't practiced the mental work.
4. The "Trust but Verify" Failure
When using Google, people naturally checked their work. If a website looked sketchy, they clicked another link.
When using ChatGPT, people felt they couldn't trust the output, but they also didn't know how to check it. The AI gave them a single, smooth answer, making it hard to see if there were other sides to the story. The study found that ChatGPT users rarely double-checked the facts, and when they did try to, they felt confused by the AI's vague answers.
The Bottom Line
The study concludes that while Generative AI is amazing for getting quick answers (like asking a friend for a recipe), it is currently bad for deep learning.
By handing over the job of "searching and sorting" to the AI, people lose the control and the mental exercise needed to truly understand a topic. They end up with a "surface-level" understanding—like knowing the name of a dish but not knowing how to cook it if the ingredients change.
In short: If you want to get an answer quickly, ChatGPT is great. But if you want to learn and become an expert, you still need to do the work yourself, just like you used to with Google.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.