Socratic generative AI feedback improves learning transfer and confidence calibration in second language learning

Language learners have long relied on teachers and textbooks to correct their mistakes, but a new tool has entered the classroom: generative artificial intelligence. These systems can instantly spot errors in a sentence and offer a fix, or they can guide a student to find the fix themselves. The question facing educators and researchers is not whether these tools work, but how they work best. Does handing a learner the correct answer help them learn faster, or does the struggle of figuring it out on their own lead to deeper understanding? This dilemma touches on a fundamental idea in learning science: the difference between feeling like you know something and actually being able to use it later. When information comes easily, the brain often mistakes that ease for mastery, leading students to believe they have learned more than they have. This study investigates whether the way an AI gives feedback changes not just immediate scores, but the ability to use language skills days later and the accuracy of a learner's own confidence.

Researchers at Central China Normal University, Zhengzhou University of Industrial Technology, and the University of Pécs designed a controlled experiment to test this. They recruited seventy adult learners whose first language was Mandarin and asked them to practice the English article system—the rules for using words like "a," "an," and "the." This specific grammar point is notoriously difficult for speakers of languages that do not use articles, making it a perfect test case. The participants were split into two equal groups. Both groups used the same language model and studied the same material for the same amount of time. The only difference was how the AI responded when they made a mistake. One group received direct answers: the AI immediately supplied the correct sentence with a brief explanation. The other group received Socratic feedback: the AI asked questions to prompt the learner to think about the error and generate the correct form themselves, withholding the answer until the learner tried.

The study tracked what happened immediately after the practice session and again seven days later, when the learners took a new test without any help from the AI. The results showed a clear divergence between the two approaches. Learners who received the direct answers felt more confident right away, but their confidence was misplaced. On average, they overestimated their accuracy by more than twelve percentage points, believing they knew the material better than they actually did. In contrast, the group that had to work through questions to find the answers felt much more realistic about their knowledge, overestimating their performance by only about two percentage points. The direct-answer group found the process easier and less mentally taxing, while the Socratic group reported a significantly higher level of mental effort during the session.

When the researchers looked at how well the learners retained the knowledge a week later, the group that had struggled with the questions performed better. On tests involving new sentences they had never seen before, the Socratic group was significantly more accurate than the direct-answer group. They were also better at using the grammar rules in a free-writing task, producing more correct sentences than the group that had simply been given the answers. However, the study found that this advantage was specific to applying the rules to new situations. The two groups performed similarly when asked to remember the exact sentences they had practiced or when asked to judge grammar quickly under time pressure. This suggests that the benefit of the questioning style was not about memorizing specific examples, but about building a flexible understanding that could be transferred to new contexts.

The researchers also explored whether the difference in confidence explained the difference in long-term performance. It is possible that the Socratic group did better simply because they knew they were struggling and studied harder, while the direct-answer group felt too confident to put in the effort. The data showed that while the feedback style strongly influenced confidence, the link between that confidence and the later test scores was too weak to prove a direct cause-and-effect chain. Instead, the evidence suggests that the extra mental effort required to generate the answer itself was the key factor. The difficulty of the task, which made the learning process feel harder in the moment, appears to have strengthened the memory and the ability to apply the rules later.

This study does not claim that one method is perfect for every situation or that it solves all problems in language learning. The sample size was modest, and the experiment focused on a single grammar rule with one group of learners. Yet, the findings offer a clear picture of how feedback style shapes learning. Providing an immediate answer is efficient and less frustrating, but it may create an illusion of competence that fades quickly. Guiding a learner to find the answer requires more work and feels more difficult, but it leads to a more accurate sense of what they know and a stronger ability to use that knowledge when the tool is gone. The research suggests that for durable learning, the path of least resistance is not always the most effective one.