📄 social science

Search in Transition: A Mixed-Methods Study of LLMs and Search Engines in English Test Preparation

This mixed-methods study reveals that university students preparing for English language tests strategically combine LLMs for efficiency and search engines for verification, with those employing a balanced hybrid approach achieving the highest accuracy despite longer task completion times, ultimately supporting a new interface model for trustworthy AI-mediated learning.

Original authors: Tarek Rahman, Md Shaharia Hossen, Mark Protik Mondol, Jannatun Noor Mukta

Published 2026-10-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tarek Rahman, Md Shaharia Hossen, Mark Protik Mondol, Jannatun Noor Mukta

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Every year, millions of university students around the world prepare for high-stakes English language exams like the IELTS or TOEFL. These tests are often the final gatekeepers to studying abroad or securing a career, requiring not just a grasp of vocabulary and grammar, but the ability to synthesize complex information quickly. For decades, the standard approach to this preparation has been the traditional search engine. A student types a question, sifts through a list of links, reads multiple articles, and pieces together the answer. It is a method built on authority and verification, but it can be slow and mentally exhausting. In recent years, a new tool has entered the scene: the large language model. These are conversational artificial intelligence systems that can answer questions, write essays, and explain rules in a single, fluid response. They feel like having a tutor available at any moment. But as these two technologies collide in the minds of students, a critical question arises: do they compete, or do they work better together?

Researchers at United International University in Bangladesh set out to answer this by observing how students actually behave when faced with real test problems. They did not simply ask students what they preferred; they watched them work. The team gathered a large group of 140 university students to fill out surveys about their habits and feelings toward these tools. To get a deeper look, they brought in 20 of those students for a more intimate session. In this second phase, the students were asked to complete seven specific tasks that mimic the challenges of an English proficiency exam, such as summarizing a difficult text, correcting grammar errors, or writing a short opinion piece. The researchers then watched closely to see which tools the students reached for, how long the tasks took, and how accurate the final answers were.

The results revealed a clear pattern in how students navigate this digital landscape. When students relied solely on the conversational AI, they worked quickly, finishing their tasks in an average of 24 minutes. Their answers were fluent and well-structured, earning an accuracy score of 88 percent. When students stuck to the traditional search engine, they took longer, averaging 31 minutes, and their accuracy dropped to 80 percent. The search engine required them to jump between many different web pages to find and verify information, a process that often led to confusion or missed details. However, the most successful group was not the one that chose just one tool. The students who adopted a hybrid strategy—using the AI to generate a quick draft or explanation and then switching to the search engine to verify facts and confirm rules—achieved the highest accuracy of 90 percent.

This superior performance came with a cost. The hybrid group took the longest to complete their work, averaging 35 minutes. They had to stop, think, and switch contexts between the conversational interface and the list of search results. Despite the extra time, these students reported feeling more confident in their answers. They understood that the AI was a powerful generator of ideas but could occasionally make mistakes or offer vague explanations, while the search engine was a reliable source of truth but could be overwhelming to navigate. By combining them, they managed to get the best of both worlds: the speed and clarity of the AI, backed by the credibility of the search engine.

The study suggests that the future of learning in this context is not about choosing one technology over the other, but about integrating them. The researchers observed that students naturally gravitated toward a workflow where the AI handled the heavy lifting of writing and summarizing, while the search engine served as a safety net for checking facts. This behavior highlights a growing need for tools that do not force users to choose between efficiency and accuracy. The authors propose a new kind of interface where the conversational AI lives directly inside the search engine. In this imagined system, a student could ask a question, get a clear, summarized answer from the AI, and see the specific links to the original sources right alongside it. This would eliminate the need to constantly switch between different tabs or applications, reducing the mental effort required to verify information.

Ultimately, the research indicates that students are already acting as their own designers, figuring out how to blend these technologies to succeed. They are not treating the AI as a replacement for the search engine, nor are they ignoring the AI in favor of old methods. Instead, they are building a personal workflow that leverages the strengths of each. The study concludes that for high-stakes learning, the most effective path forward is a hybrid approach that respects the speed of artificial intelligence while maintaining the rigorous verification of traditional search, ensuring that the answers students find are not just fast, but also true.

Technical Summary: Search in Transition

Problem Statement
As Large Language Models (LLMs) increasingly influence academic information-seeking, university students are adopting hybrid workflows that combine conversational AI with traditional search engines. However, there is a lack of empirical research regarding how these tools are strategically integrated during high-stakes English language test preparation (e.g., IELTS, TOEFL). While LLMs offer efficiency in generating explanations and content, they may produce fluent but factually incorrect or contextually irrelevant information. Conversely, traditional search engines provide verifiable, authoritative sources but require significant cognitive effort to navigate, synthesize, and verify across multiple web pages. The core problem addressed is the trade-off between efficiency and reliability, and the specific strategies students employ to manage cognitive load and epistemic trust when solving English language test problems.

Methodology
The study employs a mixed-methods research design involving two distinct phases:

Quantitative Survey: A survey was administered to 140 university students from diverse disciplines (primarily Computer Science, Engineering, Business, and English) at United International University, Bangladesh. The instrument measured four core dimensions for both LLM-based tools and traditional search engines: usage frequency, satisfaction, perceived efficiency, and ease of use. Data were analyzed using descriptive statistics, Wilcoxon signed-rank tests (for paired comparisons), Friedman tests (for within-tool dimension differences), and Chi-square tests (for gender association with preference).
Qualitative Task-Based Interviews: In-depth interviews were conducted with 20 students who completed seven representative English language test tasks (summarization, grammar correction, vocabulary selection, paraphrasing, and writing tasks). Participants were categorized into four usage groups: LLM-only, Search-engine-only, Tool-balancing (sequential use of both), and Flexible-choice. Task performance was evaluated using predefined rubrics by two independent raters (inter-rater reliability $\kappa = 0.82$ ), measuring accuracy and completion time. Thematic analysis was applied to interview transcripts and open-ended survey responses.

Key Results

Perceived Differences: Descriptive statistics indicated that LLMs were perceived as more satisfying and efficient, while traditional search engines were rated higher for ease of use and usage frequency. However, paired statistical comparisons (Wilcoxon signed-rank tests) revealed no statistically significant differences between the two tool types across any of the four measured dimensions at the $\alpha = 0.05$ level.
Performance Trade-offs: In the interview phase, the Tool-balancing group (students who alternated between LLMs and search engines) achieved the highest task accuracy (90%) but required the longest completion time (35 minutes). The LLM-only group was the fastest (24 minutes) with an accuracy of 88%, while the Search-engine-only group had an accuracy of 80% and a completion time of 31 minutes.
Qualitative Themes: Four dominant themes emerged:
1. Task Suitability: LLMs are preferred for summarization, paraphrasing, and drafting; search engines are preferred for rule verification and authoritative sources.
2. Reliability: Students trust search engines for fact-checking due to concerns about LLM hallucinations, despite appreciating LLM fluency.
3. Cognitive Load: LLMs reduce cognitive effort by providing immediate outputs, but hybrid workflows, while time-consuming, lead to higher confidence and quality.
4. Interaction: LLMs are viewed as conversational tutors, whereas search engines are seen as stable but requiring more user initiative.
Preference: The majority of participants expressed a preference for a hybrid strategy over exclusive reliance on either tool.

Key Contributions

Empirical Evidence: The study provides comparative data on the performance of LLMs versus traditional search engines specifically within the context of English language test preparation, offering numeric values for proficiency and error rates based on different information-seeking strategies.
Behavioral Analytics: It offers qualitative insights into learner cognition, trust, and the transition between conversational AI and traditional search, highlighting the context-dependent nature of tool selection.
Hybrid Interface Model: Based on the findings, the authors propose a Human-Computer Interaction (HCI) oriented hybrid interface model. This conceptual prototype embeds an LLM assistant directly within a traditional search environment. The design aims to provide conversational assistance (summaries, drafts) while explicitly linking generated outputs to underlying sources to support verification, thereby reducing workflow fragmentation and cognitive load without compromising academic rigor.

Significance and Claims
The paper claims that LLMs and traditional search engines should not be viewed as competing technologies but as complementary components of an effective academic workflow. The study suggests that students naturally adopt adaptive strategies to balance efficiency with accuracy. The significance lies in demonstrating that a "balanced hybrid" approach yields the highest accuracy, albeit with a time cost, indicating a necessary trade-off in high-stakes learning environments.

The authors modestly frame their proposed prototype as a conceptual design intended to illustrate interaction principles rather than a fully implemented system. They acknowledge limitations, including the single-institution sample, the reliance on self-reported data, and the use of controlled tasks that may not fully reflect long-term learning experiences. Consequently, the paper calls for future research involving larger, cross-cultural samples, objective behavioral data (e.g., clickstream, eye-tracking), and longitudinal studies to validate the proposed hybrid interaction model in authentic educational settings.

Technical Summary: Search in Transition

More like this