Comparative Analysis of LLM-Based Conversational Agents as an Assistive Technology for Web Interaction
This study evaluates the effectiveness of four leading LLM-based conversational agents as assistive technologies for web navigation, finding that while they demonstrate promising potential to enhance accessibility for diverse user groups by interpreting page structures and guiding interactions, their performance varies significantly across different tools.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine the internet as a massive, bustling city. For some people, walking through this city is easy; the streets are wide, the signs are clear, and the buildings have ramps. But for others—like seniors, people who are blind, or those who find modern technology confusing—the city can feel like a maze of dead ends, hidden doors, and confusing signage.
This paper is like a field test for a new kind of digital tour guide. The researchers wanted to see if a specific type of Artificial Intelligence (called an LLM-based Conversational Agent, or "chatbot") could act as that guide, helping anyone navigate the web by reading the "blueprints" of a website and explaining how to get where they need to go.
Here is a simple breakdown of what they did and what they found:
The Problem: The Web is a Maze
Websites are getting more complex. They are filled with pop-ups, hidden menus, and complicated forms. For someone who isn't tech-savvy or who uses a screen reader (a tool that reads text out loud), finding a "Login" button or a "Search" bar can feel like trying to find a specific needle in a haystack while wearing blindfolds.
The Solution: The "Blueprint-Reading" Guide
The researchers tested four popular AI chatbots (Copilot, ChatGPT, Gemini, and Perplexity). They didn't just ask these bots to guess what a website looked like; they gave the bots access to the website's source code (the raw computer instructions that build the page).
Think of it this way:
- Old way: A guide looks at a building from the outside and guesses where the door is.
- New way: The guide is handed the building's architectural blueprints. They can see exactly where the door is, even if it's hidden behind a painting or a fancy curtain, and they can tell you exactly how to open it.
The Experiment: Two Rounds of Testing
The researchers ran two "field tests" on 13 different real-world websites (like Amazon, Wikipedia, government sites, and news portals).
Round 1 (Late 2024):
They tested Microsoft Copilot first. At the time, Copilot was unique because it could "see" the current webpage you were looking at directly.
- The Result: The bot was surprisingly good. It could describe the page, find hidden menus, and explain how to search or log in. It was like a knowledgeable local who knew the shortcuts.
- The Glitch: When websites were messy or used tricky computer tricks (like hiding menus behind scripts), the bot sometimes got confused, much like a human getting lost in a confusing building.
Round 2 (Late 2025):
Technology moved fast. The researchers tested four different bots (Copilot, ChatGPT, Gemini, and Perplexity) using a new method called RAG.
- What is RAG? Imagine the bot doesn't just have its own memory; it has a "search engine" attached that instantly grabs the current website's blueprints and feeds them to the bot before it answers. This lets all the bots "see" the page, not just Copilot.
- The Result: They asked the bots over 100 questions, such as "Where is the menu?" or "How do I buy this?"
What They Found (The Scorecard)
The researchers acted as judges, grading the bots on a scale of 0 to 3 (where 3 is a perfect, expert answer).
- The Winners: Gemini and Copilot were the top performers. They were the most reliable tour guides. They could usually find the right buttons, explain the menus, and tell you exactly how to complete a task.
- Analogy: If the website was a complex puzzle, these two bots could usually solve it and hand you the pieces in the right order.
- The Runner-Up: ChatGPT did a decent job but was slightly less consistent than the top two.
- The Struggler: Perplexity had a harder time. It often tried to answer by searching the internet for general info rather than looking strictly at the specific website's blueprints.
- Analogy: If you asked Perplexity, "Where is the exit in this building?", it might say, "Well, usually buildings have exits on the left," instead of looking at the specific floor plan you gave it.
The "Gotchas"
Even the best bots had trouble in certain situations:
- The "Magic" Doors: Some websites use complex computer code to make menus appear only when you click something. The bots sometimes missed these "magic doors" because they were hard to read in the blueprints.
- The "Do Not Enter" Signs: Some websites (like Amazon or The Wall Street Journal) have security guards that block the bots from reading their blueprints. When this happened, the bots simply couldn't answer.
- Hallucinations: Occasionally, a bot would confidently give you a phone number or address that wasn't actually on the page, making up facts based on what it knew from the past.
The Bottom Line
The paper concludes that these AI chatbots are promising new tools for helping people navigate the web. They aren't perfect yet, but they are much better than guessing.
- For the average user: They offer a friendly, conversational way to get help without needing to be a tech expert.
- For people with disabilities: They could eventually replace the need for a human helper, giving users more independence.
The researchers say that while the technology is still evolving (like a car that is being upgraded every month), the idea of using these "blueprint-reading" guides to make the internet accessible to everyone is not just a dream—it's working today.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.