How Generative AI Disrupts Search: An Empirical Study of Google Search, Gemini, and AI Overviews
This empirical study analyzes how Generative AI disrupts traditional web search by comparing Google Search, AI Overviews, and Gemini, revealing significant differences in source selection, lower consistency, and reduced visibility for sites blocking AI crawlers, while calling for sustainable revenue frameworks to support the ecosystem.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine the internet as a massive library. For years, if you asked a librarian (Google Search) for a book on "how to fix a leaky faucet," they would hand you a list of the most popular, trusted manuals from the top shelves. You'd pick one, walk over, and read it yourself.
This paper is a report card on what happens when that librarian gets a new, super-smart, but sometimes over-eager assistant (Generative AI) who doesn't just hand you a list of books—they read them all, summarize the best parts, and write a new story for you right there at the desk.
Here is what the researchers found when they compared the old librarian, the new AI assistant, and a different AI chatbot (Gemini) using over 11,000 real questions.
1. The New Assistant is Everywhere (But Picky)
The researchers found that for about half of all real questions people ask, this new AI assistant (called an "AI Overview" or AIO) jumps in and gives you a summary before showing you the list of links.
- The Analogy: It's like walking into a store and having a salesperson immediately shout out, "Here's the answer!" before you even look at the shelves.
- The Catch: The assistant loves long, specific questions (like "How do I clean a pizza stone?") but ignores simple keyword searches (like "pizza stone"). It also loves controversial topics, showing up 94% of the time for political debates.
2. They Are Looking at Different Shelves
The biggest shock was that the three "librarians" (Traditional Google, the AI Overview, and Gemini) are rarely looking at the same books.
- The Analogy: Imagine three friends trying to describe the same movie. You'd expect them to mention the same actors and scenes. Instead, the researchers found that the AI assistants and the traditional search engine only agreed on about 15-20% of the sources they used. They are essentially reading from completely different sections of the library.
- The Bias: The traditional search engine sticks to the "famous" books (popular news sites, government, and universities). The AI assistants, however, are much more likely to pull information from Google-owned properties (like YouTube or Google's own sites) and ignore the famous, trusted sources.
3. The "Do Not Disturb" Sign Doesn't Work
Some big publishers (like The New York Times or Facebook) put up a "Do Not Disturb" sign (a technical block) to stop the AI from reading their content for training.
- The Expectation: You'd think the AI would respect the sign and skip those books.
- The Reality: The traditional search engine respects the sign. But the AI Overview? It seems to ignore the sign. However, the researchers found a twist: even though the AI can read these blocked sites, it still rarely cites them. It's as if the AI sees the sign, decides not to read the book, and then forgets the book exists entirely. This means publishers who block the AI are actually hurting their own visibility in the new system.
4. The Assistant is Fickle (Inconsistent)
If you ask the same question twice, the traditional librarian gives you the same list of books. The AI assistant? Not so much.
- The Analogy: If you ask the AI "What's the best pizza place?" today, it might tell you about "Tony's." If you ask again in five minutes, or if you ask from your phone instead of your laptop, it might suddenly tell you about "Luigi's."
- The Problem: The AI is also very sensitive to tiny changes. If you type "What is" instead of "What's," it might give you a completely different answer. This makes it hard to trust that you are getting the same information twice.
5. The "High-Stakes" Danger Zone
When people ask about serious topics like politics or trending news, the AI is very eager to answer, but it's often wrong or biased.
- The Finding: For political questions, the AI often picks sources that are less credible (like fan wikis or satirical news) compared to the traditional search engine.
- The Hallucination: In one real example, the AI confidently declared a boxer had won a fight that hadn't happened yet, citing a joke post from a satirical Facebook page as proof. It's like a student guessing the answer on a test and writing a whole essay to justify a wrong guess.
The Big Picture
The paper concludes that this new way of searching is a double-edged sword.
- For Users: It's convenient, but you have to be careful because the AI might be making things up or using unreliable sources, especially for serious topics.
- For Publishers: The rules of the game have changed. The "popular" websites are getting less attention, while niche sites are getting more. Also, blocking the AI from reading your content might actually make your site disappear from the AI's answers entirely.
- The Solution: The authors suggest we need a new deal. Publishers and AI companies need to figure out how to share the money (revenue) so that high-quality websites stay in business, and the AI has access to the best, most reliable information to give us.
In short, the AI assistant is a powerful new tool, but it's currently reading from a different, less stable, and sometimes less trustworthy set of books than the old librarian.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.