Category-Conditioned Web Exposure Tracks Brand Presence in ChatGPT Fanout Queries
This study reveals that ChatGPT's selection of brands for initial web-search fanout queries is driven by a limited ranking process influenced by category-specific web presence rather than mere brand recognition, as evidenced by the fact that well-known brands are often omitted despite being known to the model.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
When you ask a modern, internet-connected artificial intelligence for the best cameras or the top project management tools, the machine does not simply scan the web and pick the first few results it finds. Before it ever looks at a single webpage, the AI writes its own search query. This first draft, created in a split second, often includes specific brand names like Canon or Sony. This happens before any human or computer has seen the actual search results. The machine is essentially deciding, in advance, which companies it is willing to talk about. This early decision is crucial because the search engine will then look for pages about those specific brands, making them far more likely to appear in the final answer. If a brand is left out of that first draft, it is almost invisible, even if the machine knows the brand exists.
This raises a puzzling question for anyone trying to understand how these systems work: if the AI knows a brand, why does it sometimes leave it out of that first draft? Is the machine forgetting the brand, or is it making a deliberate choice? A new study by independent researcher Antony Deroshan investigates this exact moment of decision. The research focuses on a specific type of artificial intelligence that can search the web, looking at how it chooses which company names to include in its initial search request. The study asks whether the machine is simply unaware of certain brands, or if it is actively filtering them out based on some hidden rule. The findings suggest that the machine knows the brands perfectly well but is forced to leave them out because it has a strict limit on how many names it can list at once.
To understand the mechanics of this, the researcher first needed to separate what the machine knows from what it chooses to say. He asked the AI, with its internet access turned off, to list the leading brands in categories like email marketing, customer service software, and cameras. The machine was able to name many brands, including some that it rarely mentions when actually searching the web. For example, when asked what brands came to mind for email marketing, the machine named "Campaign Monitor" in 49 out of 50 attempts. However, when the same machine was asked to perform a live search for the best email marketing tools, it included "Campaign Monitor" in the initial search query only once out of 50 times. The machine clearly knew the brand, yet it consistently omitted it from the search request. This gap between knowledge and action proved that the omission was not a failure of memory.
The researcher then built a simulation to figure out why this happened. He created a version of the AI that could write search queries but was not allowed to look up any real websites. He tested this simulator by asking it to list brands under different constraints. When he told the simulator to write a very short query, it dropped the same brands that the real machine dropped. But when he told the simulator to list as many brands as possible, it included all of them, including the ones that were usually left out. This revealed that the machine is not excluding brands because it dislikes them or doesn't know them; it is excluding them because it has a tight budget for how many names it can fit into a single sentence. The machine is essentially a librarian with a very short list to write down. If the list can only hold three names, it will pick the top three, and the fourth-best brand, no matter how famous, gets left off the list entirely.
The study also looked at what determines which brands make that short list. The researcher found a strong connection between how often a brand appears on the internet in relation to its specific category and how high it ranks on the machine's internal list. For instance, if a camera brand is mentioned frequently on websites that also talk about cameras, it is more likely to be included in the search query. However, this only works if the count is specific to the category. A company that makes many different products, like a giant tech firm that sells cameras, phones, and computers, might appear on millions of websites. If you count every single mention, that company might look like the most popular camera brand. But if you only count the websites that specifically mention cameras, a different brand might actually be more prominent. The study showed that the machine pays attention to this specific, category-focused popularity. It ranks brands based on how visible they are within their specific field, not just how famous the company is in general.
To test if this pattern held true for new categories, the researcher applied the same method to electronic signature software, a group of brands he had not studied before. He measured how often each brand appeared on websites related to signing documents and compared that to how often the machine included them in its simulated search lists. The results matched almost perfectly. The brands that appeared most often on relevant websites were the ones the machine chose to include in its short list. This suggests that the machine's internal ranking is deeply tied to how the brands are represented on the web, specifically in the context of the question being asked. The machine is not inventing these preferences; it is reflecting the structure of the information it has learned from the internet.
The research also ruled out several other possible explanations. It tested whether the machine simply preferred certain brands over others when reading text, regardless of the search query. In one test, the machine was shown three identical descriptions of a fictional company, with only the brand name changed. It consistently picked one brand over the others, showing a slight preference. However, this preference did not match the brands it chose for its search queries. The machine might like a brand when reading a description, but it chooses different brands when deciding what to search for. This proves that the decision to include a brand in a search query is a separate process from simply liking a brand name. The study also checked if removing low-quality or duplicate websites from the data would change the results. It did not. The connection between web visibility and the machine's choices remained strong even after cleaning up the data.
In the end, the study paints a clear picture of how these search-enabled machines operate. They are not random generators of text, nor are they simple search engines that list everything they find. They are systems that make a series of constrained choices. They know a vast amount of information, but when they need to ask a question to the internet, they must compress that knowledge into a very short list. The brands that make the cut are those that are not only known but are also highly visible in the specific context of the question. For a brand to be seen by these systems, it is not enough to just exist; it must be prominent in the right conversations. The machine's first draft is a gatekeeper, and the key to getting through that gate is not just being famous, but being the most relevant option for the specific topic at hand. This insight changes how we should think about visibility in the age of artificial intelligence, shifting the focus from simply being present online to being clearly and specifically associated with the questions people are asking.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.