Measuring brands in machine answers: A scoping review and validity framework for large language model brand-visibility audits
This scoping review of 38 empirical studies reveals that current AI brand-visibility audits suffer from significant validity gaps and conflicting conclusions due to methodological inconsistencies and commercial biases, prompting the author to propose a two-stage theoretical framework and new reporting standards to reconcile contradictions between brand inclusion and selection in large language model answers.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern marketplace, a brand's success often depends on whether it enters a consumer's mind at the right moment. For decades, marketers have studied the "consideration set," a mental shortlist of products a person actively thinks about before making a purchase. If a brand is not on that list, it is effectively invisible, no matter how good the product is. Today, a new gatekeeper has appeared between the shopper and that mental list: the artificial intelligence assistant. When a person asks a large language model for the best hiking jacket or a reliable accountant, the machine does not simply return a list of links. It generates a short, ranked list of specific brands, often with a brief explanation. This answer acts as a pre-selected menu, deciding which options the human will even consider.
The question facing researchers and business leaders is whether these machine-generated lists are fair, accurate, or even stable. Do they simply favor the biggest, most famous companies, or do they reward quality? Do they give a chance to smaller, local businesses, or are those brands locked out entirely? Because these systems are constantly changing and their internal logic is hidden, answering these questions requires a new kind of investigation. We cannot simply look inside the machine; we must ask it thousands of questions and carefully analyze the answers it gives back.
A recent scoping review by independent researcher Charbel Sami Ibrahim tackles this exact problem. The study acts as a map of a rapidly growing field, examining thirty-eight different investigations conducted between 2024 and 2026 that asked large language models to recommend brands. The goal was not to find a single, perfect answer, but to understand why these different studies often reach conflicting conclusions. Some reports claim that famous brands dominate every answer, while others suggest that smaller brands can compete if they have good ratings. By comparing the methods used in each study, the review reveals that these contradictions are not necessarily about the brands themselves, but about how the questions were asked and how the answers were counted.
The review found that the field is still in its early, fragile stages. Most of the studies examined were not peer-reviewed by independent experts, and many were conducted by groups with a commercial interest in selling tools to improve brand visibility. Furthermore, the data is often incomplete; many studies did not repeat their questions enough times to ensure the results were stable, and few checked their findings against real-world market data like sales figures or brand awareness surveys. The researcher identified a critical flaw in how many studies are built: they often fail to distinguish between two very different stages of the process. The first stage is whether a brand appears in the answer at all. The second stage is whether that brand is ranked highly or recommended over others once it has already appeared.
When the review separates these two stages, a clearer picture emerges. The evidence suggests that getting a brand into the answer in the first place is heavily dependent on how much text exists about that brand online. If a company is well-documented on websites, in reviews, and in directories, it is much more likely to be included in the machine's list. This is a matter of "membership." However, once a brand is on the list, the rules seem to change. The review suggests that when the machine has to choose between several brands that are already included, it may rely more on specific attributes, such as customer ratings or price, rather than just the brand's fame. This means a smaller brand with excellent reviews might beat a famous brand with poor reviews, but only if the smaller brand was lucky enough to get included in the list in the first place.
The study also highlights how easily the results can be swayed by the language used or the location of the person asking. Asking a question in a local language often brings up local businesses that would never appear in an English-language query. Similarly, the specific version of the AI model being used can change the entire set of recommended brands. Because of this, a single number claiming to show a brand's "visibility" is often misleading. A brand might appear in 90 percent of answers on one platform but only 10 percent on another, making a combined score meaningless without knowing the context.
Perhaps the most significant finding is that the design of the study itself dictates what it finds. Studies that simply ask the machine to name brands from scratch tend to show a massive advantage for the biggest, most famous companies. In contrast, studies that provide the machine with a fixed list of options and ask it to pick the best one often show that famous brands lose their advantage if a smaller competitor has better attributes. This explains why some reports say big brands rule the AI world while others say the playing field is level; they are measuring different parts of the process. The review concludes that to truly understand how these machines treat brands, researchers and managers must stop looking for a single score. Instead, they need to measure inclusion and selection separately, ask many different versions of the same question, and verify that the machine is actually identifying the correct brands. Until these standards are met, the numbers used to guide marketing budgets remain uncertain, and the true impact of artificial intelligence on brand competition remains a puzzle with many missing pieces.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.