WARP: Wasserstein-Aligned RAG for Population Opinions

In the modern digital age, when people want to know what others think about a product, a service, or a policy, they often turn to artificial intelligence to summarize thousands of reviews. These systems, known as retrieval-augmented generation tools, scan vast libraries of text to find relevant information and weave it into a single, coherent answer. The promise is efficiency: instead of reading hundreds of conflicting opinions, a user gets a quick snapshot of the consensus. However, this convenience comes with a hidden flaw. Standard systems are designed to find documents that look most similar to the question asked. In doing so, they often favor the loudest or most common voices, effectively silencing minority perspectives. If sixty percent of hotel guests are happy but forty percent complain about noise, a standard summary might only reflect the happy guests, presenting a distorted picture that feels factual but misses the full reality. The challenge for researchers is to build a system that does not just find relevant documents, but faithfully represents the true distribution of human opinion, ensuring that the final summary reflects the actual balance of views, including the less frequent ones.

A team of researchers at Amazon has developed a new method called WARP to solve this problem of skewed summaries. Their approach treats the collection of opinions not just as a list of documents to be ranked by relevance, but as a population that needs to be represented fairly. The core idea is to measure how well a selected group of reviews matches the known distribution of opinions across the entire dataset. To do this, the team uses a mathematical concept called the Wasserstein distance, which is a way of measuring the difference between two distributions by considering the order and intensity of the data. Unlike older methods that simply count how many positive or negative reviews are present, this new metric understands that confusing a strongly positive opinion with a strongly negative one is a much bigger error than confusing a positive opinion with a neutral one. By respecting this natural ordering of intensity, the system can select a set of evidence that mirrors the population's true sentiment profile.

The researchers tested their system across three different domains: online forums for sellers, hotel reviews, and automotive reviews, covering more than thirty-five thousand documents. They found that standard search methods often bury underrepresented views, leading to summaries that feel one-sided. WARP addresses this by first checking if the initial batch of retrieved documents is missing any specific types of opinions. If it is, the system performs a targeted search to find those missing voices before selecting the final set of documents. It then uses its specialized metric to choose the final group of reviews that best matches the population's distribution. The results were significant: across all three domains, the new method reduced the error in representing the population's opinions by at least forty-three percent compared to standard methods. In some cases, the improvement was as high as seventy-nine percent.

Beyond just selecting better documents, the researchers wanted to know if these improvements actually mattered to the final answer generated by the AI. They asked a panel of five different large language models to act as judges, comparing summaries created with the new method against those created with standard methods. In eighty-six percent of the cases where the judges could decide a winner, they preferred the summaries generated from the WARP-selected evidence. This suggests that the technical improvements in selecting documents directly translate to better, more balanced answers for the user. The system achieved these results while maintaining a speed suitable for real-world use, adding less than three hundred milliseconds to the process, which is a fraction of a second.

The study also explored how the method performs under different conditions, such as when there are very few documents available for a specific topic or when the initial data is noisy. The researchers found that their approach is robust; even when the labels describing the sentiment of the reviews were intentionally corrupted or when the population data was imperfect, the system still outperformed standard methods. They demonstrated that the success of the method relies on the specific way it measures the difference between opinions, rather than just random shuffling of the results. By using a metric that understands the ordered nature of human sentiment, the system can navigate complex opinion landscapes where simple counting methods fail.

One of the key insights from the work is that diversity alone is not the goal. Previous attempts to fix biased summaries often focused on simply making the selected documents different from one another. However, the researchers showed that once a certain level of variety is reached, simply adding more different documents does not help if they do not reflect the correct proportions of the population. The goal is not just to have a mix of views, but to have a mix that accurately reflects how common each view is in the real world. This distinction is crucial for applications where understanding the weight of an opinion is as important as knowing that the opinion exists.

The researchers acknowledged that their work has limitations. The system relies on having pre-labeled data to know what the population distribution looks like, and it operates on a single scale of sentiment intensity. It does not yet handle complex scenarios where a review might praise one aspect of a product while criticizing another, or where multiple distinct issues need to be balanced simultaneously. Furthermore, the study was conducted offline, meaning the results show how well the system performs in a controlled environment, but the impact on real-world user trust and decision-making has not yet been tested in live traffic. Despite these boundaries, the work provides a clear path forward for building AI systems that do not just summarize what people say, but represent how people think.

In the end, the research demonstrates that it is possible to engineer a retrieval system that respects the nuance of human disagreement. By moving beyond simple similarity matching and incorporating a method that understands the structure of opinion, the team has created a tool that can produce summaries that are not only relevant but also representative. This ensures that when a user asks what people think, the answer they receive includes the full spectrum of experience, from the enthusiastic majority to the critical minority, offering a clearer and more honest picture of the collective voice.