Robustness of IR Models to Collection Growth

Imagine a vast library where books are constantly being added, updated, and removed. In the digital world, this library is the internet, and the task of finding a specific book within it is called information retrieval. When you type a question into a search engine, a complex system scans millions of documents to find the ones that best answer your query. Ideally, this system should be stable; adding new, unrelated books to the library should not make it harder to find the original, relevant ones. If a search engine works well today, it should work just as well tomorrow, even if the collection of documents has grown significantly with content that has nothing to do with your search. This stability is the core question researchers at the University of Glasgow set out to investigate. They wanted to know if the mathematical engines behind modern search tools can handle a growing library without losing their way, or if the act of adding new, irrelevant information inevitably confuses the system.

To test this, the researchers created a controlled experiment by merging two very different collections of text. One collection, known as TREC-COVID, contains documents specifically about the pandemic, created after 2019. The other, MS MARCO, is a massive collection of general web passages created before 2019. By combining them, they formed a single, heterogeneous library where the pandemic documents made up only a tiny fraction—about 1.9 percent—of the total. They then asked search queries designed for the pandemic collection to run against this new, mixed library. The goal was to see if the search results for pandemic questions would degrade because the system was now distracted by millions of unrelated, pre-pandemic web pages. This setup allowed them to measure a specific property they call robustness: the ability of a search model to maintain its effectiveness when non-relevant documents are added to the mix.

The study examined two main types of search models, distinguished by how they look at the documents they are ranking. The first type, which the researchers call multi-document-agnostic, treats each document as an isolated island. When it scores a document, it looks only at the relationship between the search query and that single document, ignoring everything else in the library. The second type, called multi-document-dependent, is more like a group discussion; it looks at the context of other documents to decide how relevant a specific one is. For example, some of these models might look at the top results from a first pass to refine their scoring, or they might use statistics about how common certain words are across the entire collection to adjust their answers. The researchers hypothesized that this reliance on the broader collection might make the second type of model more fragile when the library grows with unrelated content.

The results of the experiment revealed a clear pattern. When the researchers added the millions of unrelated web pages to the pandemic collection, the search models that relied on the broader context of the collection suffered a significant drop in performance. Their ability to find the correct pandemic documents weakened noticeably. In contrast, the models that treated each document independently were much more resilient. They maintained their ability to find the relevant information even when the library was flooded with irrelevant noise. This suggests that for the initial stage of searching, where a system must sift through a massive pool of candidates, ignoring the surrounding context and focusing strictly on the match between the query and the document is a safer strategy. The models that tried to use the "group context" of the entire library were easily swayed by the sheer volume of new, unrelated material, effectively getting lost in the noise.

The researchers also tested a common technique called pseudo-relevance feedback, where a system tries to improve its search by pretending the top results it found are correct and using them to refine the query. In this mixed-library setting, this technique backfired. Instead of helping the system focus, the feedback mechanism pulled the search results toward the dominant collection of general web pages, further reducing the accuracy for the specific pandemic queries. This happened because the system was using the wrong documents to guide its thinking, reinforcing a bias toward the larger, unrelated part of the library. However, the story changed slightly at the second stage of the search process, known as re-ranking. Once the initial search had narrowed down the list to a smaller set of candidates, both types of models—those that looked at context and those that did not—performed equally well. At this stage, the addition of irrelevant documents did not significantly hurt their ability to order the final results correctly.

Ultimately, the study demonstrates that current search architectures have a systematic weakness when collections grow. The way a model conditions its ranking on other documents plays a critical role in its stability. While models that ignore the broader collection are more robust during the initial search, the addition of non-relevant documents still causes some performance degradation across the board, meaning no system tested was perfectly immune to the problem. The findings suggest that as digital libraries continue to expand, relying on models that are explicitly designed to handle this growth is essential. The research does not claim to have solved the problem, but it provides a clear measurement of the issue and a taxonomy for understanding why some systems fail when the library gets bigger. It highlights that for search engines to remain effective in a dynamic world, their underlying logic must be rethought to account for the constant influx of new, unrelated information.