The "Curse of Knowledge" in LLM Query Simulation: Concept Provenance for Tracing Answer-Side Intrusion
This paper introduces "concept provenance," a framework for detecting and categorizing answer-side knowledge intrusion in LLM-generated search queries, revealing that while such intrusion is prevalent and distinct from natural human variation, it has minimal impact on aggregate retrieval evaluation metrics but can be effectively eliminated through post-generation selection.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
When a person searches for information online, they start with a question born from what they already know and what they need to find out. They type words into a search box based on their current understanding, hoping the computer will guide them to answers they do not yet possess. This moment of asking is a strict boundary: the user has the question, but not the answer. If the question itself contains the answer, the search process is broken, because the user is no longer searching; they are simply confirming what they already know. In the modern world, researchers are increasingly using artificial intelligence to simulate these human searchers. They ask large language models to generate thousands of different search questions to test how well search engines work. However, these artificial minds often suffer from a hidden flaw. Because they have read vast amounts of text, they sometimes accidentally include the answer inside the question they are supposed to ask. This creates a false test, where the search engine appears to work perfectly simply because the question gave it the answer in advance.
A team of researchers at RMIT University set out to expose this flaw, which they call the "curse of knowledge" in search simulations. They developed a new way to look at the words inside a search query, tracing the origin of every single concept to see where it came from. They asked a simple but critical question: could a real human, knowing only the story of their problem and nothing else, have come up with this specific word? To find the answer, they analyzed over 77,000 search queries generated by eight different artificial intelligence models across 100 different search topics. They compared these machine-made questions against a large collection of real questions written by thousands of human volunteers who were given the same stories but no access to the search results.
The researchers created a system to sort every word in a query into one of several categories. Some words came directly from the story the user was told. Others were common words that many humans used. A third group consisted of rare words that humans did use, but only occasionally. The final group, which the researchers were most interested in, contained words that no human had ever used in that context, yet were highly specific to the correct answer. These were the intruders. They found that in nearly every single topic, the artificial intelligence slipped in these answer-side concepts. Across all the queries, these intruding words made up about 7.4 percent of the non-generic terms. In 97 out of the 100 topics tested, the artificial intelligence included at least one word that a real human, starting from scratch, would never have thought to type.
The study revealed that this intrusion happens regardless of how the researchers tried to stop it. They tried giving the artificial intelligence specific instructions to stay within the bounds of the story, and they tried framing the task as if the machine were a medical expert or a layperson. While these changes reduced the number of intrusions slightly, they did not eliminate them. The artificial intelligence kept reaching into its vast internal library of facts and pulling out terms that belonged to the answer, not the question. The researchers discovered that this behavior was not just a random error; it was a fundamental mismatch between how the machine thinks and how a human searches. The machine knows the answer, so it assumes the question should reflect that knowledge.
Interestingly, the presence of these intruding words did not drastically change the overall score of how well the search engine performed in a broad sense. The researchers found that the total difference in search results caused by these words was surprisingly small when looking at the big picture. However, when they looked at individual searches, the effect was significant. When they removed these specific intruding words from a query, the search engine's ability to find the right documents dropped noticeably. This showed that while the intrusion might not ruin a whole test, it does distort the specific path a single search takes, pulling it toward documents the user was not supposed to see yet.
To solve this, the researchers tested whether they could simply filter out the bad queries after the artificial intelligence generated them. They tried a strategy of generating many possible questions and then selecting only the ones that passed a strict check for boundary compliance. By doing this, they were able to remove the intruding words from 99 percent of the topics. They found that they could get almost perfect results by generating a large pool of options and then carefully choosing the ones that stayed within the human limits. This proved that the problem was not unsolvable, but it required a step of verification that the artificial intelligence could not do on its own.
The study also highlighted a deeper issue with how we trust artificial intelligence to judge itself. When the researchers asked other artificial intelligence models to act as judges and decide which words were intrusions, the machines failed. They consistently agreed with each other that the intruding words were normal, common knowledge, whereas human judges correctly identified them as specialized terms that a general person would not know. This suggests that artificial intelligence models tend to project their own advanced knowledge onto the average person, assuming that what is obvious to them is obvious to everyone.
Ultimately, this research provides a new lens for understanding how artificial intelligence interacts with human tasks. It shows that simply asking a machine to act like a human is not enough; the machine must be checked to ensure it is not using information it should not have. The researchers concluded that while artificial intelligence can be a powerful tool for simulating search behavior, it requires a human-designed safety net to ensure the simulation remains honest. Without this check, the tests we run to improve search engines might be measuring the machine's ability to guess the answer, rather than the engine's ability to find it.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.