SimpleWikiSearch: A Clean Offline Wikipedia Environment for Agentic Search
This paper introduces SimpleWikiSearch, a fully specified and reproducible offline Wikipedia environment with explicit corpus construction, retrieval stacks, and tool contracts designed to standardize the evaluation of agentic search systems by isolating environmental variables from agent performance.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to teach a super-smart robot how to answer questions using a giant library. You give the robot a brain (a Large Language Model) and tell it, "Go find the answer in the books!" But here's the catch: the robot's success doesn't just depend on how smart its brain is. It depends entirely on how the library is organized. Is the library a messy attic where books are torn into tiny, confusing scraps? Is the librarian handing you the right page, or just a random sentence? Is the robot allowed to ask for the whole book, or only a specific paragraph?
In the world of artificial intelligence, researchers often build these "robot librarians" to solve complex questions. They usually compare them by looking at the final score: "Did the robot get the right answer?" But for a long time, everyone ignored the messy details of the library itself. One team might use a library where books are cut into 100-word pieces, while another uses full chapters. If Team A's robot wins, is it because the robot is smarter, or because the library was easier to navigate? This paper steps in to say, "Wait a minute, we need to stop guessing and start measuring the library itself." The authors want to create a fair, standardized playing field where we can see exactly how the environment shapes the robot's behavior, rather than just blaming or praising the robot's brain.
Enter SimpleWikiSearch, a new, super-clean "offline library" built specifically for testing these AI agents. Think of it as a perfectly organized, digital version of Wikipedia that the robot can explore without ever touching the real internet. The authors didn't just build a library; they built the rules of the library. They took the entire English Wikipedia, cleaned it up, and chopped it into logical chunks—not tiny, broken 100-word snippets like old systems did, but larger, sensible sections that keep the context intact, like a whole chapter or a full table of data.
The robot interacts with this library using three simple tools, like a set of magic wands:
- Search: The robot can ask a question, and the library hands back a list of the most relevant "snippets" with links.
- Open URL: If the robot likes a snippet, it can click the link to read the full section or even the entire article.
- Submit Answer: Once the robot is confident, it hands in its final answer.
The paper's main finding is that by standardizing this environment, we can finally compare different AI models fairly. The authors tested this setup with several open-source AI models (specifically versions of Qwen3.5 with 4 billion and 9 billion parameters) on six different question-answering challenges. They found that bigger brains (the 9B model) generally did better, but not always in a straight line—sometimes the bigger model got slightly lower scores on specific tasks, showing that "bigger" isn't a magic fix-all. They also compared these open-source models against some of the most powerful commercial "black box" models. Interestingly, the commercial models often got higher "judge" scores (meaning a human-like AI thought their answers were factually correct even if the wording was different), while the open-source models sometimes had better "exact match" scores.
Crucially, the paper reveals that the way the robot uses the library matters just as much as the answer it gives. By tracking every move the robot made, the authors discovered that harder questions (like those requiring multiple steps of reasoning) forced the robots to take many more "rounds" of searching and clicking. On the toughest challenges, the robots often ran out of time or turns before they could submit an answer, even if they were close. This suggests that future improvements shouldn't just focus on making the robot's brain smarter, but also on helping it navigate the library more efficiently.
The authors are very clear that they aren't claiming to have invented a new, super-smart robot algorithm. Instead, their contribution is the environment itself. They provided a reproducible, transparent "harness" where anyone can run the same tests, see the exact same results, and understand exactly why a robot succeeded or failed. They even released all the data, including the robot's thought process and every click it made, so the whole community can study the behavior. In short, SimpleWikiSearch is the referee that finally ensures everyone is playing by the same rules, making it possible to tell if a robot is truly brilliant or just lucky because the library was easy.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.