UIS-Digger: Towards Comprehensive Research Agent Systems for Real-world Unindexed Information Seeking

This paper identifies the critical limitation of current LLM-based agents in accessing unindexed information, introduces the first dedicated UIS-QA benchmark to quantify this challenge, and proposes UIS-Digger, a multi-agent framework that significantly outperforms state-of-the-art models by effectively combining dual-mode browsing and file parsing to retrieve vital unindexed data.

The Gist

Imagine the internet as a massive, sprawling library.

For a long time, AI assistants (the "librarians") have been incredibly good at finding books that are already cataloged on the main shelves. If you ask, "Who won the 2024 World Cup?", they instantly pull the right book off the shelf. This is what the paper calls Indexed Information Seeking (IIS). The search engines have already read these pages, summarized them, and put them in a neat index.

The Problem: The "Hidden" Library

But here's the catch: A huge part of the library is unindexed. These are the books hidden in the back rooms, the files tucked inside drawers, the dynamic displays that change every minute, and the pages that search engine robots (crawlers) simply can't read or aren't allowed to enter.

The paper argues that current AI librarians are terrible at finding these hidden items. If you ask a question that requires digging into a specific PDF on a government website, or clicking through a complex menu to find a date, the AI often gives up, guesses wildly (hallucinates), or says, "I don't know," even though the answer is right there if you just knew how to look.

The authors call this missing skill Unindexed Information Seeking (UIS).

The Solution: UIS-QA (The Test)

To prove how bad the current situation is, the researchers created a new test called UIS-QA.

• The Analogy: Imagine giving a librarian a list of 110 very specific questions. Some ask for facts hidden inside a company's annual report PDF, others ask for data hidden behind a login-free but complex dropdown menu on a museum's site.
• The Result: The top-tier AI librarians, who usually score 70% or higher on standard tests, crashed on this new test. They only got about 25% right. It was like a brilliant scholar failing a test because they didn't know how to open a locked filing cabinet.

The Hero: UIS-Digger (The New Librarian)

To fix this, the team built a new AI system called UIS-Digger. Think of it not as a single librarian, but as a specialized excavation team.

1. The Team Structure: Instead of one brain trying to do everything, UIS-Digger uses a team of four specialized agents:

   • The Planner: The boss who breaks the big question into small steps.
   • The Searcher: The one who uses Google to find the general area.
   • The Surfer: The explorer who actually goes into the website. Crucially, this agent has two modes: it can read text quickly, but if the page has a chart or a weird button, it can switch to "visual mode" (like taking a screenshot) to understand what it's seeing.
   • The Reader: The specialist who opens downloaded files (PDFs, Excel sheets) and reads them line by line.

2. The Training (SFT & RFT):

   • SFT (Supervised Fine-Tuning): They taught the team by showing them examples of how to solve these hard problems. It's like giving them a map and saying, "Here is how you open that specific drawer."
   • RFT (Rejection Sampling Fine-Tuning): This is the "try, fail, learn" phase. The team tried to solve thousands of problems. If they got the answer right, they kept the method. If they failed or took a silly path, that attempt was thrown away. Over time, the team learned to be much more efficient and less likely to get lost.

The Results

The new team, UIS-Digger, didn't just beat the old librarians; it completely changed the game.

• While the best existing systems scored around 25% on the hard test, UIS-Digger scored 27.3%.
• The Twist: They did this using a relatively small, efficient brain (a 30-billion parameter model), beating systems that use massive, expensive super-brains (like O3 or GPT-4.1).

Why This Matters

The paper concludes that we can't just rely on bigger and bigger AI brains to solve these problems. We need to teach them how to interact with the messy, unindexed parts of the internet.

In a nutshell:
Current AI is like a genius who can recite the encyclopedia but can't find the hidden treasure map in the attic. UIS-Digger is a new kind of AI that learns to climb ladders, open locked boxes, and read the fine print in the dark, proving that for real-world research, how you look is just as important as how smart you are.

Technical Summary

Here is a detailed technical summary of the paper "UIS-DIGGER: Towards Comprehensive Research Agent Systems for Real-World Unindexed Information Seeking".

1. Problem Definition: Unindexed Information Seeking (UIS)

The paper identifies a critical blind spot in current Large Language Model (LLM) based information-seeking agents: Unindexed Information Seeking (UIS).

• Context: Existing agents excel at Indexed Information Seeking (IIS), where information is retrieved via search engines (e.g., Google) that have crawled and indexed webpages.
• The Gap: Vital information often resides in "unindexed" zones, including:
  • Content hidden behind interactive elements (e.g., date pickers, dynamic filters).
  • Information embedded in downloadable files (PDFs, XLSX) not directly parsed by search snippets.
  • Deeply nested pages not reached by standard crawlers.
  • Content excluded from search indices due to ranking algorithms or technical limitations.
• The Challenge: Current agents fail when the answer cannot be found in the initial search engine results (SERP) or the immediate snippets. They either hallucinate, refuse to answer, or get stuck because their action space is limited to text-based search and simple browsing.

2. Methodology

The authors propose a two-pronged approach: a new benchmark to evaluate the problem and a novel agent framework to solve it.

A. The Benchmark: UIS-QA

To systematically evaluate UIS capabilities, the authors introduced UIS-QA, the first dedicated benchmark for this task.

• Dataset: 110 expert-annotated Question-Answer (QA) pairs.
• Construction Principles:
  • Objectivity: Answers are factual, deterministic, and unique (e.g., specific numbers, dates).
  • Authoritativeness: Sources are official (government, corporate, code repos) but require deep navigation to find.
  • Static Nature: Answers are time-invariant to ensure reproducibility.
  • Verifiability: Answers can be verified by rule-based LLMs.
  • Accessibility: No CAPTCHAs or login walls.
• Filtering: A rigorous pipeline ensures questions cannot be solved by standard search engines (IIS) or internal LLM knowledge. If a question requires downloading a file or interacting with a dynamic UI element to find the answer, it is classified as UIS.
• Comparison: Unlike benchmarks like GAIA or BrowseComp, UIS-QA explicitly requires agents to navigate unknown starting points, interact with complex web elements, and rely on unindexed sources.

B. The Framework: UIS-Digger

UIS-Digger is a multi-agent system designed to handle the complex action spaces required for UIS.

• Architecture: It employs four specialized agents working in a ReAct-style loop:
  1. Planner: Decomposes queries into sub-tasks.
  2. Web Searcher: Handles initial search and crawling.
  3. Web Surfer: The core component for deep browsing. It features a Dual-Mode Browsing Strategy:
     • Textual Mode: Efficient for reading standard content.
     • Visual Mode: Uses screenshots for complex layouts, graphs, or interactive elements.
     • Shared Memory: Maintains a unified browser state across both modes to avoid synchronization overhead.
  4. File Reader: Parses downloaded files (PDF, XLSX, DOCX) which often contain the "unindexed" answers.
• Training Strategy: The underlying LLM (approx. 30B parameters) is optimized using a two-stage process:
  1. Synthetic Data Construction: QA pairs are generated from real-world deep websites and simulated environments (virtual sites with interactive widgets like date pickers).
  2. Supervised Fine-Tuning (SFT): Cold-start training using high-quality trajectories from a teacher model to teach basic tool usage and navigation.
  3. Rejection Sampling Fine-Tuning (RFT): The model generates multiple trajectories; only correct and non-trivial ones are retained. Crucially, samples are re-weighted by difficulty (favoring harder tasks) to bootstrap complex reasoning capabilities.

3. Key Contributions

1. Problem Formalization: Defined and formalized the UIS problem, distinguishing it from IIS and highlighting its necessity for robust real-world agents.
2. UIS-QA Benchmark: Created the first rigorous, expert-annotated benchmark specifically for evaluating unindexed information seeking, featuring 110 high-quality, cross-validated samples.
3. UIS-Digger System: Developed a versatile multi-agent framework with dual-mode browsing and file parsing capabilities.
4. Training Pipeline: Demonstrated that a relatively small (~30B) LLM, when properly tuned with SFT and difficulty-weighted RFT, can outperform much larger, sophisticated models on UIS tasks.

4. Experimental Results

The authors evaluated UIS-Digger against state-of-the-art (SOTA) agents, including commercial systems (GLM-4.5, Doubao, Gemini) and research frameworks (Memento, Tongyi-DR, WebSailor).

• Performance on UIS-QA:
  • UIS-Digger (Pangu-38B/Qwen3-32B): Achieved 27.27% accuracy.
  • SOTA Baselines: Even top-tier systems like Memento (using O3 + GPT-4.1) and Tongyi-DR (GPT-4o) dropped significantly, scoring only 25.5% and 23.6% respectively.
  • Comparison: While these models scored >70% on GAIA and >45% on BrowseComp-zh, their performance collapsed on UIS-QA, proving the distinct difficulty of unindexed tasks.
• Training Impact:
  • SFT provided a massive boost (e.g., +13.6% for Pangu backbone).
  • RFT provided further gains (+4.6%), demonstrating the value of reinforcement-style refinement.
• Error Analysis:
  • Retrieval Failure: Many agents failed to find the correct root domain.
  • Source Selection: Agents often retrieved the correct domain but failed to navigate to the specific deep page or file.
  • Interaction Failure: Agents struggled with complex UI interactions (e.g., selecting dates, reading dynamic graphs) even when on the correct page.

5. Significance and Future Directions

• Paradigm Shift: The paper argues that current agent evaluation is flawed because it over-relies on indexed knowledge. True "deep research" requires the ability to excavate unindexed data.
• Efficiency vs. Capability: The results show that architectural design (multi-agent, dual-mode browsing) and targeted training (SFT/RFT) are more critical for UIS than simply scaling up model parameters. A 30B model with the right framework outperformed systems using O3 and GPT-4.1.
• Future Work: The authors note that while 27.27% is a strong baseline, UIS remains a highly difficult challenge. Future work will focus on improving self-improvement via reinforcement learning and generating higher-quality synthetic data to bridge the gap between simulated and real-world UIS scenarios.

In conclusion, UIS-Digger establishes a new standard for information-seeking agents, proving that proactive interaction with unindexed sources is essential for the next generation of AI research tools.