iAgentBench: Benchmarking Sensemaking Capabilities of Information-Seeking Agents on High-Traffic Topics

This paper introduces iAgentBench, a dynamic open-domain question answering benchmark designed to evaluate the cross-source sensemaking capabilities of information-seeking agents on high-traffic topics by requiring the integration of evidence from multiple sources rather than simple retrieval.

The Gist

Imagine you are trying to solve a mystery, like figuring out why a local park suddenly closed.

If you just ask a smart friend (an AI), they might quickly find one newspaper article that says, "The park closed for repairs." That's easy. But what if the real story is much more complex? What if you need to read a city council meeting transcript, a local news blog, a social media post from a resident, and a weather report to understand that the park closed because a storm damaged a fence, which led to a council vote, which was influenced by a resident's complaint?

This is the problem the paper iAgentBench is trying to solve.

The Problem: The "Single Clue" Trap

Most current tests for AI are like a game of "Find the Needle in a Haystack." They give the AI a question and a single document, and the AI just has to find the one sentence that holds the answer.

But in the real world, information-seeking isn't about finding one needle. It's about sensemaking. It's about taking ten different needles from ten different haystacks, realizing they are all part of the same pattern, and weaving them together to tell a complete story. Current AI is great at finding the needle, but often terrible at weaving the tapestry.

The Solution: iAgentBench (The "Detective's Case File")

The authors created a new testing ground called iAgentBench. Think of it not as a multiple-choice quiz, but as a dynamic detective case file that changes every day.

Here is how it works, using a simple analogy:

1. The "Hot Topic" Radar (Traffic-Driven Seeds)

Instead of making up fake questions like "Who was the 5th president of Peru?", the system looks at what real people are actually searching for right now. It uses a "radar" (called GDELT) to see what events are exploding in popularity.

• Analogy: Imagine a news editor who only picks stories that are currently trending on Twitter and Google. The questions are always fresh and relevant to real life.

2. The "Scavenger Hunt" (The Web Corpus)

When a topic is picked, the system goes out and grabs the top search results from the open web. It doesn't just grab one page; it grabs a whole bunch of articles, blogs, and reports.

• Analogy: The AI is sent to a library with a specific list of books to read. It can't just read one book; it has to read the whole section.

3. The "Story Map" (Graph Construction)

This is the clever part. The system doesn't just dump the text on the AI. It first reads all the articles and builds a map (a graph) of the story. It groups related ideas into "neighborhoods" (communities) and draws lines between them to show how they connect.

• Analogy: Imagine the AI is a cartographer. It takes a messy pile of notes and draws a map where "The Storm" is connected to "The Fence" which is connected to "The Council Vote." It highlights the bridges between these islands of information.

4. The "Bridge-Building" Questions

Finally, the system generates questions that force the AI to cross the bridges.

• Bad Question (Old Style): "When did the park close?" (Answer: Found in one sentence).
• iAgentBench Question: "What specific action by the city council, triggered by the resident's complaint, led to the park's closure?"
  • To answer this, the AI must find the complaint (Theme A), find the council vote (Theme B), and understand the link (the Bridge) between them. If it only reads Theme A, it fails.

Why This Matters

The paper tested several powerful AI models on this new benchmark. Here is what they found:

• Retrieval helps, but isn't enough: Giving the AI the search results (the library books) definitely helped it get better scores. But even with all the books, the AI still struggled to connect the dots.
• The "Sensemaking" Gap: The AI could find the facts, but it often failed to understand how the facts related to each other. It was like having all the puzzle pieces but not seeing the picture.
• Self-Reflection is tricky: Some AIs tried to "think again" (self-reflection) to fix their mistakes. Sometimes this helped, but sometimes it made them overthink and get the answer wrong.

The Takeaway

iAgentBench is a new, tougher gym for AI. It stops testing if the AI can just "look up" a fact and starts testing if the AI can "put the pieces together" to understand a complex, real-world situation.

It's the difference between an AI that is a dictionary (great at definitions) and an AI that is a journalist (great at connecting the dots to tell the truth). The authors hope this tool will help us build AI that doesn't just retrieve information, but actually understands it.

Technical Summary

Here is a detailed technical summary of the paper "iAgentBench: Benchmarking Sensemaking Capabilities of Information-Seeking Agents on High-Traffic Topics."

1. Problem Statement

Current Open-Domain Question Answering (ODQA) benchmarks, such as SQuAD or even multi-hop datasets like HotpotQA, often fail to adequately evaluate cross-source sensemaking.

• Limitations of Existing Benchmarks: Many rely on "passage-centric" answering where the correct answer can be extracted from a single relevant passage or by performing superficial multi-hop stitching (chaining a few facts). They do not effectively test an agent's ability to reconcile distributed evidence, track causal links, or integrate multiple themes to resolve complex dependencies.
• The Gap: As users increasingly rely on Information-Seeking Agents (ISAs) that browse and synthesize information from the open web, there is a need for benchmarks that reflect real-world, high-traffic topics and require query-conditioned sensemaking rather than simple fact retrieval.
• Contamination Risks: Static datasets are prone to data contamination (memorization) as LLM training corpora expand. There is a need for dynamic, time-indexed benchmarks that can be regenerated to ensure evaluation on fresh evidence.

2. Methodology: The iAgentBench Pipeline

iAgentBench is a dynamic benchmark construction pipeline designed to generate ODQA pairs that necessitate integrating evidence across multiple themes. The pipeline consists of four main stages:

A. Interest-Driven Seed Selection

• Source: Seed topics are derived from GDELT (Global Knowledge Graph), a real-world attention signal dataset, rather than curated trivia lists.
• Selection Criteria: Candidates are scored based on salience, geographic breadth, frequency, temporal specificity, and diversity. This ensures topics reflect what users are actively searching for at a specific time.

B. Graph Construction & Community Detection

• Retrieval: For each seed query, a query-conditioned corpus is retrieved from the open web (simulating the agent's test-time environment).
• Story Graph ($G(q)$): An LLM extracts entities and relational assertions (claims) from the retrieved documents to build a hypergraph.
• Community Clustering: The graph is partitioned into communities (themes) using Leiden clustering. Each community represents a coherent sub-story.
• Role Assignment: Communities are assigned roles based on a meta-graph analysis:
  • Core: Dominant themes with high influence.
  • Bridge: Themes that connect otherwise separate sub-stories (high betweenness centrality).
  • Satellite: Peripheral themes providing context.

C. Packet Construction

To generate questions without exposing the entire graph to the generator, the system constructs compact packets.

• A packet bundles a Core theme with one or more Bridge themes.
• It includes "Community Cards" (summaries and findings) and explicit Connector Relations (edges linking different communities).
• This structure forces the question generator to rely on cross-theme links rather than isolated facts.

D. QA Generation & Verification

• Generation: An LLM generates one-sentence, user-like questions based on the packet. The prompt enforces specific intent patterns: explainer, connection, trigger, consequence, and stake.
• Verification (LLM-as-a-Judge): A panel of three LLM judges verifies candidates against strict criteria:
  • Multi-community Necessity: The question must be unanswerable if any required community is removed.
  • Connector Necessity: The answer must depend on at least one explicit cross-theme connector.
  • Objectivity: Answers must be factual and single-valued, avoiding subjective interpretation.
  • Anti-Trivia: Questions cannot be solved by simple entity lookups or alias matching.

3. Key Contributions

1. Dynamic, Traffic-Driven Benchmark: Unlike static datasets, iAgentBench uses real-time attention signals (GDELT) and regenerates over time windows, reducing data contamination and ensuring relevance to current events.
2. Sensemaking-Focused Design: The benchmark explicitly targets cross-theme synthesis. Questions are structurally enforced to require integrating evidence from multiple distinct themes and their explicit connectors, moving beyond simple multi-hop retrieval.
3. Auditable Artifacts: Each instance is released with traceable evidence, including the story graph, community roles, connector relations, and judge decisions. This allows researchers to diagnose whether failures are due to retrieval (missing evidence) or synthesis (failure to integrate).
4. Intent Pattern Taxonomy: The framework categorizes questions into five specific information-seeking intents, enabling granular analysis of agent capabilities beyond aggregate accuracy.

4. Experimental Results

The authors evaluated four state-of-the-art LLMs (Claude, LLaMA, Mistral, Gemma) across three settings: Base (no tools), RAG (retrieval), and Reflexion (agentic self-reflection).

• Retrieval vs. Synthesis:
  • Retrieval significantly improved performance on all datasets (SimpleQA, HotpotQA, iAgentBench).
  • However, on iAgentBench, a significant accuracy gap remained even with RAG. This indicates that while agents can find the evidence, they struggle to integrate it across themes.
• Agentic Reflection:
  • The Reflexion (self-reflection) setting showed mixed results. While it helped some models (e.g., LLaMA) recover from errors, it caused performance regressions in others (e.g., Mistral, Gemma) on iAgentBench.
  • This suggests that multi-step reasoning is not universally beneficial and can introduce drift or over-correction when dealing with complex cross-theme dependencies.
• Comparison to Baselines:
  • SimpleQA: High gains from RAG suggest its difficulty is primarily evidence access.
  • HotpotQA: Moderate gains suggest a mix of access and composition.
  • iAgentBench: Retains a large gap even with RAG, confirming it successfully isolates the "sensemaking" bottleneck.

5. Significance

• Shift in Evaluation Paradigm: iAgentBench moves the field from evaluating "retrieval quality" to evaluating "evidence use." It highlights that access to information is insufficient for complex tasks; the ability to synthesize distributed information is the new frontier.
• Robustness against Contamination: By being dynamic and time-indexed, it offers a sustainable solution for benchmarking agents in an era of rapidly expanding LLM training data.
• Diagnostic Power: The release of intermediate artifacts (packets, connectors, judge logs) provides a unique capability to perform fine-grained error analysis, distinguishing between retrieval failures and synthesis failures.
• Real-World Applicability: By grounding questions in high-traffic, real-world topics and user intent patterns, the benchmark better reflects the actual challenges faced by information-seeking agents in production environments.

In conclusion, iAgentBench establishes that current LLMs, even with retrieval and self-reflection, struggle with cross-document sensemaking. It provides a necessary infrastructure to measure and improve the next generation of agents capable of reconciling complex, multi-source information landscapes.