Technical Summary: LocalSearchBench

Problem Statement

While recent advances in Large Reasoning Models (LRMs) have enabled agentic search systems to perform complex multi-step reasoning across general information sources, there is a significant gap in applying these capabilities to vertical domains, specifically local life services. Existing benchmarks primarily focus on general information retrieval or closed-ended QA, failing to address the unique challenges of local services, such as:

• Geographic constraints: Queries often require strict adherence to location-based logic (e.g., "nearby," "within the same landmark").
• Multi-service integration: Real-world user needs often involve chaining disparate services (e.g., dining, accommodation, entertainment) with spatiotemporal dependencies.
• Ambiguity and complexity: User queries are frequently ambiguous and require multi-hop reasoning across merchants, products, and events to satisfy composite constraints.

The lack of a comprehensive benchmark tailored to these scenarios has constrained research into domain-specific agentic search, leaving the performance of state-of-the-art models on real-world local life tasks largely unexplored.

Methodology

The authors propose a two-pronged methodology: the construction of a comprehensive benchmark dataset and the development of an evaluation framework.

1. LocalSearchBench Construction

LocalSearchBench is a comprehensive benchmark designed for agentic search in local life services, comprising three core components:

• Local Merchant Database:

  • Scale: Constructed from over 1.6 million raw merchant records sampled from a leading local life service platform (Platform M) across 9 major Chinese cities.
  • Coverage: Covers 6 service categories (Dining, Lifestyle, Shopping, Accommodation, Healthcare, Tourism) and 10,741 landmarks.
  • Processing: The raw data (12 fields) undergoes a three-stage pipeline:
    1. Data Augmentation: An agent expands records to 29 fields, enriching basic, locational, service, and product information.
    2. Privacy Rewriting: A privacy agent anonymizes 7 sensitive fields (identity, location, contact) while preserving business logic.
    3. Data Validation: A validation agent (LLM-as-judge) assesses quality across dimensions like completeness, logical consistency, and geographic accuracy.
  • Final Output: A high-quality, anonymized database of 1,354,185 merchant entries.

• LocalRAG (Retrieval-Augmented Generation):

  • A retrieval system built on the merchant database to support efficient search.
  • Architecture: Uses an embedding model to convert merchant profiles into vectors, indexed in a city-partitioned vector database. It employs a two-stage retrieval process: semantic retrieval (top-N) followed by a reranking model (top-K) to ensure high relevance.
  • Performance: Achieves NDCG@10 of 0.84 and MRR@10 of 0.82 on hop-level queries.

• Multi-hop QA Benchmark:

  • Task Generation: 900 multi-hop QA tasks were synthesized from 1,200 seed questions. These tasks are categorized into intelligence levels L3 (Intelligent) and L4 (Expert), requiring multi-step reasoning (3 to 5 hops).
  • Construction Process: Involves question instantiation with city-specific data, answer collection using LocalRAG and web search, and rigorous validation.
  • Validation: A two-stage process ensures every hop has a clear answer, adheres to business logic, and maintains factual accuracy. Three domain experts review the final QA pairs.

2. LocalPlayground Framework

To evaluate agentic search capabilities, the authors developed LocalPlayground, a unified environment integrating:

• Search Agent: An agent that iteratively performs reasoning and search using two tools: LocalRAG (for merchant retrieval) and Web Search (for real-time facts, events, weather). It follows a ReAct structure, recording a trajectory of states (sub-questions, evidence) and actions (reasoning, tool calls).
• Validation Agent: An LLM-as-judge (Claude Opus 4.5) that evaluates both Answer Quality (Correctness, Completeness, Fluency, Faithfulness, Safety) and Trajectory Quality (Action Relevance, Evidence Sufficiency, Causal Coherence, Search Efficiency).

Key Contributions

1. First Comprehensive Benchmark: LocalSearchBench is the first offline, high-quality, real-world agentic search benchmark specifically for local life services, containing over 1.3M merchant entries and 900 multi-hop QA tasks.
2. Systematic Evaluation Framework: The introduction of LocalPlayground, which provides a standardized environment for evaluating LLM agents' ability to coordinate multiple tools (LocalRAG and Web Search) in complex, constraint-heavy scenarios.
3. Empirical Insights: Extensive experiments across 16 state-of-the-art models (including GPT-4.1, o3, Gemini, DeepSeek-V3.2, and Qwen3 series) reveal significant performance gaps and specific failure modes in current agentic search systems.

Results

Model Performance

Experiments on 16 models demonstrate that even the most advanced LRMs struggle with local life service tasks:

• Correctness: The best-performing model, DeepSeek-V3.2 (with thinking), achieved only 35.60% correctness. The second-best, GLM-4.6, reached 32.83%.
• Completeness vs. Faithfulness: While models showed moderate completeness (average 60.32%), faithfulness (adherence to retrieved evidence) was notably low (average 30.72%), indicating a tendency toward hallucination or ignoring constraints.
• Thinking vs. Non-Thinking: "Thinking" models generally outperformed non-thinking counterparts in answer quality metrics (e.g., Correctness: 30.49% vs. 21.30%) but showed similar search efficiency.
• Task Difficulty: Performance dropped significantly on L4 (Expert) tasks (25.10% correctness) compared to L3 (Intelligent) tasks (30.57%).

Ablation and Sensitivity Studies

• Tool Integration: Adding tools significantly improved performance. Models without tools achieved <10% correctness. Adding LocalRAG improved correctness to ~23-27%, and adding Web Search further boosted it to ~32-35%. However, web search introduced a trade-off, slightly reducing faithfulness due to noise.
• Max Rounds (N): The optimal number of conversation rounds was found to be N=5. Increasing beyond this (N>5) led to diminishing returns and decreased correctness due to information overload and noise.
• Retrieval Parameters: The optimal LocalRAG configuration was top-N=100 and top-K=20. Higher values (e.g., 150/30) degraded performance, suggesting information overload.

Failure Analysis

Analysis of 580 failure trajectories identified four primary causes of failure:

1. Unstable Tool Call Strategy (38.6%): Models failed to time tool calls correctly, violated rules (e.g., using web search for merchant queries), or made redundant calls.
2. Missing Explicit Multi-hop Planning (30.9%): Models failed to decompose tasks into ordered sub-goals, leading to unordered retrieval and cascading failures.
3. Query Generation Dilution (11.7%): Generated queries often omitted critical constraints, weakening retrieval signals.
4. Long-Context Noise (18.8%): Excessive retrieved information overwhelmed the model, reducing its ability to distinguish relevant evidence.

Significance

The paper claims that LocalSearchBench addresses a critical gap in the evaluation of agentic search systems. By providing a realistic, large-scale benchmark for local life services, it highlights that current state-of-the-art models are not yet robust enough for complex, real-world multi-step reasoning tasks involving geographic and service constraints. The authors emphasize that the low correctness rates (max 35.60%) underscore the urgent need for specialized benchmarks and domain-specific agent training. The open-sourcing of LocalSearchBench and LocalPlayground is intended to facilitate future research into improving agentic search capabilities in vertical domains.