🤖 AI

VeriTrip: A Verifiable Benchmark for Travel Planning Agents over Unstructured Web Corpora

This paper introduces VeriTrip, a verifiable benchmark that evaluates travel planning agents on unstructured multimodal web corpora by establishing a synchronized knowledge base to quantify factual reliability and revealing a critical trade-off between autonomous retrieval cognitive load and instruction retention.

Original authors: Yuting Xu, Jiayi Tian, Jian Liang, Xin Xiong, Hang Zhang, Mu Xu, Xiao-Yu Zhang

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuting Xu, Jiayi Tian, Jian Liang, Xin Xiong, Hang Zhang, Mu Xu, Xiao-Yu Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a travel agent to plan a perfect vacation. In the past, you gave this agent a neat, pre-printed menu of options (like a flight schedule or a hotel list) and asked them to pick the best ones. They just had to be good at math and logic to make a good plan.

But in the real world, there is no neat menu. There is only the chaotic, messy internet: thousands of travel blogs, blurry photos from social media, conflicting reviews, and outdated websites.

VeriTrip is a new "test drive" for AI travel agents designed to see if they can handle this real-world mess. Here is how the paper explains it, using simple analogies:

1. The Problem: The "Clean Room" vs. The "Jungle"

Most previous tests for AI agents were like putting them in a clean, white room where all the facts were written on the wall in perfect handwriting. The AI just had to read the wall and write a plan.

The authors say this doesn't test if the AI is actually smart. Real life is a jungle.

The Noise: The internet is full of "noise"—fake ads, confusing typos, and people arguing about whether a restaurant is good or bad.
The Visual Puzzle: Sometimes a user sends a blurry, cropped photo of a landmark (like a statue) and says, "I want to go here." The AI has to figure out exactly which statue it is without a name tag.
The Hallucination Trap: If the AI can't find the answer, it might just make one up because it "remembers" something from its training data. In travel, making up a flight number is a disaster.

2. The Solution: The "VeriTrip" Test

The researchers built a special sandbox called VeriTrip to test these agents. Think of it as a frozen snapshot of the internet.

The Library (MRB): They collected over 8,000 documents and 4,000 images from real travel sites. This is the "library" the AI is allowed to search. It's messy and unorganized, just like the real web.
The Answer Key (VKB): Hidden away from the AI is a "Gold Standard" answer key. This contains the true facts extracted from that messy library.
The Test: The AI gets a user request (e.g., "Plan a trip to Changsha for 3 days, budget $500, and here is a blurry photo of a bronze artifact"). The AI must:
1. Look at the blurry photo and figure out what it is.
2. Search the messy library to find the right facts.
3. Build a plan.
4. The Catch: The researchers check every single detail (flight numbers, hotel names, prices) against the hidden Answer Key. If the AI made up a fact, it gets a zero for that part.

3. What They Found: The "Brain Overload" Surprise

The researchers tested the smartest AI models available (like GPT-4o, Claude, and Gemini). Here is what happened:

The "Lazy" vs. "Hard" Work: On easy tasks, the AIs were "lazy." They didn't search enough and just guessed based on what they remembered. This led to mistakes. On hard tasks, they were forced to search more, which actually made them more accurate.
The Trade-Off (The "Juggling Act"): This is the most interesting finding.
- When the AI had to use its "eyes" to solve the blurry photo puzzle, it got better at finding the facts (it didn't make up fake names).
- BUT, because solving the photo puzzle used up so much of its "brain power," it forgot the user's other preferences. It might find the right hotel, but forget that the user wanted a vegetarian meal or a specific budget.
- Analogy: It's like a student who is so focused on solving a difficult math problem that they forget to write their name on the test paper. They got the math right, but they failed the whole assignment.

4. The Conclusion: "Seeing" is Not "Planning"

The paper concludes that current AI agents are good at two separate things, but bad at combining them:

Seeing: They can look at a picture and find the right place.
Planning: They can organize a schedule.

But when they have to do both at the same time in a messy environment, they struggle. They often get the facts right but the plan wrong, or they get the plan right but the facts made up.

In short: VeriTrip shows us that to build a truly reliable AI travel agent, we need to teach it how to juggle finding facts in a messy world without dropping the user's specific wishes. Right now, the AI is dropping the ball.

Technical Summary: VeriTrip: A Verifiable Benchmark for Travel Planning Agents over Unstructured Web Corpora

1. Problem Statement

Current benchmarks for autonomous travel planning agents suffer from a fundamental disconnect between evaluation paradigms and real-world deployment. Existing benchmarks (e.g., TravelPlanner, TripTailor) typically operate within sanitized API environments where facts are provided via structured interfaces. This approach abstracts away the core cognitive challenges of the open web: information noise, multi-source factual contradictions, and the necessity of grounding visual perception into logical planning.

Conversely, deep research benchmarks often lack rigorous verification protocols for synthesized outputs, relying on "LLM-as-a-Judge" grading which fails to detect subtle factual fabrications. The central research question addressed by this work is: Can agents produce factually correct plans when they must actively discover facts themselves from unstructured, multimodal evidence?

2. Methodology

2.1 The VeriTrip Environment

To bridge the gap between API-based planning and open-web retrieval, the authors introduce VeriTrip, a benchmark built on a Multimodal Retrieval Base (MRB) and a Verifiable Knowledge Base (VKB).

Multimodal Retrieval Base (MRB): A static, frozen snapshot of the web containing 8,210 documents and 4,146 images curated from real-world sources (tourism portals, forums, booking pages). The MRB is designed to preserve "authentic web noise," including outdated opinions, conflicting reviews, and promotional spam, while filtering out PII and stealthy ads using an LLM-based semantic filter (Qwen3-Max).
Task Formulation: Agents receive a query $Q = (q_{txt}, q_{img})$ . The text component specifies constraints (budget, duration, preferences), while the image component provides a visual anchor (e.g., a heavily cropped social media photo of a destination).
Agent Interface: Agents interact with the MRB via a constrained toolset:
- docSearch: Keyword-based text retrieval.
- imgSearch: Visual similarity search to resolve ambiguous anchors.
- getDocument: Fetching full text content.
- Specialized functions for structured restaurant metadata (to ensure evaluation completeness, as raw web data for dining is often fragmented).
Verifiable Knowledge Base (VKB): A structured gold standard extracted from the MRB, strictly inaccessible to agents. It contains ground-truth facts (transportation IDs, POI names, opening hours) used for automated verification.

2.2 Evaluation Protocol

VeriTrip moves beyond surface-form matching to a cell-wise factual verification protocol. The evaluation pipeline includes:

Delivery Rate (DR): Measures if the output is a valid, parsable JSON.
Factual Reliability (FR): The percentage of generated details (e.g., flight numbers, hotel names) that exactly match the VKB. This distinguishes between systematic reasoning failures and parametric hallucinations.
Preference Fulfillment Rate (PFR) & Pass Rate (PR): Evaluates adherence to soft constraints (dining tastes) and hard constraints (budget, chronological continuity).
Geographic Coherence: Uses a Traveling Salesperson Problem (TSP) solver to calculate the Average Margin (AM), measuring the excess travel distance compared to an optimal route to assess spatial common sense.

2.3 Experimental Setup

The benchmark was evaluated on a diverse set of state-of-the-art Multimodal Large Language Models (MLLMs), including:

Without Thinking: GPT-4o, GPT-5-mini, Claude-4.5-Sonnet, Gemini-2.5-flash, Qwen-VL-Max, etc.
With Thinking: o3, o4-mini, Gemini-2.5-pro.
The models were tested on tasks of varying difficulty (Simple, Medium, Complex) involving 15 tourist cities in China and the US.

3. Key Contributions

Unstructured Multimodal Benchmark: VeriTrip is the first benchmark to evaluate travel planning over a frozen, unstructured, multimodal web corpus rather than isolated APIs. It forces agents to resolve visual ambiguities and navigate noisy, conflicting evidence.
Cell-Wise Verification Protocol: By constructing a VKB derived directly from the MRB, the authors enable precise, automated quantification of factual reliability. This decouples genuine logical reasoning from internal parametric memory.
Discovery of Cognitive Trade-offs: The evaluation reveals a critical retrieval-reasoning trade-off. The cognitive load required for autonomous retrieval and visual grounding significantly erodes instruction retention and high-level preference fulfillment.
Visual Grounding Analysis: The paper demonstrates that while visual tools (Timg) improve factual grounding by forcing retrieval (reducing hallucinations), they simultaneously degrade planning quality (PFR) by exhausting the model's reasoning capacity.

4. Key Results

Systemic Failure in Unconstrained Environments: All tested MLLMs struggle with unconstrained retrieval-based planning. While Delivery Rates (DR) remain relatively high, Factual Reliability (FR) is consistently low (often below 50% for many models on complex tasks).
The "Laziness" Phenomenon: Models exhibit "laziness" on simple tasks, relying on parametric memory with fewer tool calls, which leads to lower FR. Forcing agents into complex tasks increases tool usage and improves FR (e.g., o4-mini improved FR by 8.74% on complex tasks), confirming that active retrieval reduces hallucinations.
The Retrieval-Reasoning Trade-off:
- Factual Reliability vs. Preference Fulfillment: As task complexity increases, FR often improves (due to forced retrieval), but Preference Fulfillment Rate (PFR) collapses. For example, GPT-4o-mini's PFR dropped by 21.45% on complex tasks.
- Visual Grounding Impact: Ablation studies show that providing visual search tools improves FR (by resolving ambiguity) but significantly lowers PFR. The cognitive load of processing visual inputs consumes the reasoning resources needed to satisfy global constraints.
Noise Sensitivity: The presence of conflicting or misleading information in the MRB generally degrades performance, particularly in Delivery Rate and Pass Rate, though some models show resilience in geographic coherence.
Failure Modes: Case studies reveal that agents often fail at fine-grained entity recognition (linking a specific visual anchor to a POI) and exhibit contextual hallucination, where strong textual priors override visual evidence (e.g., forcing a Los Angeles landmark into a Boston itinerary).

5. Significance and Claims

The authors position VeriTrip as a necessary evolution in agent evaluation, shifting the focus from "tool execution" to "evidence-grounded reasoning."

Bridging the Gap: The work highlights a significant capability gap between "seeing" (visual perception) and "planning" (logical synthesis). Current MLLMs lack the fine-grained visual discriminative capabilities to bridge this gap without external retrieval.
Robustness over Memorization: The benchmark demonstrates that high performance in structured API environments does not translate to the open web. True robustness requires the ability to verify facts against noisy, unstructured sources.
Limitations and Boundaries: The authors modestly claim that VeriTrip evaluates static evidence retrieval within a frozen sandbox. It does not measure real-time dynamic booking capabilities (e.g., live ticket purchases or handling sold-out scenarios). The scope is limited to 15 cities in China and the US, and the evaluation is bounded by the "evidence-sufficient" nature of the sandbox.

In conclusion, VeriTrip provides a rigorous foundation for developing the next generation of planning agents, emphasizing that future progress depends on resolving the tension between the cognitive demands of autonomous retrieval and the fidelity of logical planning.