MARCH: Evaluating the Intersection of Ambiguity Interpretation and Multi-hop Inference

1. Problem Statement

The paper addresses a critical gap in current Question Answering (QA) research: the intersection of multi-hop reasoning and ambiguity.

• The Challenge: Real-world queries often contain ambiguity (polysemy, syntactic ambiguity, or over-specific constraints) that can trigger multiple valid reasoning paths. In multi-hop QA, these ambiguities are often latent; a second-hop ambiguity (e.g., the word "pickup" meaning a truck vs. a guitar component) only becomes apparent if the first hop (e.g., "Mustang" meaning a car vs. a guitar) is not prematurely resolved.
• The Failure Mode: Current Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems tend to commit early to a single dominant interpretation (e.g., assuming "Mustang" is a car). This "premature pruning" causes the system to discard valid alternative reasoning paths, leading to cascading errors where subsequent retrieval steps are misdirected, resulting in incomplete or incorrect answers.
• The Gap: Existing benchmarks like MuSiQue (multi-hop) and ASQA (ambiguous) treat these challenges in isolation. There is no benchmark specifically designed to evaluate the compounded difficulty of resolving layered ambiguities across multiple reasoning hops.

2. Methodology

A. MARCH Benchmark Construction

The authors introduce MARCH (Multi-hop Ambiguity Reasoning CHain), a new benchmark containing 2,209 curated multi-hop ambiguous questions.

• Source: Derived from the MuSiQue dataset.
• Pipeline:
  1. Detection & Clarification: A multi-LLM consensus pipeline (using GPT-4.1, Llama-4-Maverick, Qwen3-235b, and Claude-Sonnet-4) detects ambiguity and generates specific "clarified questions" for each interpretation.
  2. Evidence Collection: Documents are retrieved using the clarified questions and decomposed atomic sub-questions to ensure comprehensive coverage.
  3. Answer Generation: Short answers are generated for each interpretation, and a synthesized "Long Answer" is created that integrates all valid interpretations.
  4. Filtering & Validation: A rigorous filtering process removes cases where interpretations collapse into identical answers. Human annotators validated a stratified sample, confirming high validity (>90%) and strong inter-annotator agreement (Fleiss' $\kappa$ up to 0.95).
• Ambiguity Taxonomy: The dataset categorizes ambiguity into three types:
  • Semantic: Entity collisions (e.g., "Mustang" = Ford vs. Fender).
  • Syntactic: Structural parsing differences (e.g., attachment of prepositional phrases).
  • Constraint: Over-specific modifiers that narrow the search scope unnecessarily.

B. CLARION Framework

To solve MARCH, the authors propose CLARION (CLarifying Ambiguity with a Reasoning and InstructiON), a two-stage agentic framework that explicitly decouples planning from acting.

• Stage 1: Planning Agent:
  • Analyzes the input query before any retrieval.
  • Detects ambiguity, classifies its type (Semantic, Syntactic, Constraint), and generates a set of clarified queries representing distinct reasoning paths.
  • This step prevents the model from locking into a single trajectory too early.
• Stage 2: Acting Agent:
  • Executes a ReAct-style loop (Thought $\to$ Action $\to$ Observation) for each clarified interpretation independently.
  • Maintains an interpretation set and retrieves evidence specific to each branch.
  • Includes a self-correction mechanism: If evidence is insufficient or inconsistent, the agent can re-invoke the planning stage to refine interpretations.
  • Synthesizes a final long-form answer that covers all valid interpretations.

3. Key Contributions

1. MARCH Benchmark: The first dataset specifically targeting the intersection of multi-hop inference and layered ambiguity, featuring 2,209 high-quality examples with human-validated annotations and multi-LLM consensus.
2. CLARION Framework: A novel agentic architecture that separates ambiguity resolution (planning) from evidence retrieval (acting), effectively preventing the "premature pruning" of latent reasoning paths.
3. Empirical Insights: The paper demonstrates that state-of-the-art models struggle significantly with MARCH, not because they lack reasoning capability in isolation, but because they fail to maintain multiple reasoning paths simultaneously when ambiguity is path-dependent.

4. Experimental Results

The authors evaluated CLARION against strong baselines (NaiveRAG, CoT, ReAct, DIVA) using three LLM backbones (Qwen3-235b, Gemini-2.5, DeepSeek-v3.1).

• Performance Gains: CLARION significantly outperformed all baselines across all metrics.
  • STR-EM (Strict Exact Match): CLARION achieved 38.73% (Qwen3) vs. 25.65% for the best baseline (CoT w/ RAG).
  • Disambig-F1: CLARION achieved 28.38% vs. 26.61% for the baseline. This metric is crucial as it measures the model's ability to cover all valid interpretations.
• Ablation Studies: Removing the "Clarification" module caused the largest performance drop, proving that explicit query rewriting is essential for handling multi-hop ambiguity.
• Human Performance: Human annotators achieved a STR-EM of 73.0% and Disambig-F1 of 62.0%, highlighting that MARCH remains a significant unsolved challenge for current AI systems.
• Generalization: While CLARION excels on MARCH, it also showed robustness on standard single-hop ambiguity (ASQA) and standard multi-hop (MuSiQue) tasks, suggesting its architecture benefits general reasoning.

5. Significance and Conclusion

• Diagnosis of Failure: The paper identifies that the primary failure mode in multi-hop QA is path-dependency. Early commitment to an interpretation fixes "bridge entities," steering the entire reasoning chain toward an irrelevant trajectory.
• Architectural Shift: The results suggest that future robust reasoning systems must explicitly model uncertainty and divergent paths before committing to evidence retrieval, rather than relying on linear, single-path reasoning.
• Future Direction: MARCH provides a rigorous testbed for developing systems capable of handling the complexity of real-world user queries, where ambiguity is the norm rather than the exception. CLARION offers a blueprint for agentic systems that can navigate these complexities through explicit planning and iterative verification.