L-MARS: Legal Multi-Agent Workflow with Orchestrated Reasoning and Agentic Search

1. Problem Statement

The paper addresses the critical limitation of Large Language Models (LLMs) in legal domains: hallucination of outdated or incorrect legal facts.

• The Challenge: Legal knowledge is dynamic (statutes change, new court rulings emerge). Standard LLMs rely on parametric knowledge (training data) which quickly becomes obsolete.
• The Failure of Existing Approaches:
  • Closed-book reasoning: Leads to confident but incorrect answers based on stale data.
  • Standard Retrieval-Augmented Generation (RAG): Often fails on reasoning-heavy benchmarks (like Bar Exam QA) and does not effectively handle the need for current factual updates in real-world legal scenarios.
  • Chain-of-Thought (CoT): The authors hypothesize that forcing an LLM to reason step-by-step without access to current data exacerbates errors, causing the model to construct plausible but factually wrong justifications ("confident confabulation").

2. Methodology: L-MARS Framework

The authors propose L-MARS, a multi-agent retrieval framework designed to ground legal answers in up-to-date external evidence.

Core Architecture

L-MARS is implemented using LangGraph, utilizing a directed graph where agents are nodes and control flow is conditional. It maintains a centralized WorkflowState tracking queries, search results, and iteration history.

Key Agents

1. Query Agent: Decomposes user questions into structured search intents.
2. Search Agent: Executes retrieval via three backends:
   • Online Search: Uses the Serper API (Google). It employs two modes:
     • Basic Search: Retrieves titles/snippets for quick exploration.
     • Enhanced Search: Fetches full content (HTML/PDF) and extracts context anchored to the search snippet to avoid token overflow.
   • Local RAG: Uses BM25 over user-provided local documents (Markdown).
   • CourtListener API: Retrieves authoritative case law, judicial opinions, and dockets.
3. Judge Agent: A critical component that performs evidence sufficiency checks. It analyzes retrieved content for:
   • Factual support and jurisdiction match.
   • Temporal specificity (is the info current?).
   • Contradictions.
   • Decision: It outputs SUFFICIENT or INSUFFICIENT and provides refinement notes if the loop must continue.
4. Summary Agent: Synthesizes the final answer with citations and rationale.

Operating Modes

• Simple Mode (Evaluated): A single-pass pipeline (Query $\to$ Search $\to$ Summary). Used to isolate the impact of retrieval.
• Multi-Turn Mode: An iterative loop where the Judge Agent guides the Search Agent to refine queries until evidence is sufficient.

3. Key Contributions

A. LegalSearchQA Benchmark

The authors introduce LegalSearchQA, a new benchmark specifically designed to test time-sensitive legal knowledge.

• Composition: 50 multiple-choice questions across 5 domains (Tax, Corporate, Labor, Immigration, Criminal).
• Design: Questions require knowledge from 2024–2026 (post-training data), covering specific dollar amounts, deadlines, and regulatory thresholds.
• Goal: To evaluate systems on their ability to retrieve current facts rather than rely on memorized training data.

B. Empirical Findings on Retrieval vs. Reasoning

The paper provides a nuanced view on when retrieval helps:

• Retrieval is essential for tasks requiring current factual knowledge (LegalSearchQA).
• Retrieval is negligible for tasks requiring abstract legal reasoning over static principles (Bar Exam QA).
• CoT is detrimental in fast-changing domains without retrieval, as it amplifies errors based on stale premises.

4. Experimental Results

Benchmark 1: LegalSearchQA (Time-Sensitive Facts)

• Zero-Shot (GPT-4o-mini): 58.0% accuracy.
• Chain-of-Thought (CoT): 30.0% accuracy (a significant drop). The model confidently reasoned using outdated tax years and regulations.
• L-MARS (Simple Mode): 96.0% accuracy.
  • Improvement: +38.0 percentage points over Zero-Shot.
  • Latency: 2.4 seconds per question.
  • Domain Breakdown: The largest gains were in Tax and Corporate Regulation (+61.5%), where specific 2025/2026 thresholds were required.

Benchmark 2: Bar Exam QA (Reasoning-Focused)

• Context: 594 questions testing legal reasoning on hypothetical fact patterns (static knowledge).
• Zero-Shot: 55.2% accuracy.
• L-MARS (Serper): 55.9% accuracy (+0.7 points).
• L-MARS (Tavily): 54.4% accuracy (-0.8 points).
• Conclusion: Retrieval adds noise rather than signal when the answer relies on internal reasoning capabilities rather than external facts.

Error Analysis

• L-MARS failed on only 2/50 questions in LegalSearchQA. Errors occurred when critical details were buried deep in long documents and missed by snippet-level retrieval.
• CoT degradation was attributed to "confident confabulation," where the model built logical arguments on false, outdated premises.

5. Significance and Implications

1. Paradigm Shift in Legal AI: The paper argues that for legal applications involving current events or regulations, agentic retrieval is superior to pure reasoning.
2. Warning on CoT: It highlights a critical risk: applying Chain-of-Thought prompting to domains with rapidly changing facts (without retrieval) can degrade performance below random baselines.
3. Benchmarking Necessity: It underscores the need for retrieval-focused benchmarks (like LegalSearchQA) rather than relying solely on static reasoning benchmarks (like Bar Exam QA) to evaluate legal AI systems.
4. Practical Deployment: The framework demonstrates that a structured, multi-agent workflow with a "Judge" for verification can achieve near-perfect accuracy on factual legal queries, offering a viable path for reliable legal assistance tools.

Code and Data Availability: The authors have released the code, prompts, and the LegalSearchQA dataset on GitHub and Hugging Face to ensure reproducibility.