🤖 AI

Human-on-the-Loop Orchestration for AI-Assisted Legal Discovery

This paper addresses the risk of "trajectory collapse" in autonomous AI-driven legal discovery by proposing a structured failure taxonomy, a four-layer verification architecture, and a Human-on-the-Loop framework that significantly reduces privilege-waiver risks while minimizing the volume of documents requiring attorney review.

Original authors: Anushree Sinha, Srivaths Ranganathan, Abhishek Dharmaratnakar, Debanshu Das

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anushree Sinha, Srivaths Ranganathan, Abhishek Dharmaratnakar, Debanshu Das

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a very smart, fast, but slightly overconfident robot lawyer to sort through a mountain of 5,000 legal documents. Your goal is to find the "secret" documents (those protected by attorney-client privilege) and keep them safe, while sending the rest to the other side of a lawsuit.

The problem is that if this robot makes one small mistake early on, it doesn't just stop. It keeps going, building its entire logic on that first error. By the time it finishes, it might have accidentally handed over a secret document, thinking it was safe. The paper calls this "Trajectory Collapse." It's like a domino effect where one tiny wobble knocks down the whole tower, and because the robot speaks so confidently, you might not even notice the mistake until it's too late.

Here is how the authors propose to fix this using a "Human-on-the-Loop" system. Think of this not as a robot working alone, but as a robot working with a safety harness and a human supervisor who only steps in when the robot gets shaky.

The Four-Layer Safety Net

The authors built a four-step security system to catch the robot before it makes a mess:

The "Can You Do This?" Check (Planning):
Before the robot even starts, a smaller, simpler program asks: "Do you have the right tools and permissions to solve this specific task?" If the robot tries to plan something impossible (like looking at a sealed court record without a judge's permission), the system stops it immediately.
- Analogy: It's like a bouncer at a club checking your ID before you even try to walk through the door.
The "Are You Making Progress?" Check (Reasoning):
As the robot thinks through the problem step-by-step, the system constantly asks, "Is this step actually getting you closer to the answer, or are you just spinning your wheels?" If the robot starts making up facts or going in circles, the system hits the "reset" button and makes it try again.
- Analogy: It's like a GPS that realizes you've taken a wrong turn and immediately says, "Recalculating," before you drive 10 miles down a dead end.
The "Dry Run" Check (Execution):
Before the robot actually saves a file or sends an email, it performs a "dry run" in a safe, fake environment. It simulates the action to see if it breaks any legal rules (like accidentally sending a secret document to the wrong person). If the simulation fails, the action is blocked.
- Analogy: It's like a pilot running a flight simulator before taking off. If the plane crashes in the simulation, they don't take off for real.
The "I'm Not Sure" Check (Uncertainty):
Sometimes the robot just doesn't know the answer. The system measures how "confused" the robot is. If the robot is too unsure (high uncertainty), it doesn't guess; it raises its hand and says, "Human, I need help with this one."
- Analogy: It's like a student raising their hand in class when they don't know the answer, rather than guessing and getting the whole class in trouble.

What Happened in the Test?

The authors tested this system on a fake set of 5,000 documents. Here is what they found:

The Robot Alone: When the robot worked completely by itself, it made mistakes that led to a 8.3% risk of accidentally revealing secret documents.
The Robot with the Safety Net: When they added the "Human-on-the-Loop" system (specifically, asking the human lawyer to check only the documents the robot was unsure about), the risk of revealing secrets dropped by 61% (down to 3.2%).
The Efficiency: The best part? The human lawyer only had to look at about 24% of the documents. The robot handled the rest on its own.

The Bottom Line

The paper argues that in legal work, we can't just trust a robot to do everything perfectly because one early mistake can ruin the whole case. Instead, we need a system that checks the robot's work at every stage and only calls in a human lawyer when the robot is genuinely unsure or about to break a rule. This approach keeps the secrets safe while saving the lawyers a massive amount of time.

Important Note: This study was done on a synthetic (fake) set of documents created for the experiment. The authors explicitly state that they have not yet tested this on real-world legal cases with real law firms, so these results are a promising first step, not a final guarantee for real-life use.

Technical Summary: Human-on-the-Loop Orchestration for AI-Assisted Legal Discovery

Problem Statement

The deployment of autonomous Large Language Model (LLM) agents in electronic discovery (e-discovery) introduces a novel class of failure termed trajectory collapse. Unlike single-turn retrieval tasks, agentic workflows operate via multi-step Reason-Act-Observe (ReAct) loops where an agent's output at step $t$ conditions its query at step $t+1$ . In this context, a single early misclassification can silently propagate through the reasoning chain, rendering an entire privilege review invalid by the time a privilege log is generated.

Current Risk Assessment Challenges:

Compounding Errors: Errors are not isolated; they compound across the trajectory, embedding mistakes into hundreds of downstream decisions.
Fluency Trap: RLHF-optimized LLMs often generate fluent, confident justifications for incorrect legal classifications, masking errors.
Hallucinated Tool Calls: Agents may hallucinate tool invocations rather than abstaining when a task is unsolvable.
Metric Limitations: Standard metrics (Precision, Recall, F1) measure performance only at trajectory endpoints, failing to penalize failures that occur along the trajectory or the "Fluency Trap."

Methodology: Four-Layer Verification Architecture

The paper proposes a Human-on-the-Loop (HOTL) orchestration framework designed to intercept failures at specific functional stages before they compound. The architecture consists of four verification layers:

1. Planning Validation

Before execution, a lightweight classifier ( $f_{plan}$ ) verifies if the decomposed intent is satisfiable given the available toolset ( $T$ ).

Mechanism: A fine-tuned DeBERTa-v3 classifier estimates the solvability probability $P(S | q, T)$ .
Action: If $P(S | q, T) \geq \tau_{plan}$ (set to 0.70 in the pilot), the query executes; otherwise, it escalates to a human attorney.
Goal: Prevents "Fact Derive" errors (fabricated case law) and "Task Decompose" errors (sub-goals violating Federal Rules of Civil Procedure).

2. Stepwise Reasoning Checkpoints

At each reasoning step $t$ , the system computes an attribution score ( $A_t$ ) measuring expected progress toward the terminal goal.

Mechanism: $A_t = E_{\pi}[R(\tau) | s_t, a_t] - V(s_t)$ , where $V(s_t)$ is approximated via Monte Carlo rollouts.
Action: If $A_t < \epsilon$ (minimum progress threshold, set to 0.05), the reasoning trace is discarded, and the agent resamples from the current state.
Goal: Prevents the "Reasoning Trap" where rationalization overrides factual grounding.

3. Execution Sandboxing

Before committing write actions (e.g., tagging documents as privileged or routing to production databases), actions are executed in a dry-run environment.

Mechanism: A deny-list policy engine checks compliance with FRCP constraints (e.g., preventing "Attorney Eyes Only" tags from reaching non-counsel).
Action: If the action is non-compliant, it is blocked, and the system triggers HOTL escalation. For irreversible API calls, a compensating transaction pattern issues a reversal if the sandbox check fails post-execution.
Goal: Mitigates "Missing Tool" and "Fluency Trap" execution failures.

4. Uncertainty-Gated Escalation

The system decomposes total predictive uncertainty into aleatoric (data noise) and epistemic (model knowledge) components.

Mechanism: Since model weights are inaccessible, epistemic uncertainty ( $U_{ep}$ ) is estimated via self-consistency variance. The agent generates $k=10$ independent classification rationales per document; $U_{ep}$ is the entropy of the resulting label distribution.
Action: If $U_{ep} > \tau_{esc}$ , the system suspends autonomous action and surfaces the trajectory state to a human attorney.
Goal: Escalates only when the model lacks sufficient information to classify reliably.

Key Contributions

Taxonomy of Agentic Failures: A structured classification of failures in legal information retrieval organized by functional stage (Planning, Retrieval, Reasoning, Execution), linking specific failure modes (e.g., "Summarize Misalign") to regulatory consequences under the FRCP.
Four-Layer Verification Architecture: A novel system design that integrates planning validators, stepwise reasoning checkpoints, execution sandboxes, and uncertainty-gated escalation to intercept trajectory collapse.
Preliminary Simulation Study: A demonstration on a synthetic 5,000-document corpus showing that calibrated HOTL escalation thresholds significantly reduce privilege-waiver risk compared to fully autonomous baselines.

Experimental Results

The study compared three conditions on a synthetic corpus: Autonomous (A), Threshold-HOTL (T-HOTL) with varying thresholds ( $\tau \in \{0.3, 0.5, 0.7\}$ ), and Manual (M) review.

Privilege-Waiver Risk (PWR): T-HOTL at $\tau=0.5$ reduced PWR from 8.3% (Autonomous) to 3.2%, representing a 61% reduction in risk.
Efficiency: At $\tau=0.5$ , only 23.7% of documents were routed to attorney review, compared to 100% for manual review. This suggests an estimated 76% reduction in attorney hours while maintaining comparable risk levels.
Error Recovery: The Rollback Recovery Rate (RRR) averaged 0.71, indicating that reasoning checkpoints successfully intercepted and recovered the majority of trajectories without requiring human escalation.
Error Origin: The Mean First-Error Position (FEP) for the Autonomous condition was 3.2 steps (out of an average trajectory length of 7.8), confirming that errors tend to originate early and compound.
Latency: End-to-end processing averaged 14.2 seconds per document with parallelized batched requests; sequential execution would require 290–340 seconds.

Significance and Claims

The paper argues that transitioning to agentic legal discovery requires trajectory integrity to be established as a first-class design objective. Conventional endpoint metrics are insufficient for capturing compounding failures. The authors claim that:

Calibrated HOTL escalation is a prerequisite for deploying production-grade agentic IR systems in the legal domain.
It is possible to significantly reduce the risk of privilege waiver (up to 61%) while routing fewer than one-quarter of documents to human review.
The proposed architecture provides a mechanism to satisfy FRCP auditability requirements by logging deterministic provenance hashes for every interaction unit.

Limitations and Future Work

The authors explicitly note that the results are preliminary and limited by the synthetic nature of the corpus, which may not capture the distributional complexity of production data (e.g., domain jargon, encoded attachments).

Ground Truth: Labels lack formal inter-annotator agreement; future work will report Cohen's $\kappa$ .
Ablation: The study did not ablate individual layers; future work with law-firm partners will isolate each layer's contribution.
Data Governance: Production deployment of GPT-4o for privileged documents requires on-premises or zero-egress inference.
Validation: Future work will validate the system on real discovery corpora and explore formal guarantees using backbones like TAPE and KAIJU.