A Consensus-Driven Multi-LLM Pipeline for Missing-Person Investigations

This paper introduces Guardian, a consensus-driven, multi-LLM pipeline enhanced by QLoRA fine-tuning that coordinates specialized models and a consensus engine to perform auditable, structured information extraction for critical missing-person investigations while avoiding unconstrained decision-making.

The Gist

Imagine you are trying to solve a mystery, like finding a lost child in a vast forest. The first 72 hours are the most critical, but the information you have is messy: a blurry photo, a vague witness statement, a weather report, and a map. You need to turn this chaos into a clear plan to save the day.

This paper introduces Guardian, a smart computer system designed to help police and rescue teams do exactly that. Instead of relying on just one super-smart computer brain, Guardian uses a "team of brains" approach to make sure the answers are right.

Here is how it works, broken down into simple concepts:

1. The Problem: One Brain Can Be Wrong

Usually, when we use AI, we ask one big model to read a messy police report and tell us where to look. But AI can sometimes "hallucinate" (make things up) or get confused by messy handwriting or bad grammar. In a missing-person case, if the AI guesses the wrong location, it wastes precious time and resources.

2. The Solution: The "Panel of Experts"

Guardian doesn't trust just one AI. Instead, it acts like a jury.

• The Witnesses (Multiple Models): The system asks several different AI models (like Qwen and Llama) to read the same report and give their best guess.
• The Judge (The Consensus Engine): A special "referee" AI looks at all the different answers. If two experts say the child was seen near a park, but one says "the beach," the referee checks the evidence. It doesn't just pick a random answer; it looks for agreement.
• The Safety Net: If an AI makes a mistake (like writing a messy list instead of a clean one), the system has a "repair crew" that fixes the formatting before anyone sees it.

3. The Process: A Factory Assembly Line

Think of the system as a highly organized factory:

• Stage 1: The Intake (The Parser): Raw, messy documents (PDFs, notes, texts) are dumped onto a conveyor belt. The system cleans them up, checks them for errors, and organizes them.
• Stage 2: The Assembly (The Core): This is where the "jury" works.
  • Summarizers write short, clear bullet points of what happened.
  • Extractors pull out specific facts (like "last seen at 5 PM" or "wearing a red hat").
  • Labelers guess the risks (e.g., "Is the child likely to run far?").
• The Consensus Check: Before the final report goes to the human detective, the "Judge" compares all the answers. If the models disagree, the Judge steps in to resolve the conflict, ensuring the final answer is based on facts, not guesses.

4. The "Training" (QLoRA)

To make the experts better, the system gives them a special, efficient training course called QLoRA.

• Analogy: Imagine you have a very smart student (the AI). Instead of rewriting their entire textbook (which takes forever and costs a lot of money), you just give them a few sticky notes with specific rules for this job.
• This makes the AI much better at reading police reports without needing a super-expensive computer to run it.

5. The "Zone QA" (The Map Check)

Once the system suggests a search area (a "zone"), a special safety module checks it.

• Analogy: It's like a quality control inspector on a map. If the AI suggests searching a huge, impossible-to-reach mountain range, the inspector says, "Wait, that's too big and risky. Let's narrow it down." It ensures the search plan is realistic and safe.

Why This Matters

The most important part of this paper is the philosophy: Reliability is a team sport.

In the past, we hoped one AI would be perfect. Guardian admits that AI makes mistakes. So, instead of hoping, it builds a system where mistakes are caught by the team before they reach the human investigators.

• No "Black Boxes": Every step is recorded. If the system makes a decision, you can look back and see exactly which AI said what and why the "Judge" picked that answer.
• Conservative & Safe: If the AI isn't sure, it admits it. It won't guess wildly. It prefers to say "we don't know for sure" rather than give a confident but wrong answer.

The Bottom Line

Guardian is a safety-first AI pipeline for finding missing people. It treats AI models like fallible experts, uses a "jury" to agree on the facts, fixes errors automatically, and ensures that the final search plan is built on solid, auditable evidence rather than a lucky guess. It turns messy, scary uncertainty into a clear, actionable plan for the people who need to save lives.

Technical Summary

Here is a detailed technical summary of the paper "A Consensus-Driven Multi-LLM Pipeline for Missing-Person Investigations."

1. Problem Statement

Missing-person investigations, particularly for children, are time-critical (the first 72 hours are vital) and rely on synthesizing heterogeneous, unstructured data (narrative reports, PDFs, tips, transit data). Traditional approaches suffer from:

• Data Sparsity and Noise: Early-stage information is often incomplete, ambiguous, or conflicting.
• Human Bottlenecks: Manual fusion of data is slow and prone to cognitive bias.
• LLM Limitations: While Large Language Models (LLMs) can extract information, they are prone to hallucinations, inconsistent formatting, and overconfident speculation when used as standalone decision-makers. In safety-critical domains, relying on a single model's "generative confidence" is risky.

The core challenge is to convert unstructured, noisy narratives into auditable, structured, and probabilistic search products (e.g., prioritized regions, likelihood surfaces) without introducing unverified assumptions.

2. Methodology: The Guardian LLM Pipeline

The authors propose Guardian, an end-to-end decision-support system organized into two stages, with a specific focus on the Guardian LLM Pipeline (Stage 2). The architecture treats reliability as a system-level property rather than a property of individual models.

A. Core Architecture Components

1. Multi-Model Candidate Generation:

   • The system does not rely on a single "authoritative" model. Instead, it routes tasks to multiple specialized LLMs (e.g., Qwen2.5-3B and Llama-3.2-3B).
   • Role Specialization: Models are fine-tuned via QLoRA (Quantized Low-Rank Adaptation) on curated datasets to perform specific tasks: Summarization, Entity Extraction, and Weak Labeling (movement/risk assessment).
   • Deployment: Models are deployed on separate GPU VMs with vLLM servers to ensure parallelism and isolation.

2. Centralized Consensus Layer (The "Referee"):

   • This is the core reliability mechanism. It does not merely pick the "best" answer but enforces invariants: schema conformity, factual supportability, and deterministic structure.
   • Workflow:
     • Normalization: Converts diverse model outputs into a comparable representation (e.g., enforcing fixed bullet structures for summaries, parsing JSON with type coercion for extractions, mapping free-text to closed label sets for weak labels).
     • Agreement Scoring: Computes agreement at the field level (e.g., comparing "last seen location" tokens) rather than surface text, using token overlap and similarity thresholds.
     • Conflict Resolution: If models disagree, a Consensus LLM (Gemini 2.5 Flash/Pro) acts as a referee. It is prompted to select between candidates or merge compatible ones without inventing new facts.
     • Repair: If JSON is malformed, the system attempts deterministic recovery or invokes a specific "repair prompt" to fix the structure before consensus.

3. Zone QA (Quality Assurance):

   • A module that applies LLM-assisted plausibility scoring to candidate search zones.
   • It reweights zone priorities using a transparent formula that combines original priority, LLM-derived plausibility, and zone size penalties.
   • It operates conservatively: if confidence is low, it defaults to neutral scores rather than extreme penalties, preventing the misdirection of search resources.

4. Prompt Governance:

   • Prompts are treated as "first-class system artifacts" with strict contracts (e.g., "return JSON only," "five bullets max").
   • Three Prompt Types: Task prompts (generation), Consensus prompts (adjudication/repair), and Format-guard prompts (enforcing schema).

B. Fine-Tuning Strategy

• QLoRA Integration: The system uses QLoRA to fine-tune base models on operational tasks. This updates <1% of parameters, preserving the base model's general language competence while specializing it for the specific domain.
• Interchangeability: Fine-tuned models are treated as interchangeable backends behind a unified interface, ensuring the consensus layer remains the arbiter of truth, not the specific model weights.

3. Key Contributions

• Consensus-Driven Reliability: A novel architecture where reliability is achieved through multi-model consensus, normalization, and structured conflict resolution, rather than trusting a single model's output.
• Auditable Decision Support: The system produces "trace artifacts" (logs of referee calls, agreement thresholds, and repairs), making the AI's decision process transparent and auditable for human investigators.
• Conservative LLM Usage: The paper advocates for using LLMs as structured extractors and labelers constrained by schemas, rather than unconstrained end-to-end decision-makers. This mitigates hallucination risks in high-stakes scenarios.
• Scalable Infrastructure: A distributed deployment strategy using dedicated GPU VMs and standardized APIs to handle concurrency and latency while maintaining strict timeouts for time-sensitive investigations.

4. Results and Evaluation

The evaluation was qualitative and diagnostic, focusing on reliability, structural validity, and failure modes rather than predictive accuracy (due to the lack of ground truth in real missing-person cases).

• Failure Mode Mitigation:
  • Single-Model Issues: Individual models exhibited overconfident classification, speculative inferences (e.g., inferring intent without evidence), and malformed JSON.
  • Consensus Effect: The consensus layer systematically reduced these errors. It suppressed unsupported fields, enforced closed vocabularies, and favored interpretations explicitly grounded in the narrative text.
• Structural Stability: The pipeline consistently produced parseable, schema-aligned outputs even when individual models failed or generated noisy data.
• Operational Behavior: The system successfully handled time constraints by returning "best validated candidates" if a full consensus round timed out, prioritizing timely, conservative outputs over delayed, potentially refined ones.

5. Significance and Implications

• Safety-Critical AI: The paper demonstrates a pragmatic framework for deploying LLMs in safety-adjacent domains (like child safety) where false confidence is more dangerous than uncertainty. By enforcing "conservative acceptance," the system minimizes the risk of misdirecting search resources.
• Weak Supervision Paradigm: It validates the use of LLMs as "controlled labelers" whose outputs are aggregated via consensus, aligning with research on weak supervision to create scalable training data without heavy manual annotation.
• Future of Investigative AI: Guardian shifts the paradigm from "predicting the location" to "producing calibrated uncertainty and actionable search products." It highlights that for complex, data-sparse problems, the system architecture (consensus, validation, audit trails) is more important than the raw accuracy of any single model.

Limitations

The authors acknowledge that the consensus approach increases computational costs (replicating tasks across models) and that semantic correctness remains difficult to validate without authoritative ground-truth datasets. Additionally, LLMs are not used for complex spatial reasoning; instead, they feed structured data into dedicated geospatial algorithms.