A Risk-Oriented Verification and Validation Framework for Hallucination Detection in Public-Sector LLM Systems

Technical Summary: A Risk-Oriented Verification and Validation Framework for Hallucination Detection in Public-Sector LLM Systems

Problem Statement
The paper addresses the critical challenge of "hallucination" (the generation of fluent but factually unsubstantiated content) in Large Language Models (LLMs) deployed within public-sector administrative and legal systems. Unlike consumer applications, public-sector outputs carry legal weight; erroneous procedural guidance, false legal citations, or incorrect authority designations can undermine legal certainty, administrative responsibility, and civic trust.

The author argues that existing literature primarily treats hallucination as a technical accuracy problem, relying on model-centric detection methods (e.g., self-consistency sampling, semantic entropy) or retrieval-augmented generation (RAG). These approaches are often ill-suited for public-sector deployment because they frequently assume access to internal model states, retraining capabilities, or specific inference parameters—conditions rarely met when using black-box LLM APIs. Furthermore, current methods focus on detection accuracy rather than the governance implications of detection, failing to translate risk signals into operational decisions (e.g., when to escalate to human officers).

Methodology
The paper proposes a Risk-Oriented Verification and Validation (V&V) Framework that reframes hallucination not as a model optimization issue, but as a manageable operational and governance risk. The methodology is model-agnostic, relying solely on observable outputs, external authoritative sources, and rule-based constraints, making it applicable to black-box environments.

The framework operates through four interconnected stages:

1. Slot-Based Administrative Decomposition:
   LLM responses are decomposed into five administratively meaningful "slots" corresponding to procedural elements:

   • Required documents
   • Competent authority
   • Processing timeline
   • Fees
   • Legal basis
     This decomposition allows for granular risk localization, recognizing that errors in high-impact slots (e.g., legal basis) are more consequential than errors in low-impact slots (e.g., supporting documents).

2. Slot-Level Risk Signal Detection:
   Three quantitative indicators are calculated for each slot:

   • Slot Instability Score (SIS): Measures cross-sample instability by generating multiple independent responses to the same query. High variance indicates epistemic uncertainty.
   • Citation Coverage Score (CCS): Evaluates whether the slot content is explicitly supported by valid, authoritative sources (e.g., statutory texts). A score of zero in high-risk slots is treated as a critical indicator.
   • Rule Violation Score (RVS): A binary check against predefined structural and jurisdictional constraints (e.g., verifying if a cited law exists or if an authority has jurisdiction).

3. Aggregation into Hallucination Risk Index (HRI):
   The slot-level signals are aggregated into a composite Hallucination Risk Index (HRI). This is a weighted sum where the weights ($w_i$) reflect the administrative impact of the specific slot. The formula incorporates tuning coefficients ($\alpha, \beta, \gamma$) to balance instability, lack of citation, and rule violations:
   $$HRI = \sum_{i} w_i \cdot (\alpha \cdot SIS_i + \beta \cdot (1 - CCS_i) + \gamma \cdot RVS_i)$$
   The HRI serves as an operational risk score rather than a probability of correctness.

4. Operational Decision Workflow:
   The HRI is mapped to predefined operational states based on thresholds:

   • PASS (HRI < 0.20): Direct delivery.
   • WARN (0.20 ≤ HRI < 0.35): Conditional response with clarification.
   • DEFER (0.35 ≤ HRI < 0.50): Withhold authoritative claims; isolate high-risk slots.
   • ROUTE (HRI ≥ 0.50): Mandatory escalation to human officers.
   • Override Rules: Specific conditions (e.g., RVS=1 for legal basis) trigger immediate escalation regardless of the aggregate score to prevent false assurance from statistical aggregation.

5. ISO/IEC-Style V&V Artifacts:
   The framework formalizes hallucination management into standard software assurance artifacts, including test cases with explicit objectives, preconditions, and quantitative acceptance criteria, ensuring auditability and traceability.

Key Contributions

• Governance Re-framing: Shifts the paradigm from "detecting hallucinations" to "managing hallucination risk," aligning LLM deployment with public-sector accountability and audit requirements.
• Black-Box Compatibility: Provides a viable verification mechanism for environments where models cannot be retrained or inspected internally.
• Granular Risk Quantification: Introduces the HRI, a composite metric that translates heterogeneous technical signals into a governance-oriented index interpretable by auditors and decision-makers.
• Proportional Human Oversight: Establishes a workflow where human intervention is triggered proportionally to risk levels, balancing operational efficiency with institutional safety.
• Standardization: Integrates hallucination management into existing ISO/IEC-style V&V processes, facilitating adoption within government procurement and quality assurance frameworks.

Results and Evaluation
The paper presents the framework through illustrative scenarios involving civil status registration, licensing, and social welfare inquiries. The evaluation demonstrates how the framework processes these scenarios:

• It successfully identifies high-risk outputs (e.g., fabricated legal citations or incorrect authorities) that might pass as "self-consistent" in traditional model-centric checks.
• It shows how the HRI differentiates between low-impact informational inconsistencies and high-impact procedural errors, triggering appropriate "DEFER" or "ROUTE" actions.
• The study validates that the framework can intercept dangerous outputs before they reach official channels, converting opaque model behavior into quantifiable, controllable operational risks.

Significance and Claims
The paper claims that its primary significance lies in offering a practical, governance-consistent approach to deploying LLMs in the public sector without undermining legal predictability. It argues that:

• Trust is built on control, not infallibility: Institutional trust in AI should not rely on claims of perfect accuracy, but on transparent, auditable procedures for identifying and managing failures.
• Risk differentiation is essential: Not all hallucinations are equal; the framework's slot-based approach ensures that governance resources are focused on the most consequential procedural elements.
• Integration is feasible: By mapping hallucination risks to standard software assurance practices, the framework removes institutional barriers to AI adoption, allowing public agencies to leverage LLMs while adhering to established principles of administrative law and accountability.

The author modestly notes that the study relies on demonstrative scenarios rather than massive quantitative benchmarking and that the framework is a complement to, not a replacement for, model-level mitigation strategies like RAG. Future work is suggested to include empirical threshold tuning and expansion to multimodal contexts.