Technical Summary: Citation-Closure Retrieval and Per-Rule Attribution for Real-World Regulatory Compliance Question Answering

Problem Definition

The deployment of Large Language Models (LLMs) for regulatory compliance faces a critical bottleneck: the requirement for rigorous traceability across multi-tiered authority structures. Unlike traditional multi-hop Question Answering (QA) which focuses on entity resolution, or standard legal QA centered on case-law reasoning, Regulatory Compliance QA demands structured procedural lookups and evidence-set closure.

Current Retrieval-Augmented Generation (RAG) systems fail in this domain due to three primary limitations:

1. Flattened Citation Edges: Graph-based RAG approaches often reduce explicit regulatory citations to generic entity-relation edges, stripping away the semantic distinctions necessary for navigating hierarchical delegations (e.g., from statutes to enforcement decrees to operational manuals).
2. Fragmented Retrieval: Query decomposition or expansion techniques often produce disjoint sub-queries that fail to unify the required evidence set, leading to incomplete procedural closures.
3. Fragile Post-Hoc Attribution: Generative models typically treat citations as free-form footers appended after generation. This "post-hoc" approach lacks structural binding between claims and sources, leading to systemic attribution hallucinations where models cite incorrect rules or fabricate legal provisions.

Methodology

To address these challenges, the authors introduce RegOps-Bench and the RefWalk framework.

1. RegOps-Bench: A Novel Benchmark

RegOps-Bench is an evaluation framework designed to test multi-tier procedural navigation. It is instantiated via a Korean national R&D regulatory corpus comprising 12 documents spanning five authority tiers (from Statutes down to Operational Manuals).

• Operational Knowledge Graph (OKG): The corpus is modeled into an OKG where nodes represent articles and edges are typed into six specific classes (e.g., DELEGATES_TO, SPECIFIES, REFERENCES). This deterministic graph construction avoids generative hallucinations inherent in LLM-based indexing.
• Axis-Decoupled Construction: The benchmark generates 250 QA pairs by decoupling Question Type (substantive intent) from Difficulty Level (structural complexity of the reference set). This allows for the independent control of procedural logic complexity (ranging from single-hop lookups to conditional multi-hop cross-document traversals) without confounding surface-level phrasing.
• Ground Truth: Answers are defined by a "typed citation closure," requiring the exhaustive retrieval of all governing articles and the precise mapping of claims to specific rule paragraphs.

2. RefWalk: The Proposed Framework

RefWalk is a unified RAG framework driven by a shared topic anchor that propagates through retrieval and generation.

• Topic-Anchored Multi-View Retrieval:

  • A frozen LLM extracts a structured Topic Anchor (topic, actor, temporal, magnitude, situational context) from the query.
  • Three distinct retrieval views are constructed from this anchor:
    • q_narrow: Uses the original question for dense semantic matching of explicit details.
    • q_wide: Drops the raw question, relying solely on abstracted topics and conditions to capture structural multi-hop references.
    • q_mid: A combination of both.
  • OKG Expansion: The initial candidate pool is expanded one hop along citation-bearing edges in the OKG.
  • RRM Fusion: Instead of standard sum-based aggregation (like RRF), RefWalk uses Reciprocal Rank MAX (RRM). This preserves the strongest signal from any single "specialist" view (e.g., q_wide for structural links) without diluting it, ensuring candidates highly ranked by a specific view retain priority.

• Per-Rule Attribution Schema:

  • To eliminate attribution hallucinations, RefWalk enforces a strict schema-level binding during generation.
  • The model is constrained to generate claims exclusively as array values bound to specific rule_id keys in a JSON schema.
  • This prevents the model from generating free-form text and retroactively appending citations. Instead, claims are structurally forced to be attributes of their governing rules, ensuring every generated claim is inherently bound to its source.

Key Contributions

1. Formalization of Regulatory Compliance QA: The paper formalizes the task as a problem of deterministic traversal of multi-tier regulatory hierarchies, distinct from standard multi-hop or legal QA.
2. RegOps-Bench: The release of the first benchmark specifically designed to evaluate multi-tier procedural navigation, featuring an Operational Knowledge Graph and axis-decoupled difficulty levels.
3. RefWalk Framework: A novel RAG architecture that unifies OKG traversal with per-rule attribution. It introduces RRM fusion to preserve specialist signals and schema-constrained generation to enforce traceability.
4. Empirical Validation: The framework demonstrates significant improvements over state-of-the-art baselines (including LightRAG, HippoRAG-2, and PIKE-RAG) in both retrieval recall and citation accuracy, particularly on complex cross-reference tasks.

Results

Experiments were conducted on RegOps-Bench and a derived HIPAA (U.S. health compliance) dataset.

• Retrieval Performance: RefWalk achieved state-of-the-art results on RegOps-Bench, with the most substantial gains in FullCov@10 (the fraction of queries where the entire ground-truth reference set is retrieved). While baseline retrieval methods saturated on the flat-structure HIPAA dataset, they collapsed on the complex cross-reference RegOps tasks, whereas RefWalk maintained robust performance.
• End-to-End Generation: RefWalk significantly outperformed baselines in Citation F1. The ablation study ("w/o schema") revealed that removing the per-rule attribution schema caused a sharp drop in citation precision, confirming that the schema constraint is the primary driver for mitigating attribution hallucinations.
• Difficulty Stratification: Performance analysis showed that while all models succeeded at single-anchor lookups (L1), baseline retrieval significantly failed at L3 (multi-hop/cross-doc) and L4 (conditional multi-hop) levels. RefWalk demonstrated significant relative improvements in Citation F1 across these complex tiers.
• Mechanism Analysis: The use of RRM fusion was shown to be critical; standard sum-based aggregations (RRF) diluted the structural signals required for complex cross-document references, leading to performance degradation in higher difficulty tiers.

Significance and Claims

The paper argues that regulatory compliance AI requires a paradigm shift from broad answer generation to verifiable, structure-bound attribution. The authors claim that:

• Existing RAG systems are insufficient for real-world regulatory ecosystems due to their inability to navigate complex, multi-tiered delegation networks and their reliance on post-hoc citation.
• The RefWalk framework successfully mitigates these vulnerabilities by enforcing structural binding between claims and sources, providing a robust foundation for deploying traceable AI in high-stakes domains.
• The RegOps-Bench is necessary to stress-test systems beyond flat-structure benchmarks, revealing saturation in current methods that would otherwise go undetected.

The authors acknowledge a precision-recall trade-off inherent in their conservative schema constraints, noting that while this ensures high citation precision (critical for compliance), it may sacrifice some recall compared to "eager-citing" baselines. They also note that the deterministic OKG construction relies on the formulaic citation patterns of the specific regulatory corpus used, presenting a challenge for scaling to less structured jurisdictions without hybrid extraction pipelines.