💬 NLP

GeoRisk-RAG: A Hierarchy-Aware Risk Framework for Improving RAG Reliability through Selective Answering

GeoRisk-RAG is a novel hierarchy-aware framework that enhances the reliability of Retrieval-Augmented Generation in geospatial domains by utilizing a DAG-based distance metric to estimate geographic validity and enable selective answering, thereby significantly reducing false confidence rates for location-dependent questions compared to standard semantic similarity baselines.

Original authors: Meenu Ravi, Shailik Sarkar, Lulwah AlKulaib, Yordanos Tessema, Chang-Tien Lu

Published 2026-08-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Meenu Ravi, Shailik Sarkar, Lulwah AlKulaib, Yordanos Tessema, Chang-Tien Lu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

When a computer program tries to answer a question using a vast library of documents, it faces a subtle but dangerous trap. These programs, known as large language models, are excellent at finding words that sound similar to a question. If you ask about fire safety in a specific town, the computer might find a perfect article about fire safety in a neighboring town. To the machine, the words match so well that the answer seems correct. But in the real world, rules for one town do not always apply to the next. A regulation that bans certain trees in one city might be completely absent in the next. When a computer confidently gives an answer based on the wrong location, it creates a false sense of security that can be dangerous, especially when people are making decisions about natural disasters like wildfires.

Researchers at Virginia Tech and other institutions have developed a new way to fix this problem, called GeoRisk-RAG. Instead of just looking for words that match, this system checks the actual geography of the information it finds. It treats the world not as a flat list of places, but as a structured hierarchy, understanding that a city is part of a county, which is part of a state. By using a map-like structure built from a global database of facts, the system can measure exactly how far apart two places are in terms of their administrative boundaries. This allows it to know when an answer is a perfect fit, when it is a close but broader guess, and when it is simply the wrong place entirely.

The team tested this approach using a new set of questions and answers specifically about wildfires, a topic where location matters deeply. They created a dataset of 449 questions covering 19 different states and 28 cities, ranging from general safety tips to specific rules for particular towns. They compared their new system against standard methods that rely only on word matching. The results showed a dramatic improvement in safety. The standard methods often gave confident answers even when the information was from the wrong location, failing to warn the user about the geographic mismatch about 9% of the time. In contrast, the new system reduced this error rate to less than 1%.

What makes this system unique is how it handles uncertainty. When the computer finds information that is related but not exact—such as finding a rule for the whole state when the user asked about a specific county—it does not just guess or stay silent. Instead, it provides the answer but adds a clear warning that the information applies to a broader area. If the information is about a completely different place, the system refuses to answer, preventing the user from acting on misleading advice. This behavior was preferred by human evaluators, who found the warnings and selective answers more trustworthy than the confident but potentially wrong responses from older systems.

The study also showed that this method works well even when the question does not depend on a specific location. The system did not become slower or less accurate when answering general questions about fire seasons or equipment. It simply applied the same careful geographic check, ensuring that every response was grounded in the right context. The researchers found that by treating geography as a structured relationship rather than just a keyword, they could make artificial intelligence much more reliable for real-world decision-making. This approach suggests that for safety-critical topics, the most helpful answer is not always the one that sounds the most confident, but the one that knows exactly where it applies.

Technical Summary: GeoRisk-RAG

Problem Statement

Current Retrieval-Augmented Generation (RAG) frameworks rely heavily on semantic similarity and faithfulness metrics to ensure the reliability of Large Language Model (LLM) outputs. However, in geospatial domains—particularly natural hazard management—semantic similarity often fails to distinguish between semantic relevance and geographic validity. A passage may be topically relevant (e.g., discussing wildfire safety) but geographically inapplicable to the user's specific query location (e.g., regulations in Oregon vs. California).

In these high-stakes domains, a "confidently wrong" answer carries greater risk than abstaining. Existing methods struggle to differentiate between:

Exact matches: Where the retrieved context applies directly to the query location.
Hierarchical mismatches: Where context is valid for a broader scope (e.g., state-level rules applied to a city) or a narrower scope (e.g., city-specific rules applied to a state).
Geographic mismatches: Where the context refers to a different jurisdiction entirely, despite sharing a broad ancestor (e.g., two different counties in the same state).

Standard semantic search retrieves these inapplicable passages, leading to hallucinations or misleading guidance.

Methodology: GeoRisk-RAG

The authors propose GeoRisk-RAG, a hierarchy-aware framework that integrates geographic validity into the retrieval and generation pipeline through selective answering.

1. Knowledge Base Construction

The system utilizes a heterogeneous knowledge base comprising:

Event-centric reporting: ~2,000 open-access news articles (2023–2025) filtered for credibility.
Institutional guidance: 125 official documents (PDFs) from 30 U.S. states containing rules, regulations, and preparedness materials.
Chunking: Documents are split using a semantic segmentation strategy based on sentence embeddings (BAAI/bge-small-en-v1.5) and an adaptive threshold (95th percentile of cosine distance) to preserve topic continuity.

2. Geographic Entity Resolution

NER: The flair model is selected for Named Entity Recognition due to its optimal balance of F1-score (0.929) and inference latency.
Disambiguation: A multi-criteria prioritization scheme resolves toponym ambiguities (e.g., Springfield, IL vs. MA) using explicit identifiers, context co-occurrence, population data, and Wikidata sitelink counts.
Mapping: Non-administrative entities (e.g., tribal lands) and landscape features are mapped to their nearest administrative equivalents or country-level baselines within a graph structure.

3. Hierarchical Geographic Retrieval

The core innovation lies in modeling geography as a Directed Acyclic Graph (DAG) derived from Wikidata.

Structure: Entities are nodes, and containment relationships (e.g., "located in," "part of") are directed edges.
Distance Metric: The system calculates the distance ( $\delta$ $δ$ ) between a query location ( $q$ $q$ ) and a passage location ( $t$ $t$ ) based on their Lowest Common Ancestor (LCA) in the DAG.
- $\delta(q, t) = i_q + i_t$ , where $i$ represents the index of the LCA in the respective ancestor chains.
- If no shared ancestor exists, $\delta = \infty$ .
Ranking: Retrieval is framed as a rank optimization task. Passages are sorted primarily by minimizing geographic distance ( $\delta$ ) and secondarily by semantic cosine similarity.

4. Selective Answering Strategy

Based on the calculated geographic relationship, the system executes one of three behaviors:

Answer: If the context is an exact match to the query's geographic scope.
Warn: If the context is geographically related but broader (e.g., state rule for a city query) or more specific (e.g., city rule for a state query). The system provides the answer with a calibrated warning.
Abstain: If the context refers to a different place that only shares a broad ancestor (e.g., a different county in the same state) and lacks direct applicability.

Key Contributions

GeoRisk-RAG Framework: A novel RAG architecture that treats retrieval as a joint rank optimization task, balancing semantic relevance with hierarchical geographic alignment using a Wikidata-derived DAG.
Selective-Answering Strategy: A mechanism that restricts confident answers to exact geographic matches, issues warnings for granular mismatches, and abstains for unrelated jurisdictions, thereby reducing false confidence.
Wildfire QA Benchmark: The creation of a novel, held-out dataset comprising 449 QA pairs covering 19 states and 28 cities. The dataset is annotated for geographic granularity, topic, location dependency, and public information needs.
Comprehensive Evaluation: An evaluation protocol that moves beyond standard semantic similarity to include metrics for geographic validity, selective-answering behavior (warning/abstention rates), and user preference.

Experimental Results

The framework was evaluated against four baselines: Standard RAG, Standard RAG + Cross-Encoder Reranking, Granularity Match (scope-only), and Exact Keyword Match.

False Confidence Reduction: GeoRisk-RAG reduced the false confidence rate for location-dependent queries to 0.009, a significant improvement over the 0.090 observed in standard semantic similarity and reranking baselines.
Decision Quality: It achieved the highest Good Decision Quality Rate (GDR) of 0.946 for location-dependent queries and 0.974 for non-location-dependent queries.
Faithfulness & Correctness: The system maintained high faithfulness (0.965) and factual correctness (0.761), outperforming baselines which often hallucinated when retrieving geographically mismatched but semantically similar text.
Human Preference: In a blind evaluation with two annotators, GeoRisk-RAG responses were preferred in 70.0% and 68.75% of cases over the strongest baseline, citing better transparency and risk awareness.
Stability: Performance remained consistent across different embedding and generative model configurations, indicating the improvements are driven by the retrieval logic rather than specific model choices.

Significance and Claims

The paper claims that GeoRisk-RAG provides a more comprehensive assessment of end-to-end RAG pipelines by explicitly integrating geographic validity and selective-answering behavior.

Safety in Decision-Making: By distinguishing semantic similarity from geographic applicability, the framework prevents the dissemination of confidently wrong information in natural hazard management, where such errors can have severe consequences.
Explainability: The use of a DAG allows for explainable reasoning; the system can explicitly state why it is warning or abstaining (e.g., "This information applies at the state level, not the city level").
Domain Adaptability: While demonstrated on wildfire data, the authors posit that this hierarchy-aware approach is applicable to other geospatial domains where administrative boundaries and jurisdictional rules are critical.

The authors acknowledge limitations, noting the current dataset is U.S.-centric and modest in size, and that the framework's efficacy relies on the coverage and correctness of external knowledge graphs like Wikidata. Future work is suggested to expand to other countries, languages, and hazards, and to incorporate human-in-the-loop feedback for disambiguation.