💬 NLP

Causal Evidence Extraction and Triangulation in Crisis Reports using Large Language Models: A ReliefWeb-based Study

This paper presents a two-stage Large Language Model pipeline and a triangulation framework that effectively extracts and aggregates structured causal evidence from humanitarian reports, achieving high accuracy in identifying the positive impact of cash assistance on food-related outcomes.

Original authors: Yuanjun Zhang, Mourad Oussalah

Published 2026-08-06

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuanjun Zhang, Mourad Oussalah

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a massive mystery, but instead of a single crime scene, you have a library containing millions of messy, handwritten diaries. These diaries tell stories about disasters like floods, earthquakes, and droughts, and they describe how different groups tried to help. The problem is that the stories are long, confusing, and often contradict each other. One diary might say, "Giving people cash helped them buy food," while another says, "Cash caused prices to go up and made things worse."

To make sense of this chaos, scientists use a tool called a Large Language Model (LLM). Think of an LLM as a super-smart, tireless robot reader that can read thousands of these diaries in seconds. However, just like a human, the robot can get overwhelmed. If you ask it, "What happened with cash?" it might grab every single mention of the word "cash," even if the diary was just talking about a different topic entirely. This is called "over-extraction," and it creates a lot of noise. To fix this, researchers need a way to tell the robot to focus only on the specific clues that matter and to double-check its work against the original text to make sure it isn't making things up. This paper is about building a better detective system to turn those messy diaries into clear, trustworthy answers about what actually works in a crisis.

The Detective's New Toolkit

In this study, the researchers, Yuanjun Zhang and Mourad Oussalah, set out to clean up the "noise" in humanitarian reports. They used a massive collection of reports from ReliefWeb, a global database of crisis updates, covering the years 2000 to 2024. Their goal was to figure out if a specific type of help—cash assistance—actually leads to better outcomes, like people having enough food.

To do this, they built a two-stage "pipeline" using AI. Imagine this as a factory assembly line with two very specific workers:

The Filter (Stage 1): The first worker is very strict. Instead of reading the whole diary and guessing what's important, they are given a specific query, like "cash assistance." They only look for sentences where cash assistance is the main character. If the diary mentions cash but it's just a side note, the worker ignores it. This is called query-conditioned extraction. It stops the AI from grabbing irrelevant information, which was a big problem in previous methods.
The Fact-Checker (Stage 2): Once the first worker finds a potential clue (e.g., "Cash helped families buy food"), the second worker steps in. This worker doesn't just guess if the clue is good or bad; they look at the exact sentence (the "snippet") where the clue was found. They decide: Did the outcome go up (positive), go down (negative), or stay the same? And how strong is the evidence? Is it a weak rumor or a strong, clear statement? This is called snippet-grounded classification.

The Results: Finding the Signal in the Static

The team tested their new system against several powerful AI models, including Qwen-Plus, GPT-4o-mini, and DeepSeek-V3, as well as an open-source model called Llama-3.1-8B.

They found that their two-stage method was a game-changer.

The "Over-Extraction" Fix: Without their strict filter, the AI would pull out way too many irrelevant connections. With the filter, the number of extracted facts dropped to a realistic level, matching what human experts expected.
Accuracy: The best closed-source AI (Qwen-Plus) using their two-stage method achieved a weighted F1 score of 90.73%. In the world of AI, this is a very high score, meaning it was correct almost all the time.
The Open-Source Surprise: They also tried teaching a smaller, free AI model (Llama-3.1-8B) by showing it the answers from the big, expensive AI. This is called "distillation." The result? The smaller model got even better, reaching a weighted F1 score of 94.15%. This suggests you don't always need the most expensive robot to get the best results; you just need the right training.

The Grand Finale: Triangulation

Once they had thousands of clean, verified facts, they needed to put them together to see the big picture. They couldn't just count the total number of "good" and "bad" reports because some types of disasters (like floods) are reported on much more often than others (like earthquakes). If they just added them all up, the floods would drown out the other stories.

So, they invented a method called context-preserving triangulation.

The Grid: They organized the facts into a grid based on Disaster Type (e.g., Flood, Drought) and Source Type (e.g., Government, NGO, Media).
The Score: They calculated a Level-of-Evidence (LoE) score. This score tells you how much the different stories agree with each other. A score of 1.0 would mean everyone agrees perfectly; 0.0 would mean total chaos.

When they applied this to food-related outcomes for cash assistance, the result was striking. The LoE score was 0.865. This indicates a strong positive convergence. In plain English: across different types of disasters and different reporting sources, the evidence consistently suggests that cash assistance helps people get food.

They also looked at how this changed over time. From 2000 to 2005, the data was sparse and shaky. But from 2006 to 2017, the positive signal grew stronger and stronger, peaking in 2017 with an LoE of 0.929. Even from 2018 to 2024, the signal remained highly positive, though it softened slightly, likely because real-world situations are complex and sometimes cash needs to be handled carefully to avoid risks.

What This Means (and What It Doesn't)

The authors are careful to say that this isn't a magic wand that solves every crisis. They note that their study is limited to English reports and specific types of disasters. They also warn that the "strength" of the evidence they measured is based on what was written in the reports, not necessarily the actual size of the impact on the ground.

However, the study proves that by using a smart, two-step AI process and organizing the data carefully, we can turn a mountain of confusing, messy reports into a clear, auditable map of what works. It suggests that with the right tools, we can stop guessing and start knowing which interventions truly help people survive and thrive.

Technical Summary: Causal Evidence Extraction and Triangulation in Crisis Reports using Large Language Models

Problem Statement
Humanitarian crisis reports, such as those hosted on ReliefWeb, are characterized by their length, noise, and multi-topic nature. This structure creates a significant bottleneck for decision-makers attempting to consolidate evidence regarding what interventions work, where, and under which constraints. Traditional keyword retrieval often prioritizes recall, surfacing documents where an intervention is only marginally mentioned, which leads to retrieval-induced over-extraction when followed by unconstrained information extraction. Furthermore, existing evidence-synthesis frameworks are typically designed for topically clean inputs, making them ill-suited for the complex, narrative-driven nature of humanitarian situation reports. There is a critical need for a system that can extract structured, auditable intervention-outcome evidence from these long narratives while preserving the context necessary for cross-context synthesis.

Methodology
The authors propose a two-stage Large Language Model (LLM) pipeline coupled with a context-preserving triangulation method, applied to a corpus of ReliefWeb reports spanning 2000–2024.

Two-Stage Extraction Pipeline:
- Stage 1 (Query-Conditioned Entity & Evidence Extraction): The LLM is prompted to extract only interventions belonging to a specific query-defined class (e.g., "cash assistance"). It identifies associated outcome mentions and extracts supporting text snippets. This query-conditioning (QC) step restricts the output scope, mitigating the over-extraction of irrelevant relations common in low-precision retrieval scenarios.
- Stage 2 (Snippet-Grounded Relation Classification): Using the candidate intervention-outcome pairs and their supporting snippets from Stage 1, a second LLM call predicts the causal relationship labels: direction ( $\delta \in \{INC, DEC, NO\}$ ) and evidence strength ( $\sigma \in \{WEAK, MOD, STRONG\}$ ). This snippet-grounding (SG) ensures that every extracted relation is linked to explicit textual evidence, enhancing auditability.
Distillation for Lightweight Deployment:
To enable cost-effective deployment, the authors utilize the outputs of the best-performing closed-source LLM (Qwen-Plus) as "teacher" supervision. They apply Low-Rank Adaptation (LoRA) to fine-tune an open-weight model (Llama-3.1-8B-Instruct), training separate adapters for each stage of the pipeline while maintaining the two-stage inference structure.
Context-Preserving Triangulation:
Rather than pooling raw document counts, the method synthesizes evidence across specific contexts defined by disaster type and source type ( $disaster \times source$ cells).
- Polarity Normalization: Outcomes are normalized so that positive relations consistently denote improvement (e.g., flipping "decrease in food insecurity" to a positive direction).
- Cell-Based Aggregation: Evidence is aggregated within each cell using strength-weighted counts.
- Smoothing and Convergence: Laplace smoothing is applied to handle sparse cells. The method calculates global probabilities for positive, no-change, and negative relations by equally weighting each non-empty cell.
- Level-of-Evidence (LoE) Score: A metric is derived to quantify cross-context convergence, ranging from 0 (no convergence) to 1 (strong convergence).

Key Results
The study was evaluated on an expert-annotated dataset of 100 reports (220 relations) and a larger corpus of 8,029 reports.

Extraction Performance: Query-conditioning provided the most significant performance gains by reducing over-extraction. Among closed-source models, Qwen-Plus using the two-stage QC+SG pipeline achieved the highest weighted F1 score of 90.73%.
Distillation Success: The open-weight Llama-3.1-8B-Instruct, fine-tuned with LoRA using the closed-source pipeline's outputs as supervision, achieved a weighted F1 score of 94.15%, outperforming the closed-source teacher while offering greater cost-efficiency.
Triangulation Case Study: Applied to cash assistance and food-related outcomes, the triangulation method revealed strong positive convergence across disaster and source contexts, yielding a Level-of-Evidence score of 0.865. Temporal analysis showed that while early years (2000–2005) had volatile estimates due to sparse data, the evidence trajectory stabilized and remained highly positive from 2006 onwards, peaking in 2017 before showing mild softening in recent years.

Significance and Claims
The paper claims to address the gap in extracting causal evidence from long, noisy humanitarian narratives by introducing a framework that combines query-conditioned extraction to control noise and snippet grounding for auditability. A primary contribution is the context-preserving triangulation method, which aggregates evidence across $disaster \times source$ cells to measure convergence without allowing high-volume contexts to dominate the synthesis.

The authors position their system not as a definitive decision-maker but as an assistive tool to surface and summarize reported evidence. They emphasize that the outputs are designed for human review, particularly for contested claims or downstream decisions affecting affected populations. The study demonstrates that combining structured extraction with context-aware synthesis can make large-scale humanitarian reporting more amenable to evidence-based decision support.

Limitations
The authors acknowledge several constraints: the corpus is limited to English reports and a specific keyword query ("cash assistance"); the source taxonomy is coarse; and the expert-annotated evaluation set is small. They note that the "strength" labels reflect the strength of reported evidence rather than the magnitude of actual impact, and that the system may inherit biases present in the reporting ecosystem (e.g., uneven geographic coverage). Consequently, the results should be treated as decision support requiring human validation.

The Detective's New Toolkit

The Results: Finding the Signal in the Static

The Grand Finale: Triangulation

What This Means (and What It Doesn't)

Technical Summary: Causal Evidence Extraction and Triangulation in Crisis Reports using Large Language Models

More like this