Large Language Models for Causal Relations Extraction in Social Media: A Validation Framework for Disaster Intelligence

Technical Summary: Large Language Models for Causal Relations Extraction in Social Media

Problem Statement
Disaster intelligence relies heavily on situational awareness, which is often hindered by the retrospective and labor-intensive nature of official post-event reports (e.g., from NOAA). While social media offers real-time, crowd-sourced data regarding hazards, impacts, and infrastructure disruption, extracting valid causal relations from these posts is inherently difficult. Social media content is typically informal, fragmented, and context-dependent, with causality often implied rather than explicitly stated. Traditional rule-based methods fail to capture these implicit relations, and existing NLP approaches often depend on rigid, human-annotated templates that do not generalize well to rapidly evolving disaster contexts. Although Large Language Models (LLMs) offer the potential to reason over noisy text and infer unstated relations, there is a critical risk that they may generate plausible causal links based on internal prior knowledge rather than evidence present in the specific posts. The central research question is whether LLMs can effectively extract valid causal relations from disaster-related social media posts.

Methodology
To address this, the authors propose a validation framework that compares LLM-generated causal graphs against expert-grounded reference graphs. The methodology consists of three primary components:

1. Construction of Ground-Truth Causal Graphs:
   Since pre-existing causal graphs for specific disasters are unavailable, the authors construct them using a "match-and-record" approach. They begin with a general impact-chain framework (representing causal pathways among hazards, exposure, vulnerability, and impacts) and refine it using causality evidence from standardized, authoritative post-event reports (specifically NOAA reports). Only edges explicitly supported by textual evidence in these reports are retained, creating a set of directed (Cause, Effect) relations that serve as the ground truth.

2. Causal Relation Extraction via LLMs:
   The study evaluates three LLMs with varying architectures and data access capabilities:

   • Grok4.3: A closed-source model with native access to social media data for historical and real-time posts.
   • GPT-5.5: A closed-source model with long-context capabilities but no native social media access.
   • Mistral-7B: An open-weight, smaller model with limited context windows.

   The models are prompted to identify causal mentions within batches of social media posts, map them to a fixed set of canonical variables defined by the impact-chain framework, and output directed causal pairs. For models with limited context windows (Mistral-7B), a chunking strategy is employed where posts are processed in batches of 20.

3. Evaluation Framework:
   The performance of the generated graphs is measured against the expert-grounded reference graphs using standard causal discovery metrics: Node and Edge Precision, Recall, F1 score, Structural Hamming Distance (SHD), and normalized SHD (nSHD). Crucially, the study includes a controlled ablation experiment where models are fed "non-informative" posts (lacking disaster-specific content) with external search disabled. This tests whether the models rely on the provided data or fall back on internal priors to generate causal structures.

Key Contributions

• Expert-Grounded Causal Graphs: The authors construct disaster-specific ground-truth causal graphs by aligning an impact-chain framework with post-event expert reports, providing a rigorous benchmark for evaluation.
• Social-Media Causal Extraction: The study evaluates LLMs on extracting causal relations among a fixed set of variables for social media analysis, distinguishing this task from open-ended causal discovery.
• Evidence vs. Prior Knowledge: The research explicitly examines the tension between evidence-based extraction and model priors, revealing how post quality and model architecture influence the validity of extracted relations.

Results

• Performance vs. Random Baseline: All evaluated LLMs significantly outperformed a random baseline (Erdős–Rényi model), demonstrating that the models can infer disaster-specific causal relations from social media text rather than guessing.
• Model Comparison:
  • Grok4.3 achieved the highest F1 scores (0.75 for Hurricane Irma, 0.71 for Hurricane Harvey), driven by high edge recall while maintaining strong node precision. This performance is attributed to its combination of advanced reasoning and native social media access.
  • GPT-5.5 performed competitively among models without native social media access but exhibited a more conservative extraction approach, resulting in lower edge recall.
  • Mistral-7B showed the weakest overall performance, particularly in edge precision and recall.
• Reliance on Prior Knowledge: In the ablation study using non-informative posts, model performance dropped substantially. GPT-5.5 maintained relatively higher F1 scores in this setting, suggesting a greater reliance on internal priors to produce plausible but ungrounded structures. In contrast, Grok4.3 demonstrated stricter evidence thresholds, refusing to generate causal graphs for Hurricane Harvey when evidence was insufficient and showing high selectivity for Irma.

Significance and Claims
The paper claims that while LLMs show promise for extracting causal relations from social media for disaster decision-support, their deployment requires robust evidence-grounding mechanisms. The findings highlight a dual reality: advanced models like Grok4.3 can effectively leverage real-time data to build causal graphs, but there is a significant risk that models may generate plausible but unverified causal links based on pre-trained knowledge when specific evidence is lacking. Consequently, the authors argue that extracted causal graphs should be used strictly as decision-support tools requiring source validation and expert review, rather than as standalone decision systems. The study underscores the necessity of validating LLM outputs against expert reports before integrating them into high-stakes disaster intelligence pipelines.