💻 computer science

LLM-based uncertainty assessment of social media situational signals for crisis reporting

This paper proposes an uncertainty-aware framework that leverages large language models and external proxy data to assess the plausibility of social media claims, enabling the generation of crisis reports that explicitly communicate confidence levels to aid human decision-making during disasters.

Original authors: Timothy Douglas, Roben Delos Reyes, Asanobu Kitamoto

Published 2026-05-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Timothy Douglas, Roben Delos Reyes, Asanobu Kitamoto

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a crisis manager sitting in a command center during a massive earthquake. Your screens are flooded with thousands of text messages from people on the ground. Some say, "My house collapsed!" while others shout, "The sky is falling!" or "I saw a tsunami coming!"

Right now, most computer systems that try to summarize these messages treat every single text as if it were a verified fact. They just count the votes: "Okay, 50 people said the bridge is out, so the report says the bridge is out." But in a real disaster, some of those messages are true, some are exaggerations, and some are just rumors. Treating them all the same is like listening to a choir where everyone is singing at the same volume, even if some are singing off-key or making up lyrics.

This paper proposes a new way to listen. Instead of just counting the messages, the system asks: "How likely is this story to be true, and how sure is the computer about that guess?"

Here is how the system works, broken down into three simple steps using a creative analogy:

1. The Sorting Hat (Classification)

First, the computer reads the messages and sorts them into buckets. It doesn't just read the text; it tags them.

"My house is gone" goes into the Damage bucket.
"I need water" goes into the Needs bucket.
"The roads are blocked" goes into Infrastructure.

This is standard stuff. But the old systems stopped here, assuming every message in the "Damage" bucket was equally real.

2. The Reality Check (Uncertainty Assessment)

This is the paper's big innovation. The computer now takes a message from the "Damage" bucket and compares it against a Weather Report (which the authors call "proxy data").

In this study, they used data from the USGS (a real-world earthquake monitoring system) that tells them exactly how hard the ground shook in specific areas.

The Scenario: A tweet says, "The entire city is underwater!"
The Reality Check: The computer looks at the USGS data. It sees the shaking was very light in that city.
The Verdict: The computer says, "That story is implausible (it probably didn't happen), and I am very confident in that judgment."
Another Scenario: A tweet says, "I heard a loud noise and my coffee cup fell."
The Reality Check: The USGS data shows a strong tremor in that exact spot.
The Verdict: The computer says, "That story is highly plausible, and I am very confident."
The Tricky Scenario: A tweet says, "A building collapsed," but the USGS data is vague or the area is hard to map.
The Verdict: The computer says, "That story might be true, but I'm not very confident in my judgment."

The system gives every message two scores:

Plausibility: How likely is this to be real? (1 to 5 stars).
Confidence: How sure is the computer about that rating? (0% to 100%).

3. The Filtered Report (Crisis Reporting)

Finally, instead of making one giant, messy summary that mixes rumors with facts, the system creates different reports based on these scores.

Report A (The "Go" List): Contains only stories that are highly plausible and the computer is very sure about. This is what the crisis team uses to send out immediate help.
Report B (The "Watch" List): Contains stories that seem plausible, but the computer isn't sure. The team knows to double-check these before acting.
Report C (The "Ignore" List): Contains stories that seem fake or are just rumors. The team can safely ignore these to save time.

Why This Matters

The paper tested this on over 200,000 tweets from six different earthquakes. They found that:

Not all messages are equal: The system successfully separated the "likely true" messages from the "likely fake" ones.
The reports change: When they generated reports using only the "high confidence" messages, the stories were different than when they used all the messages mixed together.

The Bottom Line:
This framework doesn't try to replace human experts or act as a "truth machine" that knows everything. Instead, it acts like a smart filter. It helps human crisis managers prioritize their time by saying, "Here are the facts we are sure about, here are the things we need to check, and here are the things we can probably ignore." It turns a chaotic flood of noise into a structured, manageable list of signals, helping people make better decisions when time is running out.

1. Problem Statement

Social media has become a critical source of situational awareness (SA) during disasters, offering real-time insights into evolving impacts and needs. However, existing automated crisis reporting systems that leverage Large Language Models (LLMs) suffer from a critical flaw: they implicitly assume that all classified social media posts are equally plausible.

In reality, information quality during crises varies significantly due to incomplete data, contradictions, rumors, and source reliability. Treating all signals as equally valid risks amplifying noise or discarding genuine but anomalous signals. Furthermore, current pipelines rarely communicate the degree of certainty associated with extracted information, limiting their utility for human decision-makers who must prioritize actions under time pressure. There is a lack of frameworks that explicitly model epistemic uncertainty (uncertainty arising from incomplete knowledge) by grounding social media claims against independent, real-world data.

2. Methodology

The authors propose a three-step uncertainty-aware framework that integrates external proxy data to assess the plausibility and confidence of social media signals. The pipeline is modular and consists of:

Step 1: Situational Awareness (SA) Classification

Input: Raw social media posts (tweets).
Process: A zero-shot LLM classifies tweets into specific SA categories (e.g., "Affected People," "Infrastructure Damage," "Needs & Donations") based on an established schema.
Output: Tweets enriched with SA labels and rationales.

Step 2: Proxy-Informed Uncertainty Assessment (Core Contribution)

This step evaluates whether a specific situational claim plausibly reflects real-world conditions.

Inputs:
1. The classified tweet (text + metadata).
2. External Proxy Data: Structured, independently collected impact estimates (e.g., USGS PAGER summaries for earthquakes, including Modified Mercalli Intensity, population exposure, and fatality estimates).
Mechanism: The LLM is prompted to reason jointly over the tweet content and the proxy data. It does not treat the proxy as absolute ground truth but as a contextual anchor.
Outputs: Two distinct metrics for each tweet:
1. Plausibility Score ( $l_i$ ): A graded score on a 5-point Likert scale (1 = Implausible, 5 = Highly Plausible) indicating how well the claim aligns with expected real-world conditions.
2. Confidence Score ( $c_i$ ): A continuous score (0–100) representing the model's self-expressed certainty in its plausibility judgment.
Elicitation Strategy: To ensure robustness, the framework uses a sampling strategy (generating $M=3$ responses with varying temperatures) and an aggregation strategy (averaging scores) to capture variability and reduce overconfidence, adapting methods from Xiong et al. [7].

Step 3: Uncertainty-Aware Report Generation

Process: Instead of generating a single summary, the system aggregates tweets based on their uncertainty profiles.
Operationalization: The plausibility-confidence space is partitioned into four quadrants:
- Q3: High Plausibility / High Confidence (Actionable, verified signals).
- Q1: High Plausibility / Low Confidence (Emerging signals requiring monitoring).
- Q4: Low Plausibility / High Confidence (Confidently expressed but likely false/rumors).
- Q2: Low Plausibility / Low Confidence (Ambiguous noise).
Output: Structured crisis reports that explicitly communicate what is reported and how certain the system is, allowing for "epistemic triage."

3. Key Contributions

Novel Framework: Introduction of a proxy-informed uncertainty assessment framework that explicitly models both plausibility and confidence for social media signals, moving beyond binary classification.
Empirical Evaluation: A large-scale study involving over 200,000 tweets across six major earthquake events (e.g., 2014 Chile, 2015 Nepal, 2021 Haiti), using USGS PAGER data as the external proxy.
Operationalization: Demonstration of how uncertainty estimates can be used to generate differentiated crisis reports, enabling analysts to prioritize information based on risk and certainty levels.

4. Results

The study addresses two primary Research Questions (RQs):

RQ1: Does the framework produce variation in uncertainty estimates?
- Yes. Unlike a baseline where all tweets are treated as equally plausible (zero entropy), the proposed framework generates structured heterogeneity.
- Metrics: Shannon entropy analysis showed non-zero dispersion for both plausibility (up to 1.82 bits) and confidence (up to 1.21 bits) across events.
- Observation: While LLMs exhibited a tendency toward high confidence (a known bias), the system still successfully differentiated signals, assigning varying plausibility scores (1–5) rather than collapsing them into a single category.
RQ2: Does conditioning on uncertainty alter report structure?
- Yes. Reports generated from different uncertainty quadrants (Q1–Q4) showed measurable differences in semantic composition compared to a baseline summary.
- Metrics: Cross-report similarity analysis revealed that reports from specific quadrants (e.g., Q4 in the Fukushima event) were semantically distinct from the baseline and other quadrants.
- Internal Coherence: Reports derived from specific uncertainty profiles often exhibited higher internal semantic coherence, suggesting that grouping by uncertainty creates more focused narratives.

5. Significance and Implications

Risk-Aware Decision Making: The framework transforms raw data into "risk-aware" intelligence. It allows crisis responders to distinguish between high-confidence actionable intelligence and ambiguous or potentially misleading signals without requiring manual fact-checking of every post.
Handling Anomalies: By separating "Low Plausibility/High Confidence" (Q4) signals, the system helps identify confident rumors or localized outlier events (e.g., a specific building collapse not captured in aggregate regional data) that might otherwise be discarded or accepted blindly.
Scalability: The approach is platform-agnostic and can be applied to other disaster types (floods, wildfires) provided structured external proxy data is available.
Human-in-the-Loop: The system is designed to support human analysts, not replace them. It structures the information landscape, allowing humans to focus their verification efforts on specific uncertainty profiles (e.g., investigating Q1 emerging signals or Q4 potential misinformation).

6. Limitations

Data Dependency: The framework relies on the availability of structured external proxy data (like USGS PAGER), which may not exist for all hazard types (e.g., complex floods or civil unrest).
Threshold Sensitivity: The quadrant partitioning relies on fixed thresholds (Likert=3, Confidence=50) which were not optimized; different thresholds could alter the distribution of tweets.
Evaluation Metrics: The study uses embedding-based similarity for evaluation rather than human expert fact-checking, which is difficult to perform at scale for real-time crises.
Model Bias: The observed overconfidence in LLMs remains a challenge, though the sampling strategy mitigates it partially.

Conclusion

This paper presents a significant step forward in Crisis Informatics by moving from simple extraction to uncertainty-aware synthesis. By explicitly modeling the plausibility of social media claims against real-world proxies and eliciting model confidence, the framework provides a more transparent, nuanced, and operationally useful tool for automated crisis reporting.