💬 NLP

A Dynamic Fusion Model for Consistent Crisis Response

This paper addresses the critical need for stylistic consistency in automated crisis communications by proposing a novel evaluation metric and a two-stage fusion-based generation model that significantly improves response quality and uniformity compared to existing baselines.

Original authors: Xiaoying Song, Anirban Saha Anik, Eduardo Blanco, Vanessa Frias-Martinez, Lingzi Hong

Published 2026-04-03

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiaoying Song, Anirban Saha Anik, Eduardo Blanco, Vanessa Frias-Martinez, Lingzi Hong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are in the middle of a massive storm, like a hurricane. You are scared, you need help, and you turn to social media to ask, "Where is the nearest shelter?" or "How do I get medical aid?"

Ideally, the government or relief organizations should reply instantly with clear, professional, and helpful answers. But in reality, they are overwhelmed. They can't reply to everyone personally. So, they use AI (like a super-smart robot) to help write the replies.

The Problem: The "Robot Mood Swing"
The paper points out a big flaw in how these robots currently work. Sometimes, the robot gives you a perfect, detailed answer with phone numbers and maps. The next minute, it replies to a similar question with a vague, "Stay safe," or a confusing, "I'm not sure."

This inconsistency is dangerous. If one person gets a clear guide and another gets a vague hint, they lose trust in the organization. It's like a doctor who gives you a detailed prescription one day and just says "take some medicine" the next. You wouldn't trust them.

The Solution: The "Taste-Test Fusion Kitchen"
The authors propose a new way to fix this called a "Dynamic Fusion Model."

Think of it like a high-end restaurant kitchen trying to make the perfect soup for thousands of hungry people.

The Chefs (The Models): Instead of asking just one chef to make the soup, they ask two different chefs.
- Chef A (Instructional Prompt): A creative chef who knows how to talk nicely and sound professional.
- Chef B (RAG - Retrieval-Augmented Generation): A fact-checker chef who has a giant library of official government manuals (like FEMA guides) and knows the exact facts.
The Tasting (Evaluation): Before serving the soup, a "Food Critic" (an AI evaluator) tastes both bowls. It rates them on three things:
- Professionalism: Does it sound like a responsible authority?
- Actionability: Does it tell you exactly what to do (e.g., "Call 555-1234")?
- Relevance: Is it actually answering the question?
The Fusion (The Magic Pot): Here is the clever part. Instead of just picking the "best" bowl, the system mixes them together.
- It takes the professional tone from Chef A.
- It takes the exact phone numbers and facts from Chef B.
- It blends them into a single, perfect bowl of soup that is consistent, helpful, and trustworthy.

Why This Matters
The paper tested this "Fusion Kitchen" on real data from hurricanes.

Old Way: The robot's answers were all over the place. Some were great, some were bad.
New Way (Fusion): Every single answer was high-quality. Whether you asked about food, shelter, or rescue, the robot sounded equally professional and gave equally clear instructions.

The Result
By using this "fusion" method, the AI stops having "mood swings." It becomes a reliable, consistent helper. Just like a good emergency service should be: no matter who you are or what time you call, you get the same high-quality, life-saving guidance.

In short: They taught the AI to stop guessing and start combining the best parts of different answers to ensure everyone gets the same high-quality help, building trust when it matters most.

1. Problem Statement

The paper addresses a critical gap in automated crisis communication: response style consistency. While Large Language Models (LLMs) are increasingly used to assist government agencies and NGOs in responding to crisis-related queries on social media, existing systems often generate responses with varying styles, tones, and levels of detail.

The Issue: Inconsistent responses (e.g., some being highly professional and actionable, others vague or dismissive) can erode public trust, cause confusion, and lead to inequitable access to aid.
The Gap: Previous research has focused on semantic or factual consistency in general dialogue, but few studies have addressed style consistency in crisis contexts. Furthermore, there is no established metric to evaluate consistency across the specific dimensions required for crisis response: Professionalism, Actionability, and Relevance.
The Challenge: A "one-size-fits-all" approach fails because user needs vary significantly. However, the quality and style of the response must remain uniform to signal organizational reliability.

2. Methodology

The authors propose a Dynamic Fusion Framework that generates consistent responses by integrating the strengths of multiple candidate generation methods.

A. Definition of Consistency

The paper defines consistency not as semantic repetition, but as the uniformity of style across three critical dimensions:

Professionalism: Accuracy, reliability, and credibility of the assistance.
Actionability: Clarity and practicality of the steps provided (e.g., specific phone numbers, URLs).
Relevance: How closely the response addresses the specific user query.

Metrics:

Variation: Calculated as the average variance of scores across the three dimensions for a set of responses.
Consistency Score: Defined as $1 - \text{Variation}$ . Higher scores indicate more uniform quality across all generated responses.

B. The Fusion Framework

The framework operates in a two-stage process:

Candidate Response Generation:
Two complementary state-of-the-art methods generate initial candidates:
- Instructional Prompt (IP): Uses zero-shot prompting to leverage the LLM's reasoning and generalization capabilities.
- Retrieval-Augmented Generation (RAG): Retrieves factual information from authoritative sources (specifically FEMA guidelines) to ground responses in verified data, reducing hallucinations.
Multi-dimensional Evaluation:
The candidate responses are evaluated by an LLM (GPT-4o mini) and BERTScore to assign scores for Professionalism, Actionability, and Relevance.
Fusion-based Generation:
A fusion mechanism synthesizes a final response. The paper experiments with three fusion strategies:
- Fusion w/o Eval: Fusing candidates without explicit score guidance.
- Fusion w/ Eval: Providing numeric scores to the LLM as implicit guidance.
- Fusion w/ Eval & Instruct: Adding structured instructions to guide the LLM on how to reason over the scores (e.g., "retain strengths from Response A, integrate key elements from Response B").
- Fusion w/ Eval & Weight Instruct (Proposed Best): Introduces weighted scores (e.g., 40% Professionalism, 40% Actionability, 20% Relevance) to prioritize critical dimensions during synthesis, allowing for targeted optimization.

3. Key Contributions

Novel Metric: Introduction of a "Consistency" metric specifically designed for crisis communication, focusing on the uniformity of Professionalism, Actionability, and Relevance.
Fusion Framework: A dynamic generation approach that integrates outputs from Instructional Prompting and RAG, utilizing weighted evaluation guidance to produce responses that are both high-quality and stylistically consistent.
Comprehensive Evaluation: Extensive experiments across multiple LLMs (Llama-3.1, Mistral), diverse crisis scenarios (Hurricanes, Earthquakes, Typhoons), and human evaluations, demonstrating the framework's robustness.

4. Experimental Results

The study was conducted on a dataset of 540 crisis-related tweets from Hurricane Harvey, Irma, and Maria, and validated on a cross-crisis dataset (Earthquakes/Typhoons).

Performance vs. Baselines: The fusion approach significantly outperformed baselines (Instructional Prompt, RAG, RAG-PE, and "Prompt and Select") in Overall Quality and Consistency.
- Example (Llama): The best fusion method (Fusion w/ Eval & Weight Instruct) achieved an Overall Quality of 0.95 and Consistency of 0.94, compared to the best baseline (RAG) which scored 0.80 and 0.84 respectively.
Dimension Improvement: While relevance remained stable across methods, the fusion approach drastically improved Professionalism and Actionability, reducing the variance between responses.
Cross-Crisis Generalization: The framework maintained high performance (Consistency: 0.96) when applied to earthquakes and typhoons, proving it is not limited to hurricane scenarios.
Human Evaluation: Human annotators preferred the fused responses (Preference score: 0.86) over IP (0.48) and RAG (0.47), citing better structure, concrete instructions, and reliable references.
Iteration Analysis: Consistency scores plateaued after three fusion iterations, indicating diminishing returns beyond a single fusion step.

5. Significance and Impact

Trust and Safety: By ensuring that every user receives a response with a uniform level of professionalism and actionable guidance, the system mitigates the risk of users receiving misleading or dismissive information, which is critical during life-threatening emergencies.
Scalability: The framework offers a scalable solution for NGOs and government agencies that lack the human resources to manually respond to thousands of crisis queries while maintaining high standards.
Methodological Advancement: The paper moves beyond simple "best-of-N" selection (Prompt and Select) to a synthesis approach, demonstrating that combining the reasoning of IP with the factual grounding of RAG via weighted fusion yields superior results.
Practical Application: The use of authoritative sources (FEMA) within the RAG component ensures factual accuracy, addressing the "hallucination" problem common in LLMs during high-stakes scenarios.

In conclusion, this work establishes that consistency is a measurable and optimizable attribute in crisis AI, and proposes a robust fusion architecture that successfully balances the trade-offs between different response qualities to deliver reliable, uniform, and actionable aid information.