LLMs in the Real World: Evaluating "AI" in Emergency Contexts

Technical Summary: LLMs in the Real World: Evaluating "AI" in Emergency Contexts

Problem Statement
The paper identifies a critical "information gap" and "accountability gap" between NLP researchers and the stakeholders deploying Large Language Model (LLM) technologies in high-stakes, real-world environments, specifically emergency services. While academic research has established limitations regarding low-resource languages, domain shifts, and the inadequacy of quantitative metrics, these findings are often lost in commercial hype. Decision-makers in essential services (e.g., law enforcement, emergency response) frequently lack the technical literacy to evaluate these tools, leading to the deployment of systems that may be unsafe, ineffective, or ethically compromised. The authors argue that the research community has a responsibility to bridge this gap through science outreach, as the current lack of communication allows for the "risky and potentially harmful deployment of NLP systems" affecting vulnerable populations, such as refugees and non-native English speakers.

Methodology
The authors employ a qualitative case study approach centered on the deployment of a text-to-911 machine translation (MT) service in a U.S. city. The methodology involves:

1. Stakeholder Engagement: The authors conducted meetings with staff at a local 911 center, including dispatchers and representatives from the third-party software provider (which utilizes Microsoft Azure for language detection and translation).
2. Technical Probing: The research team, alongside colleagues from a victim services program, tested the system in real-time using native languages (Arabic and Nepali). They specifically probed the model with linguistic phenomena common in emergency texts, including dialectal variations, "text speak," typos, referential ambiguity, idioms, and code-switching.
3. Document Analysis: The authors analyzed marketing materials, press coverage, and the lack of public evaluation data or model cards associated with the deployed system.
4. Comparative Analysis: The findings were contextualized against existing NLP research regarding model performance degradation outside training domains and the specific sociolinguistic realities of the target user groups.

Key Findings and Results
The case study revealed significant discrepancies between the marketed capabilities of the MT system and its actual performance and transparency:

• Linguistic Mismatch: The system failed to accommodate the specific linguistic realities of its users. For Arabic speakers, the system only supported Modern Standard Arabic (MSA), ignoring the dialectal variations and non-Latin orthography challenges common in text messaging. For Nepali speakers, the system required the Devanagari script, whereas the local Bhutanese-Nepali community predominantly uses the Latin script for texting.
• Lack of Transparency and Consent: The interface did not disclose to the end-user that their text was being processed by an AI, nor did it reveal the detected language or the translation output. This lack of informed consent prevented users from verifying the accuracy of the translation.
• Absence of Evaluation Data: The 911 center staff had received no evaluation data, quality assurance services, or training from the software provider. There was no ongoing monitoring of the system's success in reducing response times or ensuring accuracy.
• Misconceptions: The deployment was driven by several misconceptions, including the belief that "AI" is a homogeneous, superhuman solution; that quantitative metrics guarantee safety; and that technological solutions can replace human interpreters in critical contexts without significant risk.

Key Contributions
The paper contributes to the field by:

1. Documenting Real-World Failure Modes: It provides empirical evidence of how LLM-based MT systems fail in emergency contexts due to a lack of alignment between training data (e.g., MSA, Devanagari) and real-world usage (e.g., dialects, Latin script).
2. Identifying Systemic Gaps: It highlights the "accountability gap" where vendors, regulators, and end-users lack the necessary expertise and resources to audit or safely deploy these technologies.
3. Proposing Best Practices: The authors offer a set of concrete recommendations for stakeholders, including:
   • Mandating "Model Cards" for all ML-based software to distinguish between on-label and off-label uses.
   • Requiring informed consent and transparency for both dispatchers and end-users.
   • Establishing contingency plans and human oversight (e.g., qualified human interpreters) for high-stakes applications.
   • Adopting specific ethical policies (adapted from the SAFE-AI Task Force) such as the ability to shift between AI and human interpreting and clear grievance processes.

Significance and Claims
The authors claim that the advancement of scientific research often focuses on solving "hard" problems, while the "easy" problems—those where the latest technology is unnecessary or where basic safeguards are overlooked—are the most dangerous. The paper argues that the deployment of LLMs in emergency contexts without adequate evaluation or human oversight poses a direct threat to public safety and equity.

The significance of the work lies in its call to action for the NLP research community to engage in science outreach. The authors assert that researchers must play a greater role in articulating findings to the public, policymakers, and industry partners to prevent the "inappropriate, ineffective, and sometimes even harmful deployment" of their technologies. They emphasize that while AI can be a tool, it is not a universal solution, and in critical scenarios like emergency response, prioritizing human expertise and robust governance is both an ethical and financial imperative. The paper concludes that without active engagement from researchers to counter misinformation and provide technical guidance, the potential for preventable harm in vulnerable communities will persist.