Empowering 9-1-1 Calltaking Training with Generative AI: Experiences and Lessons Learned
This paper presents the design, deployment, and evaluation of a Generative AI-powered training system for emergency call-takers in partnership with Metro Nashville, distilling four key lessons from real-world usage data to guide the implementation of AI-driven solutions in safety-critical public sector environments facing severe staffing and scalability challenges.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are training to be a 9-1-1 call-taker. It's a high-stakes job where you have to stay calm, ask the right questions, and save lives, all while talking to people who are terrified, angry, or confused. In the real world, training for this is a nightmare. There aren't enough experienced teachers, and it takes up to 720 hours of one-on-one instruction to get a new hire ready. That's like teaching someone to drive for three full months without them ever touching a real car, just sitting in a classroom. Plus, with staff shortages exceeding 25% in many centers, the few experts who are there are too busy to teach everyone.
To fix this, researchers from Vanderbilt University teamed up with the Metro Nashville Department of Emergency Communications (MNDEC) to build a "digital coach" powered by Generative AI. They didn't just build a computer program and hope for the best; they spent six months actually using it with 190 real trainees across 1,120 training sessions. They analyzed nearly 100,000 interactions to see what actually worked in the messy, real world.
Here is the story of what they learned, told through four big lessons.
Lesson 1: Don't Just Talk, Build a Tiny Prototype
The Myth: You can sit in a meeting room, interview experts, and write a perfect list of requirements before you start coding.
The Reality: That doesn't work here. The researchers found that the AI team didn't know what a real emergency call felt like, and the call-takers didn't know what AI could actually do. They were speaking different languages.
The Fix: Instead of waiting to build a perfect system, they built a "tiny, clunky" version in just one month. They let the experts play with it. Suddenly, the experts stopped saying, "We need more realistic scenarios," and started saying, "Whoa, can you make this caller sound like a panicked non-native English speaker? That's what we need!"
By playing with the toy, the experts realized that the diversity of the caller (their age, language, and panic level) mattered more than just the type of emergency (like a fire or a heart attack). The experts turned from "auditors" who just said "no" into "co-architects" who helped design the system. The researchers learned that you can't just read a manual; you have to build something small, let people break it, and then fix it together.
Lesson 2: Don't Let the AI "Guess" the Rules
The Myth: You can just tell a super-smart AI, "Check if the trainee followed the rules," and it will get it right every time.
The Reality: When the AI tried to read long, complicated call transcripts and check against hundreds of rules, it started to get confused. In preliminary experiments, a pure AI model (Llama 3.2) dropped its accuracy from 78% down to 42% when the conversation got long and messy. It was like asking a student to solve a math problem while reading a novel at the same time; they just got lost.
The Fix: The team built a "hybrid" system. They took the strict, unchangeable rules of the job (like "ask for the address in the first 3 turns") and turned them into a rigid, mathematical checklist that a computer can't mess up. They let the AI handle the "fuzzy" parts, like understanding if a caller sounds scared or if the background noise is chaotic.
Think of it like a referee in a soccer game. The AI is the camera crew capturing the action, but the "formal rules" are the referee's whistle. The referee (the math) says, "Goal!" or "No goal!" based on exact lines, while the camera (the AI) describes the drama. This mix made the final system 94.3% to 95.9% accurate, which is much better than the AI guessing alone.
Lesson 3: Not Every "Error" is a Bug
The Myth: If a trainee says, "The computer made a mistake!", the computer must be broken.
The Reality: In high-stress training, people often blame the machine when they feel stupid. The researchers found that 28.24% of the time, trainees reported "system errors" that were actually just them making a mistake.
For example, a trainee might ask for an address, hear "123 Main Street," and then move on. They think, "I got the address, the system should give me points!" But the rule says you have to verify the address (check the spelling or cross-streets). The trainee felt the system was unfair, but the system was actually doing its job perfectly.
The Fix: The team created a "triangulated" feedback loop. When a trainee complained, a human expert (a quality assurance officer) would look at the recording and the logs before the developers changed anything. This stopped the team from accidentally "fixing" the system to make it easier for trainees, which would have ruined the training. It taught the team that sometimes, the "error" is actually the lesson.
Lesson 4: Frustration is Good (If You Explain It)
The Myth: A good training game should be easy and fun so people don't get frustrated.
The Reality: If the training is too easy, the trainees won't be ready for the real chaos of an emergency. But if it's too hard, they quit. The researchers found a "sweet spot" of difficulty. When the scenarios got harder, trainees' scores went down, but they also started blaming the AI more often.
However, the way the AI gave feedback changed everything. Early on, the AI just listed all the mistakes: "You failed here, you failed there." This made trainees want to quit.
The Fix: The team changed the feedback to be "constructive." Now, when a trainee messed up, the AI also highlighted what they did right. "You missed the address check, but you did a great job calming the caller down." This kept the difficulty high (so they learned) but made the struggle feel productive rather than hopeless. After this change, the number of people finishing high-difficulty sessions jumped from 76% to 89%.
The Bottom Line
This paper suggests that putting AI into serious government jobs isn't just about having a smart algorithm. It's about how you build it, how you check it, and how you treat the humans using it.
- Don't wait for perfection; build small and learn together.
- Don't let the AI guess at strict rules; use math to double-check it.
- Don't assume every complaint is a bug; check if it's just human stress.
- Don't make it too easy; challenge people, but tell them what they did well.
The researchers are careful to say this is based on one specific city (Nashville) and one specific type of training. They don't claim this solves everything, but they do suggest that if other cities want to use AI for training, they should follow these human-centered steps to avoid the pitfalls they found. It's a reminder that in high-stakes jobs, the technology is just the tool; the real magic is in how humans and machines learn to trust each other.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.