PACE: A Personalized Adaptive Curriculum Engine for 9-1-1 Call-taker Training

In partnership with the Metro Nashville Department of Emergency Communications, the authors propose PACE, a personalized adaptive curriculum engine that leverages probabilistic skill modeling and contextual bandits to significantly accelerate 9-1-1 call-taker training time-to-competence and mastery while aligning closely with expert pedagogical judgments.

The Gist

Imagine you are training a new 9-1-1 call-taker. This isn't just a job where you learn to say "What's your emergency?" It's a high-stakes role where a split-second mistake can mean the difference between life and death.

To get good at this, a trainee needs to master over 1,000 different skills. These skills are like a giant, tangled web of dominoes. If you knock over the wrong one (like forgetting to ask if a patient is breathing), the whole chain reaction fails, and the wrong help gets sent.

The problem? There are too many trainees and not enough expert trainers. A human trainer can't possibly watch every single call, remember every tiny mistake a student made three weeks ago, and know exactly which drill to give them next. They are overwhelmed, and students are getting a "one-size-fits-all" training that doesn't work for everyone.

Enter PACE.

Think of PACE (Personalized Adaptive Curriculum Engine) as a super-smart, tireless co-pilot for the trainer. It's like having a GPS for learning that doesn't just show you the map, but actively steers the car based on how fast you're driving and how tired you are.

Here is how PACE works, using some everyday analogies:

1. The "Skill Web" (The Map)

Instead of treating skills as separate items on a checklist (like "Knows CPR" and "Knows Fire Safety"), PACE sees them as a giant, connected spiderweb.

• The Analogy: Imagine a tree. If you don't know how to climb the first branch (assessing consciousness), you can't reach the higher branches (performing CPR).
• What PACE does: It knows that if a student struggles with the "consciousness" branch, they will likely fail the "CPR" branch later, even if they haven't tried it yet. It connects the dots so it can fix the root problem before it becomes a disaster.

2. The "Mind Reader" (The Belief Tracker)

Trainers can't read minds. They only see what the student says during a simulation. But sometimes a student gets lucky, and sometimes they panic and freeze even though they know the answer.

• The Analogy: Think of PACE as a detective with a magnifying glass. It doesn't just look at the final score; it looks at how the student answered. Did they hesitate? Did they need a hint?
• What PACE does: It builds a "probability cloud" for every single skill. It says, "I'm 90% sure Sarah knows how to handle a car crash, but I'm only 40% sure she knows how to handle a drowning." It constantly updates this cloud as the student practices.

3. The "Memory Bank" (Learning & Forgetting)

Humans are weird. Some people learn fast but forget fast (like a sponge that leaks). Others learn slow but remember forever (like a stone).

• The Analogy: Imagine a leaky bucket. If you stop pouring water (practice) into the bucket, the water (knowledge) leaks out.
• What PACE does: It tracks your specific leak rate. If it knows you forget things quickly, it schedules a quick review of that skill before you even realize you've forgotten it. If you're a slow learner, it gives you more time to practice before moving on. It treats every student like a unique individual, not a robot.

4. The "Coach's Playbook" (The Contextual Bandit)

This is the part where PACE decides what to do next.

• The Analogy: Imagine a basketball coach. If a player is great at shooting but terrible at defense, the coach doesn't make them shoot 100 more free throws. They put them in a drill that fixes the defense just enough so they can keep improving.
• What PACE does: It uses a mathematical strategy (called a "Contextual Bandit") to pick the perfect training scenario. It balances two things:
  1. Practice what you're good at (to keep it fresh).
  2. Practice what you're bad at (to fix the holes).
     It picks the scenario that gives the biggest "bang for the buck" in learning time.

The Results: Why It Matters

The paper tested PACE against old training methods and found some amazing results:

• Speed: Students reached "job-ready" status 19.5% faster. That's like finishing a marathon in 2 hours instead of 2 hours and 15 minutes.
• Mastery: At the end of training, students knew 10.95% more than those trained the old way.
• The Trainer's Time: This is the biggest win. A human trainer used to take 11.58 minutes to review a call, find errors, and give feedback. PACE does this in 34 seconds. That's a 95% time savings.
• Expert Approval: When real, experienced trainers looked at PACE's recommendations, they agreed with the AI 95% of the time. It's not replacing the human; it's giving the human a superpower.

In a Nutshell

PACE is a system that turns 9-1-1 training from a "factory line" (where everyone gets the same boring drills) into a personalized video game. It knows exactly what level you are on, what boss you are struggling with, and when you are about to lose a "life" (forget a skill), so it can give you the exact power-up you need to win.

It saves trainers time, saves students frustration, and ultimately, helps save lives by ensuring call-takers are truly ready for the real thing.

Technical Summary

1. Problem Statement

Training 9-1-1 call-takers is a high-stakes, complex process requiring mastery of over 1,000 interdependent skills across diverse incident types (e.g., medical, fire, law enforcement). Current training models face three critical bottlenecks:

• Scalability vs. Personalization: A nationwide labor shortage limits the number of experienced trainers, making it impossible to tailor instruction to individual trainees' evolving competencies. Most programs default to uniform curricula, ignoring individual learning patterns (e.g., fast acquisition with rapid forgetting vs. slow acquisition with retention).
• Engagement-Learning Discrepancy: Existing recommendation systems often optimize for engagement (clicks/duration) rather than pedagogical goals. Trainees may prefer easy scenarios, but true skill acquisition requires practicing at the "edge of competence."
• Inductive Knowledge Tracing & Granularity: Traditional knowledge tracing treats skills as independent, which is infeasible for 9-1-1 training where skills are highly structured and interdependent. Furthermore, existing systems often operate at coarse granularities (e.g., "medical calls"), failing to detect specific deficits (e.g., "pediatric overdose" vs. "adult chest pain") that can lead to fatal errors in real-world dispatch.
• Persistent Learner Modeling: Longitudinal training (6–8 weeks, 200+ calls) exceeds the context limits of current LLM-based tutors, leading to "context rot" where historical performance data is lost, preventing effective scheduling of retention practice.

2. Methodology: The PACE System

The authors propose PACE (Personalized Adaptive Curriculum Engine), a co-pilot system designed to augment trainer decision-making. PACE operates as a curriculum layer over existing 9-1-1 training infrastructure.

A. Problem Formulation

The curriculum design is formalized as a decision-making problem over a Structured Skill Graph ($G = (V, E)$):

• Nodes ($V$): 1,053 nodes representing conditions (incident states), questions (information gathering), and instructions (directive delivery).
• Edges ($E$): 1,283 edges capturing sequential dependencies, implication links, and entailment links across 63 incident types.
• Objective: Maximize terminal competence while minimizing the gap between the system's belief of the trainee's state and their true latent state.

B. Core Components

PACE consists of three main modules:

1. Belief Tracker (Inductive Learning):

   • Observation Extraction: Uses an LLM to extract structured observations (compliance, partial compliance, violation, or not applicable) and error types (slip, misconception, omission) from call transcripts.
   • Probabilistic Belief: Maintains a Beta posterior distribution $Beta(\alpha_v, \beta_v)$ for each skill node $v$ to represent uncertainty over mastery $\theta_v$.
   • Inductive Propagation: When a skill is observed, evidence is propagated to similar unobserved nodes using a precomputed Similarity Index ($\Phi$). This index combines semantic embeddings (from sentence transformers) with positional compatibility in the skill graph, allowing the system to infer mastery of related skills without direct observation.

2. Dynamics Estimator (Longitudinal Modeling):

   • Instead of static profiles, PACE models individual Learning Pace ($\lambda$) and Forgetting Rate ($\psi$).
   • Learning Pace: Estimated as the average marginal mastery gain per practice opportunity.
   • Forgetting Model: Uses a power-law decay function based on real-time elapsed time ($\tau$) rather than session count, capturing the phenomenon that recently acquired skills decay faster than consolidated ones.
   • These parameters are estimated via a surrogate model fitted to observed retention outcomes.

3. Contextual Bandit (Scenario Selection):

   • Formulates scenario selection as a contextual bandit problem to balance exploration (diagnosing unknown skills) and exploitation (reinforcing known gaps).
   • Context Vector: Includes model quality (belief variance), current coverage, estimated $\lambda$ and $\psi$, average mastery of weak skills, and training progress.
   • Algorithm: Uses Thompson Sampling to select batches of $K=5$ scenarios that maximize expected learning gain under uncertainty.

C. Runtime Phases

• Cold Start: PACE observes 15 initial sessions without intervention to initialize beliefs and estimate dynamics.
• Adaptive Phase: PACE actively selects and recommends training batches based on the updated belief state and dynamics.

3. Key Contributions

1. Formalization: Defined 9-1-1 training as curriculum optimization over a structured, hierarchical skill graph, enabling fine-grained inductive reasoning.
2. System Design: Developed PACE, the first system to integrate probabilistic belief tracking, individual learning/forgetting dynamics, and contextual bandits specifically for high-stakes emergency response training.
3. Evaluation Framework: Created a robust evaluation pipeline using role-played trainee agents (LLM-based agents with controllable ground-truth parameters for learning/forgetting rates) to simulate diverse learner archetypes (Fast Learner, Struggling Learner, Quick Forgetter, etc.).
4. Real-World Validation: Conducted co-pilot studies with human training officers at a partner 9-1-1 center, validating the system against expert pedagogical judgment.

4. Experimental Results

The system was evaluated against state-of-the-art baselines (Round-Robin, Deficit-Driven, GraphRAG+LLM, and Agentic pipelines like GenMentor and Agent4Edu) using simulated trainees and real-world data.

• Efficiency (Time-to-Competence): PACE achieved a 19.50% reduction in the number of sessions required to reach competence (Zero-to-Hero) compared to the best agentic baseline (Agent4Edu).
• Effectiveness (Terminal Mastery): PACE achieved 10.95% higher terminal mastery (Coverage@50) across all learner archetypes.
  • Example: On "Quick Forgetter" archetypes, PACE maintained 91.22% coverage, whereas GenMentor degraded to 67.51% due to a lack of temporal modeling.
• Ablation Studies:
  • Removing propagation increased the sessions-to-competence by over 100% (e.g., from 22 to 46 sessions for Fast Learners), proving the necessity of inductive inference.
  • Removing dynamics estimation caused significant performance drops for non-average learners (e.g., Quick Forgetters saw a ~10% drop in exam scores), highlighting the need for personalized temporal modeling.
• Granularity: Fine-grained (skill-level) tracking significantly outperformed coarse (department or incident-type) tracking, proving that high-stakes training requires specific skill diagnosis.
• Human-in-the-Loop (Co-Pilot Study):
  • Alignment: PACE's recommendations aligned with expert trainer judgments in 95.45% of cases (881/923).
  • Efficiency Gain: PACE reduced the turnaround time for assessment and feedback from 11.58 minutes to 34 seconds (a 95.08% reduction), saving approximately 11 minutes per call and over 4,700 minutes per typical training class.

5. Significance and Impact

• Scalability: PACE addresses the critical labor shortage in 9-1-1 training by automating the personalization burden, allowing one trainer to effectively manage larger cohorts without sacrificing quality.
• Safety: By detecting fine-grained skill gaps (e.g., specific medical protocols) and scheduling retention practice based on forgetting curves, PACE reduces the risk of trainees being deployed with hidden competency gaps that could lead to fatal errors.
• Generalizability: While focused on 9-1-1, the framework is applicable to other high-stakes procedural training domains such as medical triage, air traffic control, and crisis intervention.
• Pedagogical Validity: The high alignment with human experts confirms that the system learns domain-relevant pedagogical principles rather than exploiting simulation artifacts.

In summary, PACE represents a significant advancement in AI-driven education, moving beyond simple recommendation engines to a sophisticated, temporally-aware, and graph-structured curriculum optimizer capable of handling the complexity and high stakes of emergency response training.