Context-Aware Emergency Department Triage Using Pairwise Comparisons and Bradley-Terry Aggregation

This study demonstrates that a zero-shot, pairwise comparison approach using a large language model and Bradley-Terry aggregation for emergency department triage significantly outperforms the standard Emergency Severity Index and maintains cross-site stability without requiring site-specific training, matching the performance of supervised models on external data.

The Gist

Imagine you are walking into a busy emergency room (ER). There are dozens of people waiting in the lobby, all looking sick, but the doctors and nurses can only see one person at a time. The big question is: Who should go first?

Currently, most ERs use a system called the Emergency Severity Index (ESI). Think of this like a traffic light system. It sorts patients into five colors (Red, Orange, Yellow, Green, Blue) based on how sick they look right now.

• The Problem: If five people are all "Yellow," the system doesn't know which one is more yellow than the others. So, it just lets them in the order they arrived (First-In, First-Out). This is like a grocery store line where the person with the most items in their cart gets served last, even if they are in a huge rush.

The New Idea: The "Tournament" Approach

This paper proposes a smarter way to organize the line. Instead of giving everyone a static score, the researchers suggest treating the waiting room like a sports tournament.

Here is how their new system works, broken down into simple steps:

1. The "Triage Capsule" (The Player Profile)

When a new patient arrives, the system creates a digital "player profile" for them. It gathers all their info: age, vital signs (heart rate, temperature), what they are complaining about, and their medical history (like "has diabetes" or "takes blood thinners").

2. The "Judge" (The AI Referee)

Instead of a computer just crunching numbers, they used a Large Language Model (LLM)—basically a super-smart AI that reads like a human doctor.

• The Task: The AI doesn't just look at one patient. It looks at the new patient and compares them to a few other people already waiting in line.
• The Question: "If you had to choose between Patient A and Patient B, who is more likely to get worse in the next 6 hours?"
• The Magic: The AI can read the whole story. It notices that a patient with a "mild headache" who also has a history of "heart failure" and is on "blood thinners" is actually in more danger than a patient with a "broken leg" but no other health issues. A simple checklist might miss this, but the AI sees the full picture.

3. The "Bradley-Terry" Score (The League Table)

After the AI makes these comparisons (like a referee blowing a whistle for every matchup), a mathematical formula called the Bradley-Terry model takes all those "who is sicker?" votes and creates a single, updated ranking list.

• It's like a sports league table. Even if Team A beat Team B, and Team B beat Team C, the system calculates the most logical order for the whole league based on all those head-to-head games.

What Did They Find?

The researchers tested this idea using data from two different hospitals (one in Texas and one from a massive public database). They simulated 1,000 shifts of ER chaos to see how well the new system worked compared to the old ways.

The Results:

1. Beating the Old System: The new AI ranking system was much better at spotting the sickest patients and putting them at the front of the line compared to the standard ESI traffic-light system.
2. The "Zero-Shot" Superpower: This is the coolest part. They trained a traditional computer model (XGBoost) on the Texas hospital data. When they took that model to the second hospital, it got confused and performed worse because the patients were slightly different.
   • However, the AI Judge (LLM) worked just as well at the second hospital without needing any new training. It was like bringing a world-class referee who already knows the rules of the game, regardless of which stadium they are in.
3. Saving Time: By putting the right people first, the "time-to-provider" (how long the sickest people had to wait) dropped significantly.

Why Does This Matter?

Think of the ER waiting room as a fire station. If a fire truck is available, you don't want to send it to a small trash fire when a skyscraper is burning down next door. You need to know which fire is the biggest relative to the others right now.

• Current System: Sorts fires by how big they look from the street.
• New System: Has a smart dispatcher who looks at the wind, the building materials, and the number of people inside, then compares the two fires directly to decide which one needs the truck right now.

The Bottom Line

This paper suggests that we can make emergency rooms safer and faster by using AI not just to "score" patients, but to compare them against each other. This method is smart, adaptable to different hospitals without needing retraining, and ensures that the people who need help most get it first.

Note: The authors emphasize this is a "preprint" (a draft study) and not yet a final medical rule, but the results are very promising for the future of emergency care.

Technical Summary

1. Problem Statement

Emergency Department (ED) triage systems currently rely on classification-based models (e.g., Emergency Severity Index [ESI], National Early Warning Score 2 [NEWS2]) that assign independent acuity scores to individual patients.

• Limitations: These systems do not inherently solve the operational problem of queue prioritization. Within the same acuity level (e.g., ESI-3), patients are typically processed on a First-In-First-Out (FIFO) basis, ignoring relative risk differences.
• Generalizability Issues: Supervised machine learning (ML) models trained to predict deterioration often require large, site-specific labeled datasets. When applied to new institutions (external validation), their performance frequently attenuates due to distribution shifts in patient populations and data collection practices.
• Core Question: How can ED waiting room prioritization be reframed as a ranking problem (determining who should be seen next relative to others) rather than a classification problem, and can this approach achieve cross-site stability without local retraining?

2. Methodology

Datasets

• Development Site (Site A): MC-MED dataset (118,385 visits). Used for training supervised models and developing the ranking framework.
• External Validation Site (Site B): MIMIC-IV-ED dataset (425,087 visits). Used to test zero-shot generalizability without retraining.
• Outcome: A composite indicator of clinical deterioration within 6 hours of arrival (ICU admission, intubation, vasopressors, ventilation, or death).

The Ranking Framework

The authors propose a Pairwise Comparison + Bradley-Terry Aggregation pipeline:

1. Triage Capsule Construction: For each patient, a data structure is created containing demographics, vitals, chief complaint, and Past Medical History (PMH). Two formats were tested:
   • Structured: Binary flags and counts (~200 tokens).
   • Enriched: Full diagnosis descriptions and medication names (~350 tokens).
2. Pairwise Comparison: When a new patient arrives, they are compared against "sentinel" patients (anchors) currently in the queue.
   • Judges:
     • BT-Heuristic: Deterministic NEWS2-based logic.
     • BT-LLM: GPT-4.1 using the structured capsule.
     • BT-LLM-Enriched: GPT-4.1 using the enriched capsule (full clinical context).
   • The LLM acts as a "judge," determining which of the two patients in a pair is higher priority based on clinical context.
3. Bradley-Terry Aggregation: The outcomes of these pairwise comparisons are aggregated using the Bradley-Terry model to estimate a latent severity score ($\theta$) for every patient in the queue. This produces a consistent global ranking even if individual comparisons are non-transitive.
4. Simulation: 1,000 shifts (Site A) and 500 shifts (Site B) were simulated. Time-to-Provider (TTP) was modeled as a function of queue position to evaluate operational impact.

Baselines

The framework was compared against:

• Standard of Care: FIFO, ESI, NEWS2.
• Supervised ML: XGBoost trained on Site A data and applied to Site B without retraining.

3. Key Contributions

• Paradigm Shift: Reframing ED triage from independent risk classification to context-aware relative ranking.
• Zero-Shot Cross-Site Stability: Demonstrated that an LLM-based ranking system, trained on no Site B data, maintained performance across institutions, whereas a supervised XGBoost model suffered significant performance degradation.
• Modular Comparison Function: Showed that the "judge" can be swapped (from heuristics to LLMs) and that enriching the input data (diagnoses/medications) significantly improves ranking quality.
• Bias Mitigation: Preliminary evidence suggesting the LLM approach reduces disparities in prioritization based on arrival mode (EMS vs. self-arrival) compared to ESI.

4. Results

Primary Endpoint: Recall@5 (Fraction of deteriorators ranked in the top 5 positions)

• Development (MC-MED):
  • BT-LLM-Enriched: 0.587 (Significantly outperformed ESI at 0.491; $p<0.001$).
  • XGBoost: 0.648 (Highest performance, but required site-specific training).
• External Validation (MIMIC-IV-ED):
  • BT-LLM-Enriched: 0.481 (Stable; statistically indistinguishable from XGBoost).
  • XGBoost: 0.502 (Attenuated from 0.648 on Site A).
  • ESI: 0.422.

Secondary Endpoints

• AUROC (Global Discrimination):
  • Site A: XGBoost (0.892) > BT-LLM-Enriched (0.826) > ESI (0.811).
  • Site B: BT-LLM-Enriched (0.831) > XGBoost (0.807) > ESI (0.751).
  • Key Finding: The LLM model remained stable across sites, while the supervised model's discrimination dropped significantly.
• Time-to-Provider (TTP):
  • BT-LLM-Enriched significantly reduced median wait times for deteriorators compared to ESI at both sites.
  • On external validation, BT-LLM-Enriched achieved a significantly lower 95th-percentile tail wait (129 min) compared to XGBoost (154 min), suggesting better safety for the highest-risk patients.
• Subgroup Analysis: ESI-based ordering showed a significant disparity where EMS arrivals were under-prioritized compared to self-arrivals. This disparity was not observed under the BT-LLM-Enriched framework.

Computational Cost

• Estimated at $0.01 per patient using GPT-4.1 pricing, totaling roughly $1.50 per day for a 150-patient ED.

5. Significance and Conclusion

This study provides a proof-of-concept that pairwise ranking with LLM judges offers a robust alternative to traditional classification-based triage.

• Operational Impact: It directly addresses the "who goes next" question, improving the identification of high-risk patients at the front of the queue.
• Scalability: The zero-shot capability allows hospitals to deploy advanced prioritization tools immediately without the burden of collecting and labeling local training data.
• Clinical Safety: By utilizing rich, unstructured clinical context (diagnoses/medications) via LLMs, the system captures complex clinical patterns that binary flags or simple scores miss, potentially reducing adverse outcomes and disparities in care.

The authors conclude that while supervised models may achieve peak performance with local tuning, the LLM-based ranking framework offers superior stability and generalizability, making it a promising candidate for real-world clinical decision support systems.