Actionable Real-Time Modeling of Surgical Team Dynamics via Time-Expanded Interaction Graphs

This paper proposes a real-time surgical AI system that models team dynamics using time-expanded interaction graphs to predict procedural efficiency and generate actionable, counterfactual insights for improving intraoperative communication and coordination.

The Gist

Imagine the operating room (OR) not just as a place where surgery happens, but as a high-stakes orchestra. The surgeons, nurses, and anesthesiologists are the musicians. For the surgery to go smoothly, they need to play in perfect harmony. If the violinist (the surgeon) starts arguing with the drummer (the nurse) about the tempo, or if the whole band stops listening to the conductor, the music falls apart, and the performance drags on too long.

This paper introduces a new "smart conductor" system designed to listen to that orchestra in real-time and suggest how to get back on track.

The Problem: The "Silent" AI

Current AI systems in surgery are like cameras that only watch the hands. They can see if a surgeon is cutting correctly or if a tool is being used, but they are "deaf" to the team's conversation and coordination. They miss the fact that a surgery might be running late because the team is confused, arguing, or not talking to each other effectively.

The authors argue that time is the ultimate scorecard. If a surgery takes longer than expected, it usually means the team's "orchestra" is out of sync. Longer surgeries mean higher risks for the patient and much higher costs for the hospital.

The Solution: The "Time-Expanded" Map

To fix this, the researchers built a new kind of AI model called TE-ReNN. Here is how it works, using a simple analogy:

Imagine you are trying to understand a traffic jam.

• Old AI: Takes a single photo of the traffic. It sees cars are stopped, but it doesn't know why or how the traffic flow changed over the last 10 minutes.
• This New AI: Builds a 3D movie map of the traffic. It doesn't just look at the cars; it connects every car to every other car and connects each car to its own position from one minute ago, two minutes ago, and so on.

In the paper, this is called a Time-Expanded Interaction Graph.

1. The Nodes (The People): Every team member is a dot on the map.
2. The Edges (The Talk): When someone speaks, the AI draws a line connecting them to everyone else in the room (because in an OR, everyone hears everyone).
3. The Time-Lines: The AI connects a person's "dot" from minute 1 to their "dot" in minute 2. This lets the AI see how a person's behavior evolves. Did the surgeon get quieter? Did the nurse start shouting?

By looking at this 3D map, the AI can predict if the surgery is going to run late, even before the delay becomes obvious to the human eye.

The "What If" Magic: Actionable Advice

The coolest part of this paper is that the AI doesn't just say, "You're going to be late." It acts like a flight simulator for behavior.

The researchers asked the AI: "What is the smallest change we could make to the team's behavior to make the surgery faster?" This is called Counterfactual Analysis.

The AI runs thousands of "What If" scenarios:

• What if the surgeon stopped talking to the nurse who isn't involved in this step?
• What if the team leader became more "calm and cooperative" instead of "agitated"?

The AI found that small tweaks matter a lot. For example, if a leader switches from being "agitated" to "calm," or if they stop having side conversations with people not working on the current task, the predicted surgery time drops significantly. The paper claims that changing just 10% of the team's behavior patterns could improve the predicted efficiency by 50%.

How They Tested It

They tested this on a dataset of simulated knee surgeries (like a training video game for surgeons). They had 27 simulated procedures with teams of 4 to 6 people.

They compared their new "Time-Expanded Map" AI against older methods:

• Old AI: Just looked at features (like volume of voice) without connecting people.
• Old AI: Looked at time but ignored who was talking to whom.
• Old AI: Looked at connections but ignored how things changed over time.

The Result: The new "Time-Expanded Map" AI was the clear winner. It was the best at predicting whether a surgery would be slow, medium, or fast.

The Bottom Line

This paper presents a tool that listens to the "music" of the operating room. Instead of just watching the surgery, it maps out how the team talks and moves over time. It can predict delays early and, most importantly, tell the team exactly what small behavioral changes (like speaking less or changing tone) could help them finish the surgery faster and safer.

Note: The paper explicitly states these results are based on simulated procedures and that the "operative time" is a proxy for performance. The authors emphasize that these counterfactual suggestions need validation by real medical experts before being used in actual hospitals.

Technical Summary

Technical Summary: Actionable Real-Time Modeling of Surgical Team Dynamics via Time-Expanded Interaction Graphs

Problem Statement
Surgical team performance is driven by complex interactions between technical execution and non-technical skills, such as communication and coordination. While Surgical Data Science (SDS) has advanced visual workflow modeling and technical skill assessment, intraoperative team dynamics remain under-explored. Existing approaches often lack structured representations of team interactions over time or fail to provide actionable guidance for performance improvement. Furthermore, communication breakdowns and misaligned task allocation can lead to temporal inefficiencies, which correlate with increased postoperative risks and significant operational costs. Current predictive models often struggle in small-data regimes typical of surgical teams, where standard temporal graph neural networks may overfit due to limited data and team size.

Methodology
The authors propose a framework that models surgical team collaboration as a sequence of multimodal interaction graphs constructed over fixed 15-second temporal windows. The core of the approach is the Time-Expanded Relational Neural Network (TE-ReNN), which integrates temporal dynamics directly into the graph topology rather than relying on recurrent architectures.

1. Interaction Modeling:

   • Input: Multimodal time-series data (paralinguistic, motion, human-object interaction) extracted from audio-video recordings.
   • Graph Construction: For each time window $t$, a snapshot graph $G_t$ is created where nodes represent team members and edges represent verbal communication. A "broadcast assumption" is used: when a member speaks, directed edges are added to all other present members, reflecting the shared acoustic environment of the operating room (OR).
   • Node Features: Each node integrates paralinguistic features (loudness, alpha-ratio, harmonics-to-noise ratio from eGeMAPS 2.0), motion features (position, displacement), human-tool interaction triplets, and functional roles.

2. Time-Expanded Graph ($G_{TE}$):

   • The framework constructs a unified graph $G_{TE}$ by connecting snapshot graphs across time.
   • Intra-snapshot edges: Capture communication within a specific time window.
   • Inter-snapshot identity edges: Link consecutive temporal instances of the same team member ($v^t_i \to v^{t+1}_i$) to preserve identity continuity and model longitudinal individual dynamics.
   • This structure allows information to propagate both across team members within a window and along individual trajectories over time.

3. Actionable Counterfactual Analysis:

   • To move beyond prediction, the system generates counterfactual explanations identifying minimal changes required to improve predicted outcomes (specifically, reducing operative time).
   • Topological Counterfactuals: A combinatorial optimization procedure determines which interactions (edges) should be added or removed to shift a procedure from "slow" to "medium" or "fast."
   • Feature-Level Counterfactuals: A greedy algorithm identifies minimal changes in paralinguistic features. These features are mapped to eight discrete behavioral classes (e.g., "Calm Leader," "Dominant–Aggressive," "Withdrawn"). The system simulates switching a team member's behavioral class to observe the impact on predicted efficiency.

Key Contributions

• TE-ReNN Architecture: A novel neural network architecture that simultaneously models relational interactions and temporal evolution by embedding time directly into the graph structure, addressing overfitting risks in small-data surgical settings.
• Two-Level Counterfactual Framework: A method to generate interpretable behavioral suggestions, operating at both the topological level (interaction structure) and the feature level (behavioral classes).
• Experimental Validation: Evaluation on the MM-OR dataset (simulated knee-replacement procedures) demonstrating the model's ability to improve early identification of prolonged interventions and provide coherent explanations.

Results
Experiments were conducted using a leave-one-team-out protocol on 27 simulated surgical procedures (14 complete surgeries).

• Predictive Performance: The TE-ReNN model achieved the highest macro-F1 score (70.1%) compared to baselines including Static Neural Networks (SNN), Temporal Neural Networks (TNN), Relational Neural Networks (ReNN), and combined Temporal-Relational networks (TReNN). This confirms that jointly modeling behavioral features, temporal dependencies, and relational graph structures yields superior performance compared to treating these components separately.
• Counterfactual Insights:
  • Topological: Analysis suggested that slowdowns often occur when surgeons acting as "democratic leaders" engage in conversations with members not directly involved in the current task, whereas "autocratic" focus on tasks correlates with faster procedures.
  • Feature-Level: The model identified that modifying only 10% of a team member's behavioral class over a time window could result in an approximate 50% increase in predicted performance. "Calm Leader" behavior was notably associated with slowdowns in specific contexts, while other behavioral shifts showed potential for efficiency gains.

Significance and Claims
The paper claims to advance Surgical AI toward real-time, team-aware, and actionable decision support. By framing operative time as a proxy for team performance, the work provides a system that does not merely predict outcomes but offers specific, interpretable recommendations on how team communication and behavior can be modified to achieve desired results. The authors emphasize that this approach bridges the gap between predictive accuracy and actionable feedback in high-stakes environments.

Limitations
The authors acknowledge that operative time is an approximation of team performance rather than a direct measure of care quality. Additionally, the practical utility of the generated counterfactual explanations requires further validation by domain experts, and the current scope is limited to specific modalities and simulated procedures.