Technical Summary: AI-Assisted Competency Assessment from Egocentric Video in Simulation-Based Nursing Education

1. Problem Statement

Competency assessment in clinical simulation, particularly in nursing education, traditionally relies on expert human observation. This approach is time-intensive, difficult to scale for growing cohorts, and subject to inter-rater variability. While video provides a rich record of learner performance, existing automated approaches often assume data-rich, fixed-camera environments (e.g., surgical laparoscopy) and focus on isolated technical skills rather than holistic competency.

This study addresses the challenge of assessing holistic nursing competency using egocentric (first-person) video from simulation-based learning. The specific domain is pediatric medication administration, where competency is inherently sequential; errors can arise from correct actions performed in the wrong order or with omitted safety steps. The research investigates whether visual evidence alone, analyzed through frozen visual encoders and few-shot learning, can support competency assessment, and whether the difficulty of recognizing these actions carries pedagogical signals.

2. Methodology

The authors propose a three-stage framework applied to 22 densely annotated egocentric video sessions (totaling 3.8 hours and 493 clinical action segments) from nursing students performing standardized simulations on high-fidelity mannequins.

Stage 1: Few-Shot Action Recognition

• Input: Egocentric video frames ($V_i$) and a support set ($S$) of labeled exemplars from other sessions (leave-one-out cross-sample setting).
• Feature Extraction: A frozen backbone encoder ($\phi$) extracts per-frame features. Three backbones were evaluated: ResNet-50 (ImageNet-supervised), DINOv2 ViT-B/14 (ImageNet self-supervised), and CLIP ViT-B/16. No fine-tuning was performed.
• Prototype Computation: Class prototypes are computed from the support sessions using a prototypical network paradigm. The study compares a mean prototype strategy (averaging normalized centroids) against a clustered prototype strategy (k-means sub-centroids).
• Temporal Smoothing: Continuous cosine similarity scores between query frames and prototypes are passed to a Hidden Markov Model (HMM) with Viterbi decoding. This enforces temporally coherent label sequences by learning transition probabilities from the support set, rather than classifying frames independently.
• Output: A predicted action timeline ($\hat{y}_i$) for 16 clinical action classes plus a background class.

Stage 2: Sequence Analysis

• Sequence Construction: Predicted frame-level labels are collapsed into ordered action segments (run-length encoding), discarding background and short segments.
• Feature Derivation:
  1. Action Transition Frequencies: Pairwise flow between clinical actions to construct process models.
  2. Recognition Metrics: Per-video metrics (Mean-over-Frames accuracy [MOF], mean Intersection-over-Union [mIoU], and Segmental F1) are retained as summary measures of classifier fit.

Stage 3: Competency Analysis

• Correlation: Given the small sample size ($N=22$), Spearman rank correlation is used to map sequence features and recognition metrics to instructor-rated competency scores (derived from the Creighton Competency Evaluation Instrument [C-CEI] rubric, specifically 11 video-observable items).
• Robustness: Partial associations are computed controlling for six potential confounders (annotation coverage, video duration, segment count, etc.).
• Process Modeling: Ground-truth action sequences are used to visualize and compare workflow graphs between higher- and lower-performing groups.

3. Key Contributions

1. Feasibility of Few-Shot Clinical Action Recognition: The study demonstrates that a frozen DINOv2 backbone with HMM Viterbi decoding achieves 57.4% MOF in leave-one-out 1-shot recognition of egocentric nursing simulation video. This establishes feasibility under extremely low-data conditions without fine-tuning. DINOv2 consistently outperformed ResNet-50 and CLIP, suggesting self-supervised vision transformers are better suited for fine-grained hand-object interactions in this domain.
2. Negative Correlation Between Recognition Accuracy and Competency: A primary finding is a statistically significant negative trend between recognition accuracy and instructor-rated competency ($\rho = -0.524, p = 0.012$ for mIoU). This relationship is robust to six confound controls. The authors interpret this as higher-competency students producing more diverse, fluid, and protocol-consistent workflows that are inherently harder for a prototype-based classifier to match, whereas lower-competency students exhibit more repetitive, templated behaviors that are easier to classify.
3. Identification of Pedagogically Informative Signals: Per-item analysis reveals that the negative trend is most strongly reflected in expected behaviors related to patient safety protocols (e.g., "Uses patient identifiers") and team communication. Simple sequence features (e.g., screen time ratio) showed no significant relationship with competency, suggesting that the difficulty of classification itself is the informative signal.
4. Temporal Workflow Differences: Process model comparisons show that higher-performing students exhibit more diverse action transitions and adhere more strictly to protocol-consistent progressions (e.g., direct transitions in medication preparation), while lower-performing students show more hesitation (e.g., routing through "Screen" checks unnecessarily) and irregular workflow paths.

4. Results

• Action Recognition: In the cross-sample (leave-one-out) setting, performance was lower than within-sample, highlighting the domain shift caused by individual differences in appearance, camera angle, and workflow ordering. However, performance plateaued around 10–15 shots, indicating that the bottleneck is generalization, not representation capacity.
• Competency Correlation:
  • mIoU: $\rho = -0.524$ ($p=0.012$).
  • MOF: $\rho = -0.439$ ($p=0.041$).
  • F1: $\rho = -0.433$ ($p=0.044$).
  • Lower-performing students (median split) had 9.5% higher MOF and 8.3% higher mIoU than higher-performing students.
• Per-Item Analysis: The strongest negative correlations were found for "Communicates effectively with team" ($\rho = -0.470$) and "Uses patient identifiers" ($\rho = -0.455$). Procedural items performed in a static, repetitive manner showed no association.
• Process Models: Higher-performing groups showed a higher frequency of direct "Prep Med $\to$ Apply Med" transitions (46% vs. 38% for low performers) and more consistent "Hygiene $\to$ Screen" transitions (76%), indicating better infection control and workflow fluidity.

5. Significance and Claims

The paper claims that recognition accuracy may serve as a complementary pedagogical signal in automated competency assessment. Contrary to the assumption that higher accuracy always indicates better performance, the study suggests that in clinical simulation, lower recognition accuracy can indicate higher competency because expert learners exhibit diverse, adaptive workflows that deviate from rigid templates.

The authors propose a two-tier assessment perspective:

1. Predicted Action Timelines: Provide a coarse behavioral summary of what the student did.
2. Recognition Difficulty (e.g., mIoU/F1): Serves as a complementary signal of how the student executed the task, potentially capturing holistic competency, procedural flexibility, and safety adherence that checklist-based instruments might miss.

The study emphasizes that while vision-based analysis can capture observable behaviors and workflow diversity, it is fundamentally limited in capturing clinical reasoning and the content of verbal communication. The authors remain modest about their claims, noting that with $N=22$, the findings are exploratory and require further validation. They suggest future work should focus on protocol-aware monitoring using formal methods and personalized competency trajectory mining.