Respiratory Status Detection with Video Transformers

1. Problem Statement

Respiratory distress is a critical clinical indicator often detected through subtle visual signs (e.g., nasal flaring, skin retraction, accessory muscle use). While experienced clinicians can identify these signs, they require significant training and continuous monitoring, which is operationally expensive.

• The Challenge: Automating this detection via computer vision is difficult because the visual changes in breathing mechanics are subtle and easily confounded by patient motion, body positioning, and variable lighting.
• The Goal: To determine if modern Video Transformer architectures can be trained to detect these subtle, temporally evolving changes in respiratory status without manual video curation or clinician-guided framing.

2. Methodology

A. Dataset Construction

• Cohort: 52 healthy volunteers (out of 75 recruited) who performed strenuous exercise followed by a 5-minute recovery period.
• Data Collection: Participants recorded their own recovery videos using laptops or mobile phones. No manual cropping or clinician guidance was used during recording to ensure scalability.
• Labeling Strategy: Since ground-truth respiratory distress labels were unavailable at the clip level, the authors created a temporal ordering challenge:
  • Videos were split into 6-second clips.
  • Proxy Labels: Clips from the first third of the recovery were labeled "Shortness of Breath" (SOB), and clips from the last third were labeled "No SOB." Middle clips were excluded to avoid ambiguity.
  • Task: The model must predict which of two clips from the same recovery sequence occurred earlier (higher distress) versus later (closer to baseline).

B. Model Architecture & Augmentations

The baseline model was a Factorized ViViT-Base (Video Vision Transformer). The authors tested specific architectural modifications to handle spatiotemporal data:

1. Comparison Strategies:

   • Two-Tower Cross-Attention: Processing two clips through separate towers and using cross-attention to compare them (tested at full layer and CLS-token levels).
   • Embedding-Based Comparison (Winner): A single encoder maps each clip to a fixed-length embedding vector. The model learns two class prototypes (SOB and No SOB) and minimizes the cosine distance between clip embeddings and their respective prototypes.

2. Positional Encodings:

   • Replaced standard Absolute Positional Embeddings (APE) with Lie Relative Encodings (LieRE).
   • Rationale: LieRE generalizes rotary encodings to multi-axis coordinate systems (x, y, time), providing richer relative position representations that reduce overfitting to specific token locations.

3. Input Pre-processing:

   • Implemented Motion-Guided Masking (MGM).
   • Mechanism: Optical flow is calculated for each patch. Only the top 20% most dynamic patches are retained; the rest are masked (blacked out).
   • Rationale: This forces the model to focus on regions with meaningful motion (respiratory mechanics) while ignoring static background or irrelevant lighting changes.

C. Training Setup

• Hardware: Single NVIDIA A100 GPU.
• Constraints: Max 5 epochs, <100 hours training time per model.
• Input Specs: 6-second clips, 32 frames, 224×224 resolution, 16×16 patch tokens.

3. Key Results

The study evaluated performance using Accuracy and F1 scores on a held-out test set.

• Best Performance: The combination of Embedding Distance, LieRE, and Motion-Guided Masking achieved an F1 score of 0.81 and 67.5% accuracy.
• Comparison Efficiency: The embedding-based approach significantly outperformed the two-tower cross-attention methods while requiring substantially less training time.
• Sensitivity Analysis: Model accuracy correlated with the magnitude of respiratory change. When comparing clips with large temporal gaps (large differences in distress), accuracy approached 80%. For clips with small differences (early recovery stages), accuracy dropped to 55–65%, reflecting the increasing subtlety of visual cues as the patient recovers.

4. Key Contributions

1. Novel Task Formulation: Introduced a pairwise temporal ordering task to train AI on respiratory distress without requiring explicit, granular clinical labels for every video frame.
2. Architectural Innovation: Demonstrated that combining Lie Relative Encodings (LieRE) with Motion-Guided Masking (MGM) significantly enhances the ability of Video Transformers to detect subtle, non-rigid motion patterns in medical contexts.
3. Scalable Data Pipeline: Developed a fully automated video processing pipeline that does not rely on manual cropping or clinician-guided framing, addressing a major bottleneck in real-world deployment.
4. Performance Benchmark: Established that modern video transformers can achieve high performance (F1 0.81) in distinguishing respiratory states, surpassing previous computer vision attempts that relied on 3D CNNs or manual supervision.

5. Significance and Limitations

Significance:
The study proves that AI can learn to recognize the "subtle signs" of respiratory deterioration that human clinicians look for. This suggests a pathway for automated, continuous monitoring systems in hospitals or home care settings that could provide early warnings of respiratory failure, potentially saving lives by enabling earlier intervention.

Limitations:

• Participant Supervision: While automated, the data collection still relied on participants positioning their cameras to keep their faces/upper bodies in view. It does not yet represent a truly passive, uncurated monitoring environment.
• Confounding Variables: The model may be inadvertently learning to detect perspiration (a byproduct of exercise) rather than pure respiratory mechanics.
• Cohort: The study used healthy volunteers. Performance on patients with actual pathological dyspnea or complex comorbidities remains to be validated.

Future Work:
The authors propose validating these methods on real-world patient datasets collected during standard care to confirm the model's ability to detect pathologic dyspnea in uncurated, clinical environments.