Fully Automatic Data Labeling for Ultrasound Screen Detection

This paper presents a fully automatic pipeline that generates labeled data to train a screen detector, enabling the extraction and rectification of ultrasound images from monitor photographs without human annotation, thereby bypassing the DICOM bottleneck while maintaining sufficient visual fidelity for downstream classification tasks.

The Gist

Imagine you are a doctor using an ultrasound machine to look at a patient's heart. The machine shows the moving images on a built-in screen, just like a tablet or a TV.

Usually, to save these images or send them to a computer for analysis, the machine has to go through a complicated digital "handshake" called DICOM. Think of DICOM as a very strict, high-security vault. If you want to move the data out, you need the right keys, cables, and specific software settings. It's like trying to get a secret message out of a bank vault; it works, but it's slow, clunky, and requires a specialist to set it up.

The Problem:
The researchers wanted a faster way. They asked: "What if we just took a picture of the screen with a regular phone camera?"
It sounds simple, but it's actually a nightmare for computers.

1. The Angle: You rarely take a photo perfectly straight on. The screen looks like a trapezoid, not a rectangle.
2. The Glare: Screens reflect light. Your phone might capture a reflection of the ceiling lights or your own face, confusing the computer.
3. The Labeling: To teach a computer to find the screen in a photo, humans usually have to draw boxes around the corners of the screen in thousands of photos. This is boring, expensive, and slow work.

The Solution: The "Magic Mirror" Pipeline
The team from Ultromics Ltd. built a fully automatic system that acts like a digital magic trick. Here is how they did it, using simple analogies:

1. The "Fake Reality" Factory (Synthetic Data)

Instead of hiring humans to draw boxes on thousands of real photos, the team built a video game engine.

• They took thousands of random photos of living rooms and offices (backgrounds).
• They took thousands of real ultrasound heart scans.
• They programmed a robot to "paste" the heart scan onto the living room wall, but they made it look realistic: they added fake reflections (like a glare from a window) and tilted the angle so it looked like someone took a photo with a phone.
• The Magic: Because the robot created the image, it already knows exactly where the corners of the screen are. It doesn't need a human to label it. It's like a video game that knows the coordinates of every object because it placed them there.

2. The "Screen Detective" (The AI Model)

They trained a smart computer program (an AI) using this "fake reality" data.

• The Job: The AI looks at a photo and has two jobs:
  1. Detect: "Is there an ultrasound screen in this picture?"
  2. Locate: "Where are the four corners of that screen?"
• The Training: They fed the AI millions of these "fake" photos. The AI learned to ignore the background (the sofa, the wall) and focus on the glowing rectangle of the ultrasound, even if it was covered in fake glare.

3. The "Digital Straightener" (Geometric Correction)

Once the AI finds the screen in a real photo, it uses a mathematical trick called Homography.

• The Analogy: Imagine you have a piece of paper with a drawing on it, and you hold it at an angle. The drawing looks stretched and squashed. If you could magically grab the four corners and pull them until the paper is flat and square again, the drawing would look normal.
• The AI does exactly this. It grabs the four corners it found, stretches the image, and "flattens" it back into a perfect, straight ultrasound image, removing the perspective distortion.

4. The "Clean-Up Crew" (Post-Processing)

The flattened image might still look a bit weird (maybe too dark or with weird colors). The system runs a quick filter to:

• Turn it black and white (grayscale).
• Make the background pitch black (so the heart stands out).
• Brighten the image to standard levels.

The Results: Does it work?

They tested this system in two ways:

1. On Fake Data: It was incredibly accurate. The AI could find the corners with an error smaller than a single pixel (sub-pixel accuracy).
2. On Real Data: They took photos of real ultrasound screens with a tablet. The system successfully found the screen and straightened the image.
3. The Final Test: They fed these "straightened" photos into a standard heart-view classifier (a program that says "This is a view of the left ventricle").
   • The Result: The system worked surprisingly well! It achieved about 79% accuracy compared to the original, perfect digital files.

Why This Matters

This is a game-changer because:

• No Cables Needed: You don't need to plug the ultrasound machine into a computer. You just point a phone camera at the screen.
• Instant Prototyping: Doctors and researchers can test new AI algorithms on data captured in seconds, without waiting for IT departments to set up complex data transfers.
• Mobile Friendly: It opens the door for using ultrasound data in Augmented Reality (AR) glasses or mobile apps in the field, even in remote areas without hospital networks.

In short: They built a system that turns a messy, angled photo of a screen into a clean, digital ultrasound image, all without a single human having to draw a box or plug in a cable. It's like having a robot that can look at a photo of a TV and instantly "cut out" the movie playing on it, straighten it, and save it perfectly.

Technical Summary

1. Problem Statement

Ultrasound (US) machines typically display images on built-in monitors for real-time guidance, but routine data transfer to hospital systems relies on the DICOM format. This creates a bottleneck for rapid algorithm testing and prototyping, especially in mobile or augmented reality (AR) applications where direct cable connections (e.g., HDMI) are cumbersome or require proprietary protocols.

The core challenge addressed is capturing US screen content via a standard video camera (e.g., a handheld device) and automatically:

1. Detecting and localizing the screen within a photograph.
2. Extracting and rectifying the image to correct for perspective distortion and reflections.
3. Eliminating the need for human annotation, which is traditionally required to train such detection models.

2. Methodology

The authors propose a three-stage pipeline: Synthetic Data Generation, Multi-Task Model Training, and Geometric Correction.

A. Synthetic Data Generation (Self-Annotation)

To avoid manual labeling, the authors created a fully automated synthetic dataset:

• Datasets Used:
  • Backgrounds: MIT Indoors dataset (67 categories).
  • Content: Anonymized US imaging data from 1,000 adult patients.
• Synthesis Process:
  1. Reflection Simulation: Using screen blending, synthetic reflections are added to ultrasound images ($S$) using random background images ($R$). The blending formula is: $B = Y \cdot (1-\alpha) + S \cdot \alpha$, where $Y = 1 - (1-S)*(1-R)$.
  2. Perspective Warping: A random rectangle is generated in the background image, and its corners are displaced to simulate camera perspective. The blended screen is inserted into the background using a perspective transform defined by these four points.
  3. Augmentation: The process is repeated with two different backgrounds per image to force the model to focus on the echo content rather than the background. Images without screens are also included to train the classifier branch.
• Dataset Size: ~47,582 training samples, split into training, validation, and test sets.

B. Screen Detection Model

The authors adapted a Multi-task UNet architecture (based on Treivase et al. and [2]) with two branches:

1. Localization Branch: Predicts four corner heatmaps, followed by a Differentiable Soft-Argmax (DSNT) layer to output precise 2D coordinates of the screen corners.
2. Classification Branch: Predicts the binary presence of a screen in the image.

• Loss Function: A multi-task loss combining Euclidean distance for corner localization ($L_s$) and cross-entropy for classification ($L_c$). The losses are balanced using learnable uncertainty parameters ($\sigma_s, \sigma_c$):
  $$L = \frac{L_s}{\sigma_s^2} + \frac{L_c}{\sigma_c^2} + \ln(\sigma_s + 1) + \ln(\sigma_c + 1)$$

C. Geometric Correction & Post-Processing

Once the four corners are detected:

1. Homography Transformation: A perspective transform is applied to map the detected screen to a target grid (640x480 pixels), correcting the distortion.
2. Normalization: The output is converted to grayscale, quantized to 256 levels, and normalized (background set to black, max value to 255) to match standard US data encoding (uint8).

3. Key Contributions

1. Self-Annotated Synthetic Data: A novel strategy to generate fully labeled training data for screen detection without human intervention, addressing the "data labeling burden."
2. Multi-Task CNN Architecture: Implementation of a UNet-based model with a DSNT layer for precise corner localization and a classification branch for screen presence, trained with uncertainty-weighted loss.
3. End-to-End Pipeline: A complete workflow from raw screen capture to rectified, analysis-ready ultrasound images.
4. Downstream Validation: Demonstration that the reconstructed images are viable for training secondary models (specifically, an echo view classifier).

4. Results

A. Screen Detection & Localization

• Synthetic Data: Performance improved monotonically with data volume.
  • With 1,000 samples, localization error dropped to sub-pixel levels (median 0.99 px).
  • With 47,582 samples, sensitivity reached 0.991 and specificity 0.998.
• Real Data: The model generalized well to real-world photos (100 test images).
  • Localization error stabilized around 4.2 pixels (approx. <1% of image size).
  • Sensitivity reached 0.962 and specificity 1.0 with sufficient training data.

B. Image Quality Assessment

• Synthetic vs. Real: Reconstructed synthetic images showed high similarity to originals (SSIM = 0.57). Real-world reconstructions had lower similarity (SSIM = 0.1) due to unmodeled degradation factors (e.g., complex reflections, screen bezels).
• Visual Fidelity: Despite lower SSIM scores, visual inspection confirmed that anatomical structures remained recognizable.

C. Downstream Task (Echo View Classification)

The reconstructed images were fed into a view classifier trained on standard DICOM images:

• Initial Performance: Balanced accuracy was 0.65 (synthetic) and 0.47 (real).
• Uncertainty Filtering: By removing the 20% most uncertain samples (based on prediction probability), the balanced accuracy for real data improved to 0.56, and for synthetic data to 0.72.
• Final Proof-of-Concept: With further optimization (removing 40% uncertain samples in synthetic), the system achieved a balanced accuracy of 0.79 relative to native DICOMs, proving the pipeline's utility for rapid prototyping.

5. Significance and Conclusion

This work presents a significant step toward democratizing ultrasound data access. By removing the dependency on DICOM and proprietary cables, it enables:

• Rapid Prototyping: Researchers can test new algorithms on data captured via simple cameras.
• Mobile/AR Integration: Seamless data forwarding to mobile devices for real-time analysis.
• Scalability: The self-annotating synthetic data approach solves the critical bottleneck of manual labeling for screen detection tasks.

Limitations & Future Work: The authors note a performance drop on real-world data compared to synthetic data, hypothesizing causes include ambiguity in manual labeling of real test sets, black screen frames causing confusion, and unmodeled image degradations. Future work will investigate these factors to improve robustness.