💻 computer science

Point Tracking as a Temporal Cue for Robust Myocardial Segmentation in Echocardiography Videos

The paper introduces Point-Seg, a transformer-based framework that leverages point tracking as a temporal cue to achieve robust and temporally consistent myocardial segmentation in echocardiography videos, outperforming state-of-the-art methods in low-quality data while providing essential pixel-level motion information for downstream clinical tasks.

Original authors: Bahar Khodabakhshian, Nima Hashemi, Armin Saadat, Zahra Gholami, In-Chang Hwang, Samira Sojoudi, Christina Luong, Purang Abolmaesumi, Teresa Tsang

Published 2026-01-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bahar Khodabakhshian, Nima Hashemi, Armin Saadat, Zahra Gholami, In-Chang Hwang, Samira Sojoudi, Christina Luong, Purang Abolmaesumi, Teresa Tsang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to draw a perfect outline of a moving heart muscle (the myocardium) on a series of ultrasound video frames. This is a tough job because ultrasound images are often grainy, fuzzy, and the heart is constantly squishing and stretching.

Here is the problem with how computers usually do this:

The "Snapshot" Approach: Most current AI models look at each video frame as a completely separate photo. They don't remember what the heart looked like in the previous second. This is like trying to draw a running dog by looking at a single photo of it every time; you might draw the dog in a weird position in the next frame because you forgot how it was moving.
The "Memory" Approach: Some smarter models try to remember previous frames to stay consistent. But if the first few frames are blurry or noisy (which happens a lot in real-world medical scans), the AI gets confused. It starts to "drift," like a drunk person walking in a straight line but slowly veering off course because they stumbled at the start. By the end of the video, the outline is way off.

The Solution: "Point-Seg" (The Dot-Tracker)

The authors of this paper propose a new method called Point-Seg. Instead of just guessing where the heart muscle is in each frame, they teach the AI to play a game of "connect the dots" across time.

Here is how it works, using simple analogies:

1. The "Sticky Dots" Strategy

Imagine you put hundreds of tiny, invisible sticky dots all over the heart muscle in the first frame of the video. As the heart beats and moves, the AI's job is to watch those specific dots and track exactly where they go in every single subsequent frame.

The Innovation: The AI was trained on a synthetic (fake) dataset first. Think of this as a flight simulator for the AI. It practiced tracking dots on perfect, computer-generated heart videos so it learned the rules of how heart tissue moves, even before seeing real, messy patient data.

2. The "Motion Map"

Once the AI knows where these dots are moving, it uses that information as a "motion map."

The Metaphor: If you are trying to paint a wall that is shaking, it's hard. But if you have a laser pointer showing you exactly where the wall is at every moment, you can paint a straight line easily.
In this paper, the "laser pointer" is the tracked point trajectories. The AI uses the movement of these dots to guide the outline of the heart muscle. This prevents the "drift" problem because the AI isn't guessing; it's following the physical movement of the tissue.

3. Smoothing the Jitters

The researchers also added a "temporal smoothing loss."

The Analogy: Imagine you are walking on a bumpy path. If you look at your feet every second, your path looks jagged. But if you look at your path as a whole, you smooth out the bumps. This mathematical rule tells the AI: "If the heart muscle moved smoothly from frame 1 to frame 2, the outline shouldn't suddenly jump or wiggle wildly in frame 3."

What Did They Find?

The team tested their "Point-Seg" system against the best existing methods using two types of data:

High-Quality Videos: When the images were clear, their system performed just as well as the top competitors.
Low-Quality Videos: This is where they shined. When the images were grainy or noisy (like what you often get in real emergency rooms), the old methods started to fail or drift. Point-Seg stayed stable. It kept the outline accurate because it was following the "sticky dots" rather than trying to guess based on a blurry picture.

Why Does This Matter?

The paper highlights two main wins:

Reliability: It works better when the image quality is poor, which is common in real-world medical settings.
Bonus Data: Unlike other methods that just give you a shape, this method also gives you the exact path of every single point on the heart muscle. The authors note this is crucial for future tasks like measuring how much the heart muscle is stretching (strain) or detecting weak spots in the wall, because you literally have the motion data built-in.

In short: Instead of trying to guess the shape of a moving heart frame-by-frame, Point-Seg puts invisible trackers on the heart and follows them, ensuring the outline stays true even when the video gets messy.

Technical Summary: Point Tracking as a Temporal Cue for Robust Myocardial Segmentation in Echocardiography Videos

Problem Statement
Myocardium segmentation in echocardiography (echo) videos is critical for assessing systolic function, computing myocardial strain, and detecting regional wall motion abnormalities. However, this task is hindered by low image contrast, noise, and significant anatomical variability, particularly in Point-of-Care Ultrasound (POCUS) settings where non-specialist operators often acquire lower-quality clips.

Existing deep learning approaches face specific limitations:

Frame-independent models (e.g., UNet, SwinUNet) ignore temporal information, leading to inconsistent segmentation across the cardiac cycle.
Memory-based models (e.g., MemSAM, MedSAM2) propagate features over time but are susceptible to error accumulation when early frames are noisy or off-axis.
Optical flow-based methods rely on brightness constancy and smooth motion assumptions that are frequently violated in echo due to speckle decorrelation and out-of-plane motion, resulting in temporally unstable or anatomically implausible deformations.
Foundation models (e.g., SAM) often require interactive prompting, which is impractical for real-time, label-scarce workflows.

Methodology: Point-Seg
The authors propose Point-Seg, a transformer-based segmentation framework that integrates point tracking as an explicit temporal cue to ensure stable and consistent myocardium segmentation without relying on memory-based feature accumulation.

Architecture: The model consists of an encoder (ResNet50 backbone with deformable attention transformer layers), a point decoder, fusion layers, and a mask decoder.
Point Tracking Module: The system tracks $N$ distinct points (including a dense grid and optional manual points) across $T$ frames. It utilizes a point decoder trained on the SynUS synthetic dataset to estimate trajectories ( $L_i$ ) and visibility sequences ( $V_i$ ). The decoder employs TAPTR (Transformer-based Point Tracking) fine-tuned for echocardiography.
Fusion Mechanism: The framework uses multiple attention mechanisms to integrate spatial and motion-aware information:
- Point-to-Patch Cross-Attention: Allows point tokens to attend to patch tokens, incorporating spatial features.
- Point Self-Attention: Enables tracked points to exchange information within a frame.
- Point Temporal Self-Attention: Models motion by allowing point tokens to attend to their own trajectories across frames.
- Patch-to-Point Cross-Attention: Integrates motion cues into spatial representations for the mask decoder.
Training Strategy: Training is performed in two stages. First, the point decoder is trained using an $\ell_1$ loss for trajectory estimation and cross-entropy loss for visibility prediction. Second, the point decoder is frozen, and the mask decoder is trained using Dice loss and a temporal smoothing loss. The temporal smoothing loss enforces pairwise similarity between tracked mask values across frames to enhance consistency.
Inference: The model generates segmentation masks and pixel-level motion information simultaneously, requiring no manual prompts.

Key Contributions

Echo-Tuned Point Tracking: The authors adapt and fine-tune TAPTR using the SynUS synthetic dataset to generate robust, densely sampled myocardial trajectories that remain stable even when image quality fluctuates.
Tracking-Conditioned Segmentation: A novel transformer segmentation head ingests tracked trajectories as motion priors. This approach enforces temporal consistency without memory propagation, effectively reducing drift in low-quality echo clips.
Pixel-Level Motion Information: Unlike standard segmentation methods, Point-Seg outputs explicit myocardial motion information, which is essential for downstream tasks like strain measurement and wall motion abnormality detection.

Experimental Results
The method was evaluated on the public CAMUS dataset and a private dataset containing 1,305 training, 181 validation, and 367 test videos.

High-Quality Data: On high-quality CAMUS frames, Point-Seg achieves statistical parity (p ≥ 0.05) with state-of-the-art models (e.g., nnUNet, MedSAM2 with bounding box prompts) in terms of mean Dice coefficient (mDice) and Hausdorff distance (HD95).
Low-Quality Data: In poor-quality echo clips, Point-Seg demonstrates superior performance. On the CAMUS test set, it achieves a significantly higher mDice (86.7 vs. 84.0 for nnUNet) and lower HD95 compared to baselines. On the private dataset, it also outperforms baselines in poor-quality scenarios.
Temporal Stability: Qualitative analysis (Figure 2) shows that Point-Seg maintains stable boundaries throughout the cardiac cycle, avoiding the flickering seen in frame-independent models and the drift observed in memory-based approaches.
Ablation Studies: Experiments confirm that the inclusion of point trajectories, point-to-patch cross-attention, and specifically temporal self-attention significantly improves segmentation accuracy and stability. The proposed tracker also outperforms existing echo-specific trackers (EchoTracker) and general trackers (CoTracker) on the SynUS dataset.

Significance and Claims
The paper claims that Point-Seg offers a reliable and generalizable approach for myocardium segmentation in echocardiography, particularly in challenging POCUS environments. By leveraging point tracking as a temporal cue, the method eliminates the need for memory-based feature accumulation and manual prompting while providing the additional benefit of pixel-level motion data. The authors assert that this approach targets the core needs of bedside heart failure diagnosis: trustworthy, temporally consistent segmentation that enables actionable measurements like ejection fraction and strain, even when image quality is suboptimal. The code is made available to facilitate further research.