💻 computer science

CLOT: Closed Loop Optimal Transport for Unsupervised Action Segmentation

The paper proposes CLOT, a novel unsupervised action segmentation framework that enhances the limitations of prior optimal transport methods by employing a multi-level cyclic feature learning mechanism to jointly refine frame and segment embeddings through cross-attention and iterative optimal transport problems.

Original authors: Elena Bueno-Benito, Mariella Dimiccoli

Published 2026-08-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Elena Bueno-Benito, Mariella Dimiccoli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast archive of digital video, from security footage to home movies, there is a hidden layer of meaning waiting to be unlocked: the ability to automatically understand what is happening, moment by moment. This task, known as action segmentation, involves breaking a continuous stream of video into distinct chunks, each labeled with a specific activity like "pouring water" or "opening a door." For computers to do this without human instruction, they must learn to recognize patterns in the visual data on their own. This is a difficult challenge because a computer sees a video not as a story, but as a sequence of millions of individual images. To make sense of this, researchers often use a mathematical approach called optimal transport. Imagine trying to match a pile of scattered puzzle pieces to a set of completed pictures; optimal transport provides a way to calculate the most efficient path to pair each piece with its correct image, ensuring that the final arrangement makes logical sense. This method has become a powerful tool for teaching machines to group video frames into coherent actions, yet it has struggled to handle the messy reality of short, quick movements or to ensure that the boundaries between actions are perfectly sharp.

A team of researchers from the Institut de Robòtica i Informàtica Industrial in Barcelona has introduced a new system called CLOT, which stands for Closed Loop Optimal Transport, designed to solve these specific weaknesses. While previous methods could group video frames into actions, they often treated the process as a one-way street: the computer would guess the action, and that was the end of the line. CLOT changes this by creating a feedback loop, a cycle where the computer constantly checks its own work and refines its understanding. The system works in three distinct stages. First, it looks at individual video frames and assigns them to potential action groups, creating a rough draft of the timeline. Second, it steps back to look at these groups as larger segments, learning what a complete "action" looks like as a whole unit. Finally, and most importantly, it brings these two levels of understanding together. It uses the knowledge of the larger segments to go back and correct the initial assignments of the individual frames. This back-and-forth process allows the system to smooth out errors, ensuring that a brief, fleeting movement is not missed and that the transition from one action to the next is precise.

The researchers tested this new approach on four different sets of video data, ranging from people preparing breakfast and assembling furniture to cooking salads and following online instructions. In these tests, the videos contained anywhere from a few dozen to several thousand frames, and the actions varied from long, slow processes to very short, sharp movements. The results showed that CLOT consistently outperformed the best existing methods. On the breakfast preparation dataset, the system correctly identified the action in 60.1 percent of the frames, a significant improvement over the previous best. More importantly, it excelled at measuring the quality of the segments as a whole, achieving a score of 40.1 on a metric that balances precision and recall, and improving the accuracy of the boundaries between actions by a wide margin. The system was particularly effective at detecting short-duration actions, which had previously been a major stumbling block for similar technologies. By refining the initial guesses with the help of the segment-level feedback, the model could distinguish between a momentary pause and a genuine change in activity, a distinction that earlier models often missed.

To understand why this works so well, one must look at the specific tools the researchers built into the system. They introduced a mechanism that acts like a filter, allowing the computer to focus on the most informative frames while ignoring the static or noisy parts of the video where nothing is happening. They also replaced a standard way of measuring similarity between frames with a more robust mathematical tool that handles high-dimensional data more effectively, ensuring that the computer compares the visual features in a way that respects the structure of the video. The core innovation, however, remains the cyclic nature of the learning. Instead of stopping after the first guess, the system uses a cross-attention mechanism to let the "segment" view influence the "frame" view. This means that if the system recognizes a pattern of movement that belongs to a specific activity, it can go back and adjust the labels of the individual frames within that activity to match that pattern more closely. This creates a self-correcting loop that tightens the segmentation boundaries and produces a cleaner, more accurate timeline of events.

The study confirms that this multi-level approach is not just a minor tweak but a fundamental improvement in how machines can learn to see time. The researchers found that removing any part of this cycle—whether the segment-level learning, the refinement step, or the specific way they measured distances between frames—led to a noticeable drop in performance. This suggests that the combination of these elements is essential for capturing the true structure of human activity. While the system is not perfect and still faces challenges with very rare or ambiguous actions, it represents a significant step forward in unsupervised learning. It demonstrates that by allowing a computer to review and revise its own understanding of a video, we can achieve a level of detail and accuracy that was previously out of reach. This work opens the door for more sophisticated applications in fields like sports analysis, where understanding the exact sequence of moves is critical, or in robotics, where a machine needs to understand the precise steps of a task to replicate it. The path forward lies in continuing to refine these loops, making the machine's understanding of human action ever more precise and reliable.

Technical Summary: CLOT (Closed Loop Optimal Transport for Unsupervised Action Segmentation)

Problem Definition

Unsupervised action segmentation aims to label every frame in a video with its corresponding action class without relying on labeled training data. While traditional approaches follow a multi-step pipeline of learning frame embeddings and then clustering them, recent Optimal Transport (OT)-based methods have improved by jointly learning representations and generating pseudo-labels. However, existing state-of-the-art methods like ASOT (Adaptive Segmentation via Optimal Transport) face specific limitations:

Lack of Segment-Level Supervision: ASOT generates pseudo-labels but lacks explicit feedback between frame representations and segment boundaries, limiting the effectiveness of feedback loops.
Bias in Segment Length: ASOT enforces strong structural priors that implicitly bias segment length, making it difficult to accurately detect short-duration actions.
Feedback Disconnect: Alternative methods like HVQ (Hierarchical Vector Quantization) improve short-action detection but lack the effective feedback between representation learning and clustering typical of OT formulations, resulting in poorer clustering assignments.

The core challenge addressed by this paper is achieving high-quality, fine-grained segmentation by establishing explicit feedback between frame and segment representations to ensure learned clusters align with true segment boundaries.

Methodology: CLOT Framework

The authors propose CLOT (Closed Loop Optimal Transport), a novel OT-based framework featuring a multi-level cyclic feature learning mechanism. The architecture processes precomputed frame embeddings ( $X$ ) through three distinct stages, solving three separate OT problems to refine embeddings and pseudo-labels iteratively.

1. Architecture Components

Encoder with Feature Dispatching: An MLP-based encoder processes input frames into frame embeddings ( $F$ ). It employs a learnable similarity function to dispatch frames into clusters based on similarity, dynamically updating feature representations to create structured, context-aware embeddings.
Parallel Decoder: Unlike autoregressive decoders that predict segments sequentially (prone to error accumulation), CLOT uses a parallel decoder based on query-based attention. It transforms frame embeddings ( $F$ ) into segment embeddings ( $S$ ) simultaneously, solving a temporally consistent OT problem.
Cross-Attention Refinement: A cross-attention mechanism integrates intrinsic information from segment embeddings ( $S$ ) and cluster centers to refine the frame embeddings ( $F$ ) into a refined set ( $F_R$ ). This step closes the loop between frame and segment representations.

2. Optimal Transport (OT) Formulation

CLOT solves three OT problems in a cyclic manner:

Stage 1 (Frame-to-Cluster): Solves an OT problem between initial frame embeddings ( $F$ ) and action clusters ( $A$ ) to generate initial pseudo-labels ( $T$ ).
Stage 2 (Segment-to-Cluster): Solves an OT problem between segment embeddings ( $S$ ) and action clusters ( $A$ ) to generate segment pseudo-labels ( $T_S$ ).
Stage 3 (Refined Frame-to-Cluster): Solves a third OT problem using the refined frame embeddings ( $F_R$ ) and action clusters to produce refined pseudo-labels ( $T_R$ ).

The OT formulation combines Kantorovich Optimal Transport (KOT) and Gromov-Wasserstein Optimal Transport (GWOT) to ensure temporal consistency. Crucially, the cost matrix for the visual component incorporates the Sliced Wasserstein (SW) distance alongside cosine distance. The SW distance is used to refine the initial cost matrix, leveraging its efficiency and robustness in high-dimensional spaces to capture finer inter-frame relationships.

3. Training Objective

The model is trained in an unsupervised manner. The objective minimizes the cross-entropy loss between predicted action probabilities and the soft pseudo-labels derived from the OT solutions. The total loss is the sum of losses from all three stages:
$\mathcal{L}_{train} = \mathcal{L}(\mathbf{T}, \mathbf{P}) + \mathcal{L}(\mathbf{T}_{S}, \mathbf{P}_{S}) + \mathcal{L}(\mathbf{T}_{R}, \mathbf{P}_{R})$

Key Contributions

Novel Framework: Proposes CLOT, a framework that enforces segment-level consistency through a multi-level cyclic learning mechanism, enabling fine-grained segmentation and generalization across videos.
Closed-Loop Architecture: Introduces an encoder-decoder architecture that closes the loop between frame and segment embeddings. By leveraging cross-attention, it refines frame embeddings using segment-level structure, addressing the feedback disconnect in previous OT methods.
Sliced Wasserstein Integration: Incorporates the Sliced Wasserstein distance into the cost matrix computation to enhance the initial cost matrix, improving visual similarity measurement in high-dimensional spaces.
Comprehensive Evaluation: Provides extensive experiments, ablation studies, and qualitative results across four benchmark datasets.

Experimental Results

The authors evaluated CLOT on four standard datasets: Breakfast, YouTube Instructions (YTI), 50 Salads, and Desktop Assembly. Performance was measured using Mean over Frames (MoF), F1 Score, and Mean Intersection over Union (mIoU) under two matching protocols: Activity-Level (generalization across videos) and Video-Level (per-video optimization).

Activity-Level Performance: CLOT achieved state-of-the-art results, outperforming the previous best method (ASOT) by an average of +2.54% in MoF, +5.08% in F1 Score, and +3.2% in mIoU. Notably, CLOT showed significant improvements in F1 scores on 50 Salads and Desktop Assembly, suggesting more descriptive action embeddings.
Video-Level Performance: CLOT outperformed ASOT across all datasets except YTI. It achieved F1 and mIoU scores comparable to methods specifically designed for video-level segmentation (which often do not generalize across videos), demonstrating its ability to dynamically refine action representations.
Ablation Study: Removing components such as the Sliced Wasserstein distance, Feature Dispatching, the Parallel Decoder, or the Refinement stage consistently degraded performance. Specifically, the Decoder was found essential for short-duration actions, while the Refinement stage was critical for improving F1 and mIoU scores.

Significance and Claims

The paper claims that CLOT represents a significant advancement in unsupervised action segmentation by effectively closing the loop between representation learning and clustering. The authors argue that:

Cyclic Learning is Essential: The multi-level cyclic mechanism allows the model to refine pseudo-labels and embeddings iteratively, ensuring that learned clusters align with true segment boundaries.
Parallel Decoding Prevents Drift: Unlike autoregressive models, the parallel decoder prevents error propagation across sequences, allowing the model to maintain high-quality action boundaries without segmentation drift.
Robustness to Short Actions: By integrating segment-level feedback and avoiding strong structural priors that bias segment length, CLOT improves the detection of short-duration actions, a known weakness in prior OT-based methods like ASOT.

The authors conclude that while CLOT achieves state-of-the-art performance, action mislabeling remains an open challenge, highlighting the need for further improvements in pseudo-label refinement and segment alignment. They position CLOT as a step forward in self-supervised learning and structured prediction tasks.