🤖 AI

D-CLOT: Double Closed Loop Optimal Transport for Unsupervised Action Segmentation

D-CLOT addresses the representation-prototype inconsistency in unsupervised action segmentation by introducing a double closed-loop framework that iteratively refines frame embeddings via graph constraints and re-estimates action prototypes, achieving state-of-the-art performance across multiple benchmarks including the new Assembly101 dataset.

Original authors: Elena Bueno-Benito, Mariella Dimiccoli

Published 2026-08-07

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Elena Bueno-Benito, Mariella Dimiccoli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a long, unedited video of someone baking a cake. The camera never stops rolling; it captures the flour sifting, the eggs cracking, the mixer whirring, and the oven timer beeping all in one continuous stream. Your goal is to act like a super-smart editor who can watch this raw footage and automatically cut it into perfect, labeled scenes: "Mixing," "Baking," "Cooling." This is the challenge of temporal action segmentation. While computers are great at recognizing what an object is (like a cat or a car), figuring out exactly when one action stops and another begins in a messy, real-world video is much harder.

To solve this without needing a human to write down every single second of the video (which is incredibly expensive and slow), scientists use a mathematical tool called Optimal Transport. Think of this like a logistics problem: you have a pile of "frame" packages (the video moments) and a set of "action" warehouses (the labels like "mixing" or "baking"). The computer tries to find the most efficient way to ship every frame to the correct warehouse. In the best scenarios, the computer learns to guess the labels, uses those guesses to improve its understanding of the video, and then uses that better understanding to guess the labels again, creating a helpful feedback loop. This is the foundation of a method called CLOT, which has been quite successful at untangling these video puzzles.

However, the researchers behind this new paper noticed a subtle glitch in that feedback loop. It's like a dance where the music (the action labels) keeps changing, but the dancers (the video frames) are trying to follow a map that was drawn yesterday. As the computer gets better at understanding the video, the "map" it uses to define the actions doesn't update fast enough to match the new, sharper picture. This mismatch causes the computer to get confused, especially during tricky moments like quick transitions or very short actions. The paper introduces a new method called D-CLOT (Double Closed Loop Optimal Transport) to fix this. By adding a "stabilizer" that keeps the video frames from getting too jumpy and a "re-calibration" step that constantly updates the action map to match the current dance, D-CLOT helps the computer see the video much more clearly. The results show that this approach significantly improves the accuracy of cutting these videos into the right scenes, even on very difficult, fine-grained tasks like assembling toys.

The Problem: A Map That Doesn't Match the Territory

Imagine you are navigating a city using a map. At first, the map is a bit blurry. As you walk around, you start to see the streets more clearly, and you update your mental picture of the city. But what if the paper map in your hand never updates? You are walking through a city that has changed, but your map still shows the old layout. You might try to turn left where a new building now stands, or you might get confused because the street names on your map don't match the signs you see.

This is exactly what was happening with the previous best method, CLOT. CLOT was great at refining the "mental picture" of the video frames, making them sharper and more distinct. It would take a blurry video and, through a clever process of guessing and correcting, make the frames look like they belonged to specific actions. However, the "map" it used to define those actions—the action prototypes (the mathematical definitions of what "mixing" or "baking" looks like)—wasn't being updated fast enough.

The authors of this paper identified this as a representation–prototype inconsistency. As the video frames became clearer and more organized, the definitions of the actions stayed stuck in their old, less precise state. This was particularly damaging during ambiguous transitions (when one action fades into another) and for short or infrequent actions (like a quick "snap" of a finger or a rare "add salt" step). In these cases, the old, rigid definitions would get overwhelmed by the dominant actions, causing the computer to miss the small details or merge two different actions into one.

The Solution: A Double Loop of Correction

To fix this, the team built D-CLOT, which adds a "double closed loop" to the system. Think of it as adding two new safety checks to the dance floor.

1. The Graph-Constrained Stabilizer (The Safety Net)
First, the system needed to make sure the video frames didn't get too wild when they were being refined. Sometimes, in the process of trying to group similar frames together, the computer might accidentally link two frames that are far apart in time just because they look similar. This breaks the natural flow of the video.

D-CLOT introduces a graph-constrained module. Imagine this as a safety net that keeps the dancers close to their original neighbors. It ensures that if two video frames were next to each other in the raw footage, they stay close to each other in the computer's "refined" version. This preserves the local neighborhood structure, preventing the computer from creating fake connections between unrelated moments. It stabilizes the geometry of the video, making sure the "territory" is reliable before we try to update the "map."

2. The Action-Embedding Refinement (The Map Update)
Once the video frames are stabilized, the system needs to update the action definitions to match this new, clearer picture. The previous method only updated these definitions slowly, through a process called gradient descent, which is like trying to nudge a heavy boulder with your hands. It's slow and can get stuck.

D-CLOT introduces a periodic re-anchoring step. Every so often, the system stops, looks at the stabilized video frames, and completely recalculates what the action prototypes should be. The authors tested two ways to do this:

D-CLOT (The K-Means Refresh): This method uses a standard clustering technique (k-means) to find the center of the new frame groups and moves the action definitions there. It's a quick, assignment-agnostic refresh.
D-CLOTB (The Barycentric Update): This is the more sophisticated version. It doesn't just look at the center of the groups; it calculates the optimal transport barycenter. Imagine you have a pile of sand (the video frames) and you want to find the perfect balance point. This method weighs every frame based on how confident the computer is that it belongs to that action. If the computer is very sure a frame is "mixing," it pulls the "mixing" definition strongly. If it's unsure, it pulls less. This creates an assignment-aware update that perfectly matches the current state of the video.

The Results: Sharper Cuts and Better Understanding

The team tested D-CLOT on five different datasets, ranging from cooking videos (like making breakfast or salads) to instructional videos and even a new, very difficult dataset called Assembly101, which involves assembling toys.

The results were impressive. By fixing the mismatch between the video frames and the action definitions, D-CLOT significantly improved the quality of the segmentation:

On the YouTube Instructions (YTI) dataset, the new method improved the F1 score (a measure of how well the segments match the ground truth) by +12.7 points and the mIoU (mean Intersection over Union) by +10.2 points compared to the previous best.
On the 50Salads dataset, it saw gains of +8.9 F1 points in the activity-level evaluation.
Most notably, the team established the first unsupervised baseline on Assembly101. This dataset is much harder than the others, with videos averaging over 13,000 frames and containing 11 to 42 different fine-grained actions per toy category. Even here, D-CLOT outperformed the previous methods, showing that the "double loop" approach works even when the actions are tiny and the video is long and noisy.

The authors found that the two components—the graph stabilizer and the prototype re-anchoring—worked best together. The stabilizer kept the video frames from getting chaotic, and the re-anchoring ensured the action definitions stayed in sync with the video. The D-CLOTB variant, with its assignment-aware barycentric update, tended to be the most robust, especially on datasets with complex, imbalanced action durations.

Why This Matters

This paper suggests that for computers to truly understand untrimmed videos without human help, they need to constantly update their internal "dictionary" of actions to match the clarity of the video they are seeing. It's not enough to just refine the video; you have to refine the definitions of what you are looking for, too.

By introducing this double-loop mechanism, the researchers have shown that unsupervised action segmentation can be made much more reliable, especially for the tricky, short, and rare actions that often trip up AI. They didn't just tweak the numbers; they fixed a fundamental structural issue in how these systems learn. While there is still room for improvement, especially on the very fine-grained Assembly101 dataset, D-CLOT sets a new standard for how machines can learn to watch and understand the world around them, one frame at a time.

Technical Summary of "D-CLOT: Double Closed Loop Optimal Transport for Unsupervised Action Segmentation"

Problem Statement
Unsupervised temporal action segmentation aims to assign action labels to frames in untrimmed videos without manual annotations. Recent state-of-the-art approaches, such as CLOT, utilize Optimal Transport (OT) to align frame representations with latent action prototypes, generating pseudo-labels for self-training. However, the authors identify a critical "representation–prototype inconsistency" in existing OT-based methods. While frame and segment embeddings are iteratively refined through the OT process, the latent action prototypes ( $A$ ) that define the transport costs are not re-estimated from this refined geometry. Instead, they are initialized once (typically via k-means) and evolve solely through gradient descent on the classification loss.

This inconsistency creates a bottleneck, particularly around ambiguous temporal transitions where refined embeddings of different actions overlap, and for short or infrequent actions whose transport mass is easily absorbed by dominant modes. Furthermore, because refinement stages optimize purely for OT-induced classification loss, they may introduce associations between frames that were not neighbors in the original encoder geometry, leading to representation drift.

Methodology
The proposed method, D-CLOT (Double Closed Loop Optimal Transport), extends the CLOT backbone by introducing two successive operations to close the loop between representation refinement and prototype estimation, without modifying the original CLOT training objective or architecture.

Graph-Constrained Regularization:
To stabilize the refined frame ( $F_R$ ) and segment representations, the authors introduce a graph-constrained module. This module enforces that the local neighborhood structure of the refined representations remains consistent with the local geometry of the initial encoder output ( $F_{enc}$ ).
- A graph affinity matrix is computed based on cosine similarity between frame embeddings.
- A cross-entropy loss ( $L_G$ ) is minimized between the affinity distribution of the refined representation and the fixed anchor ( $F_{enc}$ ).
- This prevents the OT refinement stages from forming spurious inter-action connections absent in the original feature space, ensuring the geometry remains reliable for prototype updates.
Action-Embedding Refinement:
Periodically (at the epoch level), the latent action prototypes ( $A$ ) are re-anchored to the stabilized representation geometry. To prevent abrupt shifts, the new prototype proposal ( $\hat{A}$ ) is blended with the current prototypes using a normalized convex combination: $A \leftarrow \text{norm}((1-\beta)A + \beta\hat{A})$ .
The paper investigates two instantiations of this refinement step:
- D-CLOT (k-means refinement): The proposal $\hat{A}$ is generated by applying k-means clustering to the pooled, graph-regularized refined frame features. This is an "assignment-agnostic" update, relying entirely on the graph penalty to maintain geometric consistency.
- D-CLOTB (Barycentric refinement): This variant enforces the graph constraint through an explicit auxiliary variable optimization (solving an inner problem to find a graph-stabilized representation $\tilde{F}_R$ ). The prototypes are then updated as OT barycenters of these stabilized features, weighted by the soft mass of the refined transport plan. This is an "assignment-aware" update, where each frame contributes to a prototype in proportion to its OT confidence, making the update consistent with the current transport geometry.

Key Contributions

Identification of Inconsistency: The paper identifies that the lack of prototype re-estimation from refined geometries in CLOT leads to representation–prototype inconsistency, degrading performance on ambiguous transitions and rare actions.
Graph-Constrained Regularization: Introduction of a module that stabilizes OT-refined representations by preserving the local neighborhood structure of the encoder output.
Prototype Refinement Loop: A mechanism to periodically re-estimate latent prototypes from refined embeddings, effectively closing the representation–prototype loop.
Two Instantiations: The proposal of D-CLOT (k-means based) and D-CLOTB (OT barycenter based), demonstrating that assignment-aware updates (D-CLOTB) offer robustness, particularly for imbalanced action distributions.
New Benchmark: The establishment of the first unsupervised action-segmentation baseline on Assembly101, a large-scale, fine-grained, procedural dataset (15 toy categories, 11–42 actions per activity), which is significantly more challenging than existing cooking or instructional datasets.

Experimental Results
The authors evaluated D-CLOT and D-CLOTB on five benchmarks: Breakfast (BF), YouTube Instructions (YTI), 50Salads (FS), Desktop Assembly (DA), and the new Assembly101 subset.

Performance Gains: Both variants improved segment-level quality over the CLOT baseline across all datasets.
- On YTI, D-CLOT achieved per-video gains of +12.7 F1 and +10.2 mIoU.
- On FS-Eval, D-CLOTB achieved an activity-level gain of +8.9 F1.
- On Assembly101, both variants substantially outperformed CLOT in F1 and mIoU, establishing a new state-of-the-art baseline for this difficult dataset.
Ablation Studies: Experiments confirmed that graph regularization and prototype refinement are complementary. Graph regularization alone improves temporal coherence, while prototype refinement corrects assignment mismatches. The full model (both components) consistently outperformed partial configurations.
Sensitivity Analysis: The method was shown to be robust to hyperparameter variations (graph weights, bandwidth, blend factor), with D-CLOTB generally exhibiting smoother sensitivity curves than D-CLOT.

Significance and Claims
The paper claims that D-CLOT addresses a fundamental limitation in unsupervised action segmentation: the decoupling of representation refinement from prototype evolution. By jointly regularizing the representation geometry and periodically re-anchoring prototypes, the method achieves state-of-the-art performance, particularly for short and infrequent actions where previous methods struggle.

The authors emphasize that their approach is general and does not require modifying the underlying CLOT backbone or training objective. The introduction of the Assembly101 benchmark highlights the scalability of the approach to fine-grained, procedural tasks with high variability. The results suggest that continuously adapting latent prototypes to an evolving representation space is a crucial component for scalable unsupervised temporal action segmentation.

The Problem: A Map That Doesn't Match the Territory

The Solution: A Double Loop of Correction

The Results: Sharper Cuts and Better Understanding

Why This Matters

More like this