💻 computer science

Deep kernel video approximation for unsupervised action segmentation

This paper proposes a novel unsupervised action segmentation method that approximates per-video frame distributions in a deep kernel space using Neural Tangent Kernels and Maximum Mean Discrepancy to achieve competitive performance without requiring large datasets or known segment counts.

Original authors: Silvia L. Pintea, Jouke Dijkstra

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Silvia L. Pintea, Jouke Dijkstra

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very long, chaotic home video of someone making breakfast. It's full of chopping, frying, flipping, and pouring. Your goal is to figure out exactly where one action ends and the next begins (e.g., "chopping" stops, "frying" starts) without having any labels telling you what's happening.

This is the problem of Unsupervised Action Segmentation. Usually, to solve this, computers need to watch thousands of videos to learn the patterns. But what if you only have one video, and you can't save it or share it because of privacy rules (like in a hospital)? You have to figure it out just by looking at that single clip.

This paper proposes a clever new way to do that. Here is the breakdown using simple analogies:

1. The Problem: The "Too Much Data" Dilemma

Think of a video as a massive library containing millions of pages (frames). Trying to analyze every single page to find the story is slow and requires a huge memory.

Old methods tried to group similar pages together (clustering) or find boundaries by guessing. But they often got lost because they didn't understand the "shape" or "geometry" of the story. They were like trying to sort a messy pile of clothes by just looking at the color, ignoring the fact that a shirt and a pair of pants might be the same color but totally different.

2. The Solution: The "Perfect Summary"

Instead of trying to analyze the whole library, the authors suggest creating a tiny, perfect summary of the video.

Imagine you have a 2-hour movie. You want to create a 10-second "highlight reel" that captures the essence of the whole movie so perfectly that if you watched the highlight reel, you'd feel like you understood the whole story.
The computer learns to create this "highlight reel" (which they call a Video Approximation). This reel consists of just a few key frames (synthetic images) that represent the different actions.

3. The Secret Sauce: The "MMD" Compass

How does the computer know if its "highlight reel" is good? It needs a ruler to measure the distance between the real video and its summary.

Old Rulers: Some methods used "Optimal Transport," which is like trying to move a pile of sand from one spot to another with the least amount of effort. It's accurate but incredibly slow and computationally heavy (like moving sand grain by grain).
The New Ruler (MMD): The authors use something called Maximum Mean Discrepancy (MMD). Think of this as a "vibe check." Instead of moving sand, MMD asks: "Do these two groups of data feel the same?" It checks if the summary and the real video have the same "shape" and "feel" in a mathematical space. It's much faster and more reliable.

4. The Brain: "Neural Tangent Kernels" (NTK)

To make this "vibe check" work, the computer needs a very smart way to compare images.

Usually, you'd train a deep neural network (a complex brain) to learn this. But training a brain requires lots of data, which we don't have.
The Trick: They use NTKs. Imagine a "super-brain" that is infinitely wide. You don't need to train it; its structure is already perfect for describing complex shapes. It's like having a master artist who can instantly sketch the "essence" of a face without needing to practice on thousands of faces first. This allows the computer to understand the video's structure without needing a massive training dataset.

5. How It Works in Practice

Create the Summary: The computer generates a small set of fake frames (the summary).
The Vibe Check: It uses the MMD ruler to see how close the summary is to the real video.
Adjust: If the summary isn't close enough, it tweaks the fake frames until the "vibe" matches perfectly.
Segment: Once the summary is perfect, the computer looks at the real video. If a real frame looks most like "Summary Frame A," it labels that moment as "Action A." If it looks like "Summary Frame B," it's "Action B."

Why Is This Special?

No Privacy Issues: It works on one video at a time. You don't need to upload your data to a cloud server.
Handles Unknowns: Sometimes you don't know how many actions are in the video. Old methods (like hierarchical clustering) force the video into a rigid number of buckets, often cutting actions in half. This new method is flexible; if the summary learns that a certain action is rare or repeated, it adapts naturally.
Better Results: In tests, this method was better at finding the exact start and stop points of actions (especially when the number of actions wasn't known beforehand) compared to previous state-of-the-art methods.

The Catch (Limitations)

Like any tool, it's not perfect.

If the actions are too similar (like tightening four different screws that look identical), the computer might get confused and think they are all the same action.
If the video has a lot of "background noise" (like a messy kitchen counter that looks like a different action), it might struggle to separate the signal from the noise.

In a nutshell: The authors built a system that creates a tiny, perfect "highlight reel" of a video to understand what's happening, using a fast mathematical ruler to ensure the summary is accurate. It's a privacy-friendly way to teach computers to understand human actions without needing a massive library of training videos.

1. Problem Statement

The paper addresses the challenge of per-video unsupervised action segmentation.

Context: In applications like healthcare and assisted living, privacy concerns often prevent the storage of large datasets. Consequently, models cannot rely on training across massive video corpora; instead, they must segment a single video without prior training data or annotations.
Task: The goal is to segment a video into mutually exclusive action segments (actions) without knowing the number of actions beforehand.
Limitations of Prior Work: Existing unsupervised methods often fail to model the underlying geometry of the video distribution. They rely on clustering or optimal transport (OT), which can be computationally expensive, hard to optimize, or unstable when the distribution support deforms. Furthermore, many methods assume a fixed number of segments or rely on cluster centers that do not fully capture the data distribution.

2. Methodology

The authors propose a novel approach that learns a small synthetic video approximation ( $\{e_{v_m}\}$ ) to represent the original video ( $\{v_i\}$ ) in a deep kernel space. The core idea is to minimize the distance between the distribution of the real video frames ( $P$ ) and the distribution of the synthetic approximation ( $Q$ ).

A. Metric: Maximum Mean Discrepancy (MMD)

Instead of using Optimal Transport (OT), the authors use MMD as the geometry-preserving metric.

Advantages: MMD is easier to optimize, faster ( $O(n^2)$ vs. $O(n^3 \log n)$ for OT), and less susceptible to the curse of dimensionality.
Objective: Minimize the squared MMD between distributions $P$ and $Q$ :
$\text{MMD}^2(P, Q, \mathcal{H}) = \|\mu_P - \mu_Q\|_{\mathcal{H}}^2$
where $\mu_P$ and $\mu_Q$ are the kernel mean embeddings in the Reproducing Kernel Hilbert Space (RKHS).

B. Kernel Choice: Infinite Neural Tangent Kernels (NTKs)

To define the kernel space, the authors avoid fixed kernels (like Gaussian) due to limited descriptive power and avoid learning the kernel jointly with the inputs (which leads to trivial solutions).

Solution: They use Infinite Neural Tangent Kernels (NTKs) derived from fully connected networks. NTKs combine the non-parametric flexibility of kernel methods with the descriptive power of deep neural networks.
Kernel Construction: Since standard NTKs are not "characteristic" (meaning MMD=0 does not strictly imply $P=Q$ $P = Q$ ), they combine the NTK with a Gaussian kernel:
$k(v_i, v_j) = \alpha \cdot k_{\text{NTK}}(v_i, v_j) \cdot k_{\text{Gauss}}(v_i, v_j)$
- The Gaussian kernel ensures the metric is characteristic.
- The NTK captures deep feature representations.
- The length-scale $\lambda$ for the Gaussian kernel is fixed to the median of squared distances to avoid trivial solutions.

C. Optimization and Segmentation

Learning the Approximation: The method optimizes a small set of synthetic frames $\{e_{v_m}\}_{m=1}^M$ (where $M \ll N$ ) to minimize the MMD loss against the real video frames.
Segmentation: Once the approximation is learned, real frames are assigned to action segments by finding the synthetic frame with the maximum kernel similarity:
$m^* = \arg \max_m k(v_i, e_{v_m})$
Handling Unknown Segment Counts: Unlike hierarchical clustering which forces a specific tree structure, this method allows some synthetic frames to remain unassigned if they do not align well with any real frame. This naturally handles cases where the true number of segments is unknown or differs from the initialized $M$ .
Smoothing: To prevent over-segmentation, input video frames are smoothed using a Gaussian filter over time before optimization.

3. Key Contributions

Deep Kernel Formulation: First work to propose learning a video approximation in a deep kernel formulation specifically for unsupervised action segmentation.
Geometry-Preserving Metric: Introduction of MMD with Infinite NTKs as a superior alternative to Optimal Transport for this task, offering better computational efficiency and optimization stability.
Robustness to Unknown Segment Counts: The method outperforms prior agglomerative clustering works (like TW-FINCH) when the number of segments is not known or deviates from the ground truth, as it does not strictly enforce a fixed number of clusters.
Privacy-Preserving: The approach operates on a per-video basis without requiring large training datasets, addressing privacy constraints in sensitive domains.

4. Experimental Results

The method was evaluated on six standard benchmarks: Breakfast, 50 Salads, YTI, Desktop Assembly, Hollywood Extended, and MPII Cooking 2.

Performance: The method achieves competitive results against state-of-the-art per-video methods (TW-FINCH, ABD, ASOT, CLOT).
- It outperforms ASOT on the 50 Salads dataset.
- It shows competitive F1 scores on YTI and Breakfast.
Unknown Segment Counts: In experiments where the number of segments was randomized (mimicking real-world scenarios where ground truth is unknown), the proposed method consistently achieved higher F1 scores than TW-FINCH. This confirms the hypothesis that the method better respects the underlying data distribution without forcing arbitrary boundaries.
Kernel Analysis: Experiments confirmed that the combination of Gaussian and NTK kernels yields the best segmentation accuracy compared to using NTK or Gaussian kernels alone.
Smoothing: Input smoothing was found to be critical, especially for long videos (e.g., 50 Salads), significantly improving Mean Frame Accuracy (MoF).

5. Limitations and Significance

Limitations:

Similar Actions: The method struggles to segment actions that are visually very similar (e.g., "tighten_screw_1" vs. "tighten_screw_2" in Desktop Assembly), as it learns a single synthetic frame per action.
Heterogeneous Classes: It fails when distinct actions are grouped under a single label (e.g., background noise in MPII Cooking 2).
Hyperparameter Sensitivity: Performance is sensitive to the temporal smoothing hyperparameter.

Significance:
This work provides a robust, privacy-compliant solution for action segmentation in data-constrained environments. By leveraging deep kernel spaces and MMD, it moves beyond simple clustering to model the intrinsic geometry of video data. The ability to handle unknown segment counts without hierarchical constraints makes it particularly suitable for real-world deployment where action counts vary and ground truth is unavailable. The code is open-source, facilitating further research in unsupervised video understanding.