💻 computer science

HOI-aware Adaptive Network for Weakly-supervised Action Segmentation

This paper proposes AdaAct, a weakly-supervised action segmentation network that leverages video-level human-object interaction priors through a HyperNetwork to dynamically adapt its temporal encoder parameters, thereby effectively resolving ambiguities between similar actions.

Original authors: Runzhong Zhang, Suchen Wang, Yueqi Duan, Yansong Tang, Yue Zhang, Yap-Peng Tan

Published 2026-04-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Runzhong Zhang, Suchen Wang, Yueqi Duan, Yansong Tang, Yue Zhang, Yap-Peng Tan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to watch a cooking show and write down exactly what steps the chef is taking, second by second. The robot knows the list of ingredients and steps (like "pour juice," "cut orange," "squeeze"), but it doesn't know when each step starts or ends. This is the challenge of Weakly-Supervised Action Segmentation.

The problem is that many cooking steps look almost identical. Pouring coffee looks a lot like pouring orange juice. If the robot only looks at the few seconds right in front of it, it might get confused and say, "Oh, that's coffee!" when the chef is actually making juice.

The paper introduces a new system called AdaAct to solve this confusion. Here is how it works, using simple analogies:

1. The "Context Clue" Detective

Most old methods are like a detective who only looks at the crime scene right in front of them. They see a hand holding a cup and pouring liquid. Without more info, they guess.

AdaAct is like a detective who looks at the entire video before making a guess. It asks: "What objects are in the video?"

If the video contains a knife, an orange, and a squeezer, the robot knows, "Ah, this is definitely juice!"
If the video contains a coffee pot and beans, it knows, "This is coffee!"

The paper calls these objects and their interactions HOI (Human-Object Interactions). Instead of just watching the action, the system scans the whole video to find these "clues" (like the orange or the coffee pot) and uses them to guide its understanding.

2. The "Shape-Shifting" Brain

Here is the clever part. Usually, a computer program has a "fixed brain." Once it's trained, its settings don't change. It's like a chef who always cooks the same way, regardless of the ingredients.

AdaAct has a Shape-Shifting Brain (technically called an "Adaptive Network").

Think of it like a smart chef who changes their cooking style based on the ingredients in the kitchen.
When the system sees the "orange" clue, it instantly tweaks its internal settings to become an expert at recognizing "juice-making."
When it sees the "coffee pot" clue, it instantly shifts its settings to become an expert at "coffee-making."

It does this by using a special "instruction manual" (a HyperNetwork) that rewrites the robot's own rules on the fly, based on the clues it found in the video.

3. How It Learns (The Two-Step Process)

The system learns in two ways, like a student studying for a test:

General Knowledge (HOI-Independent): It learns general rules about cooking videos that apply to everything (e.g., "cooking videos usually have a lot of cutting").
Specific Clues (HOI-Dependent): It looks at the specific video it is watching right now to find unique clues (e.g., "This specific video has a squeezer").

It mixes these two types of knowledge together to make the final decision.

The Results

The researchers tested this on two popular cooking video datasets: Breakfast (making morning meals) and 50Salads (making salads).

The Problem Solved: The system became much better at telling the difference between similar actions, like pouring coffee vs. pouring juice.
The Outcome: It achieved the best results ever recorded for this specific type of task. It didn't just guess; it used the "clues" in the video to know exactly what was happening.

In short: AdaAct is a video watcher that doesn't just look at the action; it looks at the tools being used to figure out what the action is, and it changes its own brain settings to match the tools it sees. This stops it from getting confused when two actions look the same.

1. Problem Statement

The paper addresses the challenge of Weakly-Supervised Action Segmentation (WSAS), where the goal is to predict the action label for every frame in a video using only a transcript (an ordered list of actions) as supervision, without precise start/end timestamps.

Key Challenges Identified:

Ambiguity in Similar Actions: Existing methods often rely on fixed temporal encoders that analyze local neighborhoods of frames. This fails to distinguish between semantically similar actions (e.g., "pouring coffee" vs. "pouring juice") because the visual features of the immediate frames are highly similar.
Lack of Global Context: Current approaches treat video clips as fixed-length inputs, missing the broader context provided by the entire video sequence.
Static Networks: Most methods use a fixed network architecture for all videos, preventing the model from adapting to the specific semantic context (e.g., the specific objects present) of a given video at test time.

2. Methodology: AdaAct

The authors propose AdaAct, an adaptive network that leverages Human-Object Interactions (HOI) as video-level prior knowledge to dynamically adjust network parameters during inference. The architecture consists of two main components:

A. Video HOI Encoder

This module extracts, selects, and integrates HOI information from the entire video to create a global context vector. It operates in three stages:

Extracting: The video is down-sampled (15 FPS), and a pre-trained HOI detector (frozen weights) identifies bounding boxes for hands and objects, along with interaction confidence scores.
Selecting (Video-NMS): A novel Video-NMS algorithm filters the detected interactions. Unlike standard NMS which only uses Intersection over Union (IoU), this adds a temporal constraint to remove duplicate high-score detections across time. The top- $K$ interactions are selected.
Integrating: The selected $K$ interactions are processed by a Transformer-based network (ViT). A learnable token is appended to the sequence of HOI embeddings, and the output state of this token serves as the HOI-dependent knowledge ( $s$ ).

B. Adaptive Temporal Encoder (HyperNetwork)

Instead of using a fixed temporal encoder, AdaAct employs a two-branch HyperNetwork to generate the weights and biases of the temporal encoder (a GRU-based backbone) dynamically based on the input video's HOI context.

HOI-Independent Branch ( $z$ ): Learns general characteristics of instructional videos. It maintains a learnable embedding list that is iteratively updated during training. This branch captures transferable knowledge common across all videos.
HOI-Dependent Branch ( $s$ ): Takes the encoded HOI feature vector (from the Video HOI Encoder) as input. It adapts the encoder specifically to the unique object interactions present in the current video.
Fusion: The weights ( $W$ ) and biases ( $b$ ) for the temporal encoder are generated by fusing the outputs of both branches (element-wise multiplication for weights). This allows the network to "on-the-fly" adjust its parameters to distinguish similar actions based on the specific objects detected (e.g., seeing a "squeezer" and "orange" biases the model toward "pouring juice" rather than "coffee").

C. Training and Decoding

Loss Function: The model is trained using a Constrained Discriminative Forward Loss, which maximizes the energy difference between valid and invalid segmentations.
Decoding: During inference, the Viterbi algorithm is used to find the optimal action sequence and segment lengths based on the frame-wise probabilities generated by the adaptive encoder.

3. Key Contributions

First Adaptive Temporal Encoder for WSAS: The paper introduces the first framework where the temporal encoder's parameters are dynamically adapted to the input video's content during test time, rather than using a static model.
HOI as Global Prior: It proposes exploiting temporally global but spatially local HOI information to resolve ambiguity. By identifying key objects (e.g., a knife vs. a coffee pot) across the whole video, the model gains crucial context to distinguish similar actions.
State-of-the-Art Performance: The method achieves superior results on two challenging datasets (Breakfast and 50Salads) for both action segmentation and action alignment tasks.

4. Experimental Results

The method was evaluated on the Breakfast and 50Salads datasets using four metrics: Mean over Frame (MoF), MoF without Background (MoF-BG), Intersection over Union (IoU), and Intersection over Detection (IoD).

Breakfast Dataset: AdaAct achieved 51.2% MoF and 48.3% MoF-BG, outperforming the previous state-of-the-art (CDFL/TASL) by 1.4% and 1.2% respectively.
50Salads Dataset: AdaAct achieved 55.6% MoF and 50.3% MoF-BG, improving upon the baseline by 0.9% and 0.5%.
Action Alignment: Even when transcripts are provided during inference (a stronger supervision setting), AdaAct still outperformed existing methods, showing a +1.8% MoF gain on 50Salads.
Ablation Studies:
- Removing HOI-dependent knowledge dropped MoF by 3.7%.
- Removing HOI-independent knowledge dropped MoF by 0.9%.
- The optimal knowledge dimension was found to be 128.
Qualitative Analysis: The model successfully corrected errors in "pouring" actions (coffee vs. juice vs. milk) by leveraging the presence of specific objects (e.g., a squeezer) detected in the HOI encoder, which fixed methods missed.

5. Significance

This work represents a significant shift in weakly-supervised action segmentation by moving from static feature extraction to dynamic, context-aware parameter adaptation.

Solving Ambiguity: It effectively solves the "similar action" problem by injecting video-level semantic context (HOI) directly into the network's decision-making process.
Efficiency: It achieves these gains without requiring expensive frame-level annotations, relying only on transcripts and pre-trained HOI detectors.
Generalizability: The "extract-select-integrate" pipeline for HOI and the HyperNetwork approach for parameter adaptation offer a new paradigm for handling temporal ambiguity in video understanding tasks.