💻 computer science

MoAKE: Toward Unified All-in-One Action Quality Assessment via Mixture of Action Knowledge Experts

This paper introduces MoAKE, a novel Mixture of Action Knowledge Experts framework that unifies Action Quality Assessment for diverse action types within a single model by dynamically aggregating segment-aware expert knowledge and modeling multi-granularity temporal dynamics, thereby overcoming the limitations of traditional one-by-one paradigms and achieving superior performance in all-in-one, zero-shot, and few-shot scenarios.

Original authors: Huangbiao Xu, Huanqi Wu, Xiao Ke, Jiaxin Cai, Junyi Wu, Jinglin Xu

Published 2026-07-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Huangbiao Xu, Huanqi Wu, Xiao Ke, Jiaxin Cai, Junyi Wu, Jinglin Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge at a massive, chaotic talent show where the acts range from Olympic gymnastics to brain surgery and deep-sea diving. In the old days, if you wanted to score a gymnast, you'd hire a gymnastics expert; for a diver, a diving expert; and for a surgeon, a medical professional. You'd need a different judge for every single act because the rules, the movements, and the "good" looks completely different for each one. This is how computers used to judge video actions: they needed a separate, specialized brain for every type of movement. But what if you wanted one single, super-smart judge who could watch any video—whether it's a figure skater spinning or a diver flipping—and give a fair score without needing to know beforehand what the act is? That's the dream of "Action Quality Assessment" (AQA). It's the field of teaching computers to watch videos and say, "That was a 9.5 out of 10," just like a human expert would. The big problem is that mixing these different skills usually confuses the computer, making it worse at everything, like trying to teach a fish to fly and a bird to swim at the same time.

Enter MoAKE, a new framework that acts like a "Mixture of Action Knowledge Experts." Instead of forcing one giant brain to learn everything at once, MoAKE builds a team of specialized experts who work together in a shared office. Imagine a panel of judges where each one is a master of a specific style—say, one loves rhythmic gymnastics, another knows figure skating, and a third understands artistic swimming. When a video comes in, a smart "router" (think of it as a quick-thinking stage manager) looks at the video and decides which experts should weigh in. If a gymnast is on screen, the gymnastics expert speaks up loudly, while the others listen quietly. But here's the magic: they don't just work alone. They share their notes in a common language, helping each other understand the nuances of movement. This allows the system to handle a chaotic mix of different actions without getting confused, turning the "noise" of different sports into a helpful chorus of knowledge.

The researchers found that this approach works incredibly well. When they tested their "all-in-one" model on three long-term sports datasets (rhythmic gymnastics, figure skating, and artistic swimming), it didn't just match the old "one-judge-per-sport" methods; it actually beat them. In fact, the new model improved the accuracy of its rankings by about 4.7% on average and reduced its scoring errors by a massive 34.4% compared to trying to force a single model to learn everything without this expert team. Even more impressively, when they threw the model into the deep end with completely new, unseen actions like diving, skiing, and surgery (a test called "zero-shot"), it didn't crash. It managed to give decent scores without ever having seen those specific videos before, proving that the experts learned general rules of movement that apply across different worlds.

The secret sauce lies in how MoAKE handles the messy details of time. Videos aren't all the same length; a gymnastics routine might be two minutes, while a surgery clip could be ten. MoAKE uses a clever trick called "Segment-Aware Temporal Aggregation" to chop these videos into manageable, standard-sized chunks, like cutting a long movie into short, equal scenes so the experts can compare them fairly. Then, it uses a special module called AIISRM (Adaptive Intra- and Inter-Segment Relationship Modeling) to study how those scenes connect. It looks at what's happening inside a single scene (like a skater's arm position) and how scenes connect to each other (like the flow from a jump to a landing). By modeling these relationships at different levels of detail, the experts can catch tiny mistakes that a simpler model would miss.

The paper explicitly rules out the idea that you can just take an existing computer model and train it on all these different sports at once without any special help. The authors showed that doing this "naively" leads to "negative transfer," where the knowledge from one sport actually hurts the performance on another, causing scores to drop significantly (an average drop of over 10% in some tests). They also argue against the idea that you need a separate model for every single action type, showing that this "one-by-one" approach is too rigid and expensive for real-world use where you might have thousands of different videos.

In short, MoAKE suggests that the best way to judge a chaotic mix of actions isn't to build a bigger, dumber brain, but to build a smarter team of specialists who know how to talk to each other. The results, measured on six different datasets, show that this team approach not only solves the problem of mixing different sports but also creates a system that is surprisingly good at guessing how to judge new, unseen actions. It's a step toward a future where a single AI can walk into any arena, watch any performance, and give a fair, expert-level score, no matter what the act is.

Technical Summary: MoAKE – Toward Unified All-in-One Action Quality Assessment

1. Problem Statement

Action Quality Assessment (AQA) aims to objectively evaluate performance quality from action videos by learning a mapping from motion semantics to expert-provided scores. While recent advancements have improved performance in specific domains (e.g., sports, medical rehabilitation), existing methods predominantly follow a "one-by-one" paradigm. In this setting, a separate model is trained and deployed for each specific action type.

This fragmentation creates significant barriers to real-world deployment:

Dependency on Priors: It requires reliable metadata (known action types) to select the correct model, which is often unavailable in open-domain platforms.
Lack of Generalization: It hinders content-agnostic services that must process heterogeneous or mixed actions without manual intervention.
Resource Intensity: Maintaining high-capacity models for every new or rare action is data- and compute-intensive.

The paper identifies the core challenge of moving to an "All-in-One" AQA paradigm (a single unified model for heterogeneous actions) as the substantial semantic discrepancies between different action types. Naively training a single model on diverse actions often leads to "negative transfer," where conflicting action patterns interfere with each other, causing significant performance degradation (observed as an average Spearman's Rank Correlation Coefficient (SRCC) drop of ~10% in baseline experiments).

2. Methodology: MoAKE Framework

To address negative transfer while leveraging complementary knowledge across domains, the authors propose MoAKE (Mixture of Action Knowledge Experts). The framework is designed to dynamically fuse knowledge from multiple specialized experts within a shared semantic space.

2.1 Overview and Soft Routing

Unlike traditional Mixture-of-Experts (MoE) designs that often rely on token-level sparse routing for parameter scaling, MoAKE operates at the macro action level with a lightweight design.

Input: Visual features $X$ are extracted from video snippets.
Soft Router: A lightweight router $R(\cdot)$ dynamically computes a set of weights $\{w_k\}$ based on the input features. Unlike hard routing, this soft routing mechanism adaptively blends complementary knowledge, transforming the semantic gap into positive transfer.
Feature Fusion: The final mixed feature $F_{mix}$ is a weighted sum of outputs from $K$ distinct experts: $F_{mix} = \sum w_k F_k$ .

2.2 Action Knowledge Experts

Each expert $E_k$ is custom-designed to handle the unique temporal demands of AQA, differing from standard feed-forward MoE experts.

Shared Adapter: Features are projected into a shared semantic space using a shared Multi-Layer Perceptron (MLP) to encourage a "common language" for action quality.
Segment-Aware Temporal Aggregation (SATA): To handle variable-length action videos, each expert employs learnable, segment-aware prototypes ( $P_k$ ). Using a Q-Former style cross-attention mechanism, these prototypes distill variable-length inputs into fixed-length representations ( $N$ tokens), normalizing temporal variations.
Adaptive Intra- and Inter-Segment Relationship Modeling (AIISRM): This novel module captures multi-granularity temporal dynamics through three parallel branches:
- Intra-Segment Global: Models global correlations along the feature dimension (e.g., evolution of velocity or posture) using self-attention.
- Inter-Segment Local: Uses 1D convolutions with kernel sizes (1, 3, 5) to model local temporal correlations between sub-actions, dynamically fused via attention weights.
- Inter-Segment Global: Applies self-attention to relate distant segments, capturing long-range dependencies like overall rhythm and coherence.
- The outputs are aggregated with residual connections to preserve discriminative characteristics.

2.3 Score Prediction and Optimization

Grade-Aware Decoder: The mixed feature is processed by a Transformer decoder using learnable grade queries to generate grade-aware representations, which are mapped to a final score via an MLP.
Loss Function: The model is optimized with a composite loss:
- $\mathcal{L}_{score}$ : MSE loss for the final score and auxiliary scores (to encourage expert specialization).
- $\mathcal{L}_{route}$ : Soft cross-entropy loss with label smoothing to guide the router without enforcing rigid expert isolation.
- $\mathcal{L}_{div}$ : A diversity loss (triplet loss) to ensure grade representations capture distinct quality levels.

3. Benchmarks and Experimental Setup

The authors establish comprehensive benchmarks to evaluate All-in-One and Zero/Few-shot AQA:

All-in-One Training: Trained on three long-term datasets: Rhythmic Gymnastics (RG), Figure Skating (Fis-V), and Artistic Swimming (LOGO).
Zero/Few-Shot Evaluation: Evaluated on three short-term datasets: Diving (MTL-AQA), Skiing/Snowboarding (AQA-7), and Surgery (JIGSAWS).
- Zero-shot: Direct evaluation without parameter updates.
- Few-shot: Full-network fine-tuning on 5-10% of data.
Metrics: Spearman's Rank Correlation (SRCC) and Relative L2-distance (R- $\ell_2$ ).

4. Key Results

Extensive experiments on six datasets demonstrate MoAKE's superiority:

All-in-One Performance: MoAKE significantly outperforms existing state-of-the-art (SOTA) methods (e.g., GDLT, HGCN, T²CR, PAMFN, CoFInAl, ASGTN) in the all-in-one setting. Notably, it achieves an average SRCC of 0.753, surpassing the strong Conditional Baseline (0.718) and even outperforming SOTA models trained in the standard "one-by-one" paradigm (improving average SRCC by 1.2% and reducing R- $\ell_2$ by 28.9%).
Generalization: In Zero-shot scenarios, MoAKE achieves significantly higher SRCC (e.g., 0.319 on Diving vs. 0.207 for the best baseline) and lower error rates, demonstrating robust generalization to unseen action types.
Efficiency: Despite the MoE architecture, MoAKE maintains practical computational costs (8.01M parameters, 1.44G FLOPs), avoiding the massive parameter inflation typical of large-scale MoEs.
Ablation Studies: Removing the soft router causes severe performance drops (30% SRCC decrease), confirming the necessity of adaptive mixing. The AIISRM module and SATA mechanism are also shown to be critical for capturing multi-granularity temporal dynamics.

5. Significance and Claims

The paper claims to pioneer a new direction in Action Quality Assessment by enabling the assessment of multiple action types within a unified framework.

Paradigm Shift: It moves away from the fragmented "one-by-one" approach toward a scalable, content-agnostic "All-in-One" solution, addressing the practical needs of open-domain platforms and low-resource scenarios.
Positive Transfer: The work demonstrates that the challenge of semantic diversity can be transformed into an opportunity for positive knowledge transfer through the Mixture of Action Knowledge Experts, effectively mitigating negative transfer.
Foundational Representation: The proposed framework provides a superior foundational representation for downstream adaptation, enabling strong performance in zero-shot and few-shot settings where previous methods fail.
Community Resource: By establishing the first comprehensive benchmarks for All-in-One and Zero/Few-shot AQA, the authors aim to facilitate future research in this under-explored area.

The authors conclude that while the method shows superior results, the zero-shot scenario remains challenging, and future work may explore self-supervised learning or multimodal integration to further enhance generalization.