🤖 AI

SkillMoV: Mixture-of-View Routing with Prototype-Conditioned Gating for Unified Multi-View Proficiency Estimation

SkillMoV is a unified, parameter-efficient framework for multi-view proficiency estimation that leverages a Mixture-of-View Projector with prototype-conditioned gating to adaptively aggregate synchronized camera features, achieving state-of-the-art performance across diverse skill domains while training only a fraction of its parameters.

Original authors: Edoardo Bianchi, Antonio Liotta

Published 2026-06-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Edoardo Bianchi, Antonio Liotta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching a Robot to Judge Skills

Imagine you are trying to teach a computer to judge how good someone is at a skill, like playing basketball, cooking, or climbing a rock wall. You don't just want the computer to know what they are doing (e.g., "they are shooting a ball"); you want it to know how well they are doing it (e.g., "they are a beginner" vs. "they are a pro").

This is hard because "good" looks different depending on where you are standing. If you are the player (first-person view), you see your hands but not your feet. If you are watching from the side (third-person view), you see the whole body but maybe miss the details of the grip.

Most existing computer programs are like specialized coaches: one coach only knows basketball, another only knows cooking. Or, they try to use one single coach to watch everyone, forcing every camera angle to look at the action through the same pair of glasses. This often misses the subtle clues that make a pro look like a pro.

SkillMoV is a new, smart system designed to be a unified, multi-angle judge that can handle many different skills and camera angles at once, without needing a separate coach for every single sport.

How SkillMoV Works: The "Expert Panel" Analogy

The core of SkillMoV is a clever trick called Mixture-of-View (MoV) Routing. Here is how it works, step-by-step:

1. The Camera Crew (The Inputs)

Imagine a sports event where you have four cameras filming the same athlete from different angles.

Camera 1 sees the athlete's face.
Camera 2 sees their feet.
Camera 3 sees their hands.
Camera 4 sees the whole body.

2. The Expert Panel (The "Mixture of Experts")

Instead of having one brain try to process all four cameras at once, SkillMoV has a panel of 12 different "expert" mini-brains (called MLPs).

In a normal system, all cameras would be forced to talk to the same brain.
In SkillMoV, the system acts like a smart traffic controller. It looks at the footage from Camera 1 and asks, "Which expert is best at analyzing this angle?" Maybe it sends Camera 1 to Expert #3. Then it looks at Camera 2 and asks, "Who is best for this angle?" Maybe it sends Camera 2 to Expert #7.

The Magic: The experts are shared. The same pool of 12 experts helps judge basketball, cooking, and dancing. But the system learns to send the right camera angle to the right expert automatically, without anyone telling it which camera is which.

3. The Team Huddle (Cross-View Attention)

After the experts do their job, the system brings the results together. It's like a team huddle where the experts compare notes. "Hey, Camera 1 saw the footwork, and Camera 3 saw the hand grip. When we combine those, it looks like a pro move." This step ensures the different angles agree with each other before making a final decision.

4. The Reference Cards (Prototype Anchoring)

To decide if someone is a "Novice" or an "Expert," the system uses Reference Cards.

Imagine four cards on a table: one for "Novice," one for "Early Expert," one for "Intermediate," and one for "Late Expert."
The system doesn't just guess; it compares the athlete's performance against these four mental cards. It asks, "Does this performance look more like the 'Novice' card or the 'Expert' card?"
Interestingly, the paper found these cards aren't perfect, rigid rulers. They are more like flexible magnets that pull the decision in the right direction, helping the system make a smarter guess even if the performance is messy.

5. The Final Verdict (Gated Projection)

Finally, the system uses a smart gate. It looks at the evidence and the reference cards, then decides: "Based on what I see and how it matches the 'Expert' card, I will let this specific detail count more than that one." This fine-tunes the final score.

Why Is This Better? (The Results)

The researchers tested SkillMoV on a huge dataset called EgoExo4D, which contains videos of people doing six different skills (Basketball, Rock Climbing, Cooking, Dancing, Music, and Soccer) filmed from multiple angles.

The "Third-Person" Win: The system worked best when it used only the external cameras (the "Exos" view). It reached 50.17% accuracy, beating the previous best method by a significant margin.
The "First-Person" Struggle: When they added the "GoPro on the head" camera (Ego view), the score actually dropped slightly.
- Why? The paper suggests that for judging skill, seeing the whole body from the outside is often more important than seeing what the player sees. The "head-cam" sometimes hides the feet or the body's posture, which are crucial for judging skill.
Efficiency: The system is very efficient. It doesn't need to retrain a whole new brain for every sport. It uses a "low-rank" adaptation (LoRA), which is like adding a few sticky notes to a giant textbook rather than rewriting the whole book. It only trains about 23% of its parameters, making it fast and cheap to run.

What the Experiments Showed

The authors did a "autopsy" of their system by removing parts to see what happened:

Removing the Expert Panel: If they forced all cameras to use the same brain instead of routing them to different experts, accuracy dropped by 6.6%. This proves the "traffic controller" idea is the most important part.
Removing the Team Huddle: If they didn't let the cameras talk to each other, accuracy dropped by 4.9%.
Removing the Reference Cards: If they took away the "Novice/Expert" cards, accuracy dropped by 4.1%.

The Bottom Line

SkillMoV is a smart, flexible system that judges human skills by:

Letting different camera angles talk to different "expert" brains.
Having those experts compare notes.
Checking the performance against learned "reference cards" for different skill levels.

It proves that to judge a skill well, you don't need a separate AI for every sport. You just need a smart system that knows how to look at the action from the right angle, using the right expert, for the right moment.

Technical Summary: SkillMoV

Problem Statement

Estimating human proficiency from video is a critical challenge for automated skill assessment in domains such as sports coaching, music pedagogy, and surgical training. Unlike action recognition, which identifies what is being performed, proficiency estimation determines how well an activity is executed, relying on subtle visual cues like posture, timing, and coordination.

Existing approaches face two primary limitations:

Scenario Specificity: Many models are trained for single activity domains, limiting scalability and requiring separate models for different tasks.
Ineffective Multi-View Aggregation: Current unified methods often rely on shared aggregation schemes (e.g., standard attention or gating) that force all camera views through the same transformation. This fails to account for the fact that different synchronized viewpoints may expose distinct aspects of skill execution (e.g., limb extension vs. body-environment interaction), potentially diluting view-specific proficiency cues.

The goal is to develop a unified, parameter-efficient framework that can estimate proficiency across heterogeneous activity domains using synchronized multi-view video without requiring camera-identity supervision or scenario-specific heads.

Methodology: SkillMoV

The authors propose SkillMoV, a framework built upon a LoRA-adapted TimeSformer backbone. The core innovation is the Mixture-of-View Projector (MoVP), a hierarchical module that adapts the Mixture-of-Experts (MoE) paradigm to camera-specific view features.

The MoVP consists of four sequential stages:

Mixture-of-View (MoV) Routing:
- Instead of routing generic tokens, MoV treats camera-specific view features as the routed units.
- A soft router predicts a probability distribution over a pool of 12 expert MLPs for each view.
- Each view feature is transformed by a weighted combination of these experts.
- Crucially, the expert pool is shared across all cameras, clips, and activity domains, but the routing preferences are learned without explicit camera-identity supervision. A load-balancing regularizer prevents expert collapse.
Cross-View Attention:
- The routed features are passed through a multi-head self-attention layer where views act as a sequence.
- This allows synchronized cameras to align complementary evidence before final aggregation.
Prototype Anchoring:
- The model learns four prototype vectors (one per proficiency level: Novice, Early Expert, Intermediate Expert, Late Expert).
- These prototypes act as learnable anchor tokens in a cross-attention block, where view features serve as queries.
- The resulting representation is pooled, and cosine similarities between the pooled feature and the prototypes are computed.
Prototype-Conditioned Gated Projection:
- The final skill embedding is generated via a gated projection module.
- The gate is conditioned on both the expanded hidden features and the similarity vector derived from the prototypes.
- This allows the projection to adapt dynamically based on the sample's relation to the learned proficiency anchors.

Training Strategy:

Backbone: TimeSformer-base pre-trained on Kinetics-600, adapted via LoRA (Rank 64, $\alpha$ 128). Only LoRA adapters and the MoVP are trainable.
Objective: Class-balanced cross-entropy loss to handle dataset imbalance.
Regularization: Stochastic view dropout (probability 0.2) is applied during training for multi-camera configurations to prevent overfitting to dominant views.
Data: Evaluated on the EgoExo4D benchmark across six skill domains (Basketball, Bouldering, Cooking, Dance, Music, Soccer) using three view configurations: Ego (1 camera), Exos (4 cameras), and Ego+Exos (5 cameras).

Key Contributions

SkillMoV Framework: A parameter-efficient, unified framework for multi-scenario proficiency estimation from synchronized multi-view video.
Mixture-of-View (MoV) Routing: A novel adaptation of MoE where routing is applied to camera-specific view features, enabling soft, view-dependent expert preferences without camera-identity labels.
Mixture of View Projector (MoVP): A hierarchical architecture integrating MoV routing, cross-view attention, prototype anchoring, and prototype-conditioned gating.
Systematic Evaluation: Comprehensive ablation studies and comparisons across view configurations, architecture components, and temporal sampling strategies on the EgoExo4D benchmark.

Results

The model was evaluated on the EgoExo4D proficiency estimation benchmark.

Overall Performance (Exos Setting): SkillMoV achieved 50.17% overall accuracy in the Exos (4 exocentric cameras) configuration. This surpasses the strongest reported Exos result among compared methods (SkillFormer+PATS at 46.60%) by 3.57 percentage points.
Ego+Exos Setting: SkillMoV achieved 47.63%, remaining competitive with the best reported result in this setting (ProfVLM at 48.20%), with a gap of only 0.57 pp.
Ego-Only Setting: Performance was lower at 40.00%, consistent with a design optimized for multi-view routing.
Per-Scenario Performance: SkillMoV achieved the best result in the Soccer scenario (79.17%) across all view configurations and showed the most balanced performance profile across activities in the Exos setting, with the lowest standard deviation (14.65 pp) among top performers.

Ablation Studies (Exos Configuration):

MoV Routing: Replacing MoV with attentive pooling caused the largest drop (-6.61 pp), confirming the efficacy of view-conditioned expert routing.
Cross-View Attention: Removal reduced accuracy by 4.92 pp.
Prototype Anchoring: Removal reduced accuracy by 4.07 pp.
Stochastic View Dropout: Removal reduced accuracy by 3.90 pp.
Efficiency: SkillMoV trains only 23.32% of its parameters (via LoRA and the lightweight projector), adding minimal overhead (training time increased by ~3.5 minutes; inference memory increased by 0.02 GB) compared to a LoRA-only baseline.

Significance and Claims

The paper claims that SkillMoV demonstrates that proficiency estimation benefits from view-conditioned, softly shared representations rather than forcing all views through a single shared projection.

View-Conditioned Routing: The routing analysis confirms that the model learns soft, view-dependent expert preferences without explicit supervision. Different cameras and activity domains activate different subsets of experts, yet the expert pool remains shared and active (no collapse).
Exocentric Dominance: The results highlight that exocentric views (Exos) provide more reliable proficiency cues than egocentric views for this specific benchmark, likely because proficiency labels are annotated based on external perspectives. Adding the egocentric stream did not improve performance, suggesting that more views are not automatically better if they do not align with the visual evidence underlying the labels.
Parameter Efficiency: The framework achieves state-of-the-art results in the Exos setting while maintaining a parameter-efficient profile suitable for unified multi-domain training.

The authors modestly note limitations, including that the learned prototypes do not form a calibrated ordinal scale (their cosine similarity structure is not monotonic) and that per-scenario test subsets (e.g., Soccer) are small. Future work is proposed to address prototype geometry constraints, adaptive temporal sampling, and detailed failure analysis.