💻 computer science

Parameter-Efficient Multi-View Proficiency Estimation: From Discriminative Classification to Generative Feedback

This paper presents three parameter-efficient methods—SkillFormer, PATS, and ProfVLM—that advance multi-view proficiency estimation on the Ego-Exo4D dataset by achieving state-of-the-art accuracy with significantly fewer resources while shifting from simple classification to generating interpretable, expert-style feedback.

Original authors: Edoardo Bianchi, Antonio Liotta

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Edoardo Bianchi, Antonio Liotta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach someone how to play basketball, cook a perfect meal, or climb a rock wall. You don't just want to know what they are doing (e.g., "they are shooting a ball"); you want to know how well they are doing it. Did they balance correctly? Was their timing perfect? This is the challenge of Proficiency Estimation.

The paper you shared describes a new way for computers to act like expert coaches. Instead of just giving a grade (like "A" or "B"), these computer systems can watch a person from multiple camera angles and explain exactly what they did right or wrong, all while using very little computer power.

Here is a breakdown of their three main "tools" using simple analogies:

1. The Problem: Too Much Data, Not Enough Focus

Usually, to watch a video, a computer looks at every single frame from start to finish, like a student reading every word of a book. But for sports or skills, the most important moments are tiny "micro-events" (like the split second a dancer lands a jump or a climber grabs a hold). If a computer spreads its attention too thin, it misses these details. Also, looking at a video from just one angle is like trying to judge a sculpture by looking at it from only one side; you miss the depth.

2. The Solution: Three Smart Tools

The authors built three different methods to solve this, moving from "just guessing the score" to "giving a detailed report."

Tool A: SkillFormer (The "Smart Filter")

Think of SkillFormer as a team of scouts watching a game from different seats in the stadium.

The Old Way: The computer tries to memorize every single pixel from every camera, which is heavy and slow.
The New Way: SkillFormer is "parameter-efficient." Imagine it's a scout who only writes down the most important notes. It uses a "gate" to decide which camera angles are useful at any given moment and fuses them together.
The Result: It achieves high accuracy but uses 4.5 times less memory and trains 3.75 times faster than older, heavier systems. It's like getting a perfect report card without having to read the entire encyclopedia.

Tool B: PATS (The "Highlight Reel" Maker)

Imagine you have a 10-minute video of a soccer game, but the computer is forced to look at exactly one frame every second. It might miss the actual goal because it wasn't looking at that exact second.

The Problem: Uniform sampling (checking at regular intervals) often misses the "action."
The Fix: PATS (Proficiency-Aware Temporal Sampling) is like a video editor who knows where the exciting parts are. Instead of spreading the camera's attention evenly, it zooms in and takes many quick snapshots of the same short action (like a jump or a throw) and then moves to the next action.
The Result: By focusing on the "dense" moments of movement, it improves accuracy, especially in complex skills like rock climbing or music, without needing a bigger computer.

Tool C: ProfVLM (The "Generative Coach")

This is the biggest leap. Previous systems were like multiple-choice quizzes: they just picked a label (e.g., "Novice" or "Expert").

The New Approach: ProfVLM is like a human coach who can talk. It doesn't just pick a label; it writes a sentence. It looks at the video and generates a response like: "Proficiency Level: Intermediate Expert. Commentary: Your form was good, but you lost balance on the landing."
How it works: It freezes the heavy video "eyes" (so it doesn't have to relearn how to see) and connects them to a small, smart "brain" (a language model). It uses a special bridge (called AGP) to translate what the eyes see into words the brain understands.
The Result: It is incredibly efficient. It uses 20 times fewer parameters (computer memory) than the old heavy systems and trains in 3 times fewer sessions. It gives you the score and the advice, all in one go.

3. The Big Takeaways

The paper highlights four main lessons for building these systems:

More Cameras ≠ Better Results: Just adding more cameras doesn't help if the computer doesn't know how to combine them. You need a smart "fusion" (like SkillFormer) to blend the views correctly.
Quality Over Quantity: You don't need to watch the whole video at high speed. Watching the right moments (using PATS) with fewer frames is often better than watching everything poorly.
From Grading to Coaching: Moving from simple classification (picking a label) to generation (writing feedback) gives us useful, interpretable advice without losing accuracy.
One Size Does Not Fit All: Different skills need different approaches. Rock climbing needs different timing than cooking. The best systems adapt to the specific activity.

Summary

In short, the authors created a new generation of AI coaches. These systems are lighter, faster, and smarter. They don't just watch videos; they understand the subtle details of human movement from multiple angles and can explain why someone is good or bad at a skill, all while using a fraction of the computer power required by older methods. They are moving us from "What did they do?" to "How well did they do it, and how can they improve?"

Technical Summary: Parameter-Efficient Multi-View Proficiency Estimation

1. Problem Definition

The paper addresses Action Quality Assessment (AQA) and proficiency estimation, which differ from standard action recognition by focusing on how well an action is performed rather than what action is performed. This task requires modeling subtle variations in timing, balance, body mechanics, and execution consistency. These cues are often distributed across multiple camera views and manifest as short, micro-temporal events that uniform sampling strategies frequently miss.

The authors argue that applications in coaching, rehabilitation, and talent identification require systems that are not only accurate but also interpretable and capable of providing expert-style feedback, moving beyond simple closed-set classification labels. The work is evaluated on the Ego-Exo4D dataset, which provides time-synchronized multi-view videos (one egocentric and up to four exocentric views) across six skill domains (cooking, basketball, soccer, dancing, music, bouldering), accompanied by proficiency labels and free-form expert commentary.

2. Methodology and Contributions

The paper details three recent contributions that collectively advance proficiency estimation through parameter efficiency, improved temporal sampling, and generative feedback:

A. SkillFormer: Parameter-Efficient Discriminative Architecture

SkillFormer introduces a multi-view discriminative architecture designed for selective fusion.

Backbone: It utilizes a shared TimeSformer backbone pretrained on Kinetics-600.
Efficiency: The backbone is adapted using LoRA (Low-Rank Adaptation) on attention projections, output layers, and feed-forward layers, resulting in only 14–27M trainable parameters (compared to 121M in baselines).
Fusion Mechanism: It employs CrossViewFusion, a module that normalizes view-specific embeddings and fuses them via multi-head cross-view attention. This is followed by mean aggregation, a feed-forward transformation, an element-wise learnable gate, and adaptive self-calibration.
Goal: To treat synchronized camera streams as complementary evidence that must be aligned and weighted rather than merely concatenated.

B. PATS: Proficiency-Aware Temporal Sampling

PATS is an architecture-agnostic sampling strategy designed to address the limitations of uniform sampling, which often fails to capture the evolution of fundamental movements.

Mechanism: Instead of spreading a fixed frame budget across the entire clip, PATS concentrates frames within short, continuous temporal segments. It selects $N_s$ segments of duration $d_s$ , distributes the frame budget across them, and samples densely within each segment.
Adaptability: Segment starts are spread to retain global coverage, while segment duration is adjusted to avoid overlap.
Integration: PATS replaces the standard sampler in existing models (like SkillFormer) without altering the model architecture or training setup.

C. ProfVLM: Generative Feedback via Vision-Language Modeling

ProfVLM reformulates proficiency estimation as a conditional language generation task, eliminating the need for a dedicated classification head.

Architecture: It uses a frozen TimeSformer backbone to encode 8-frame clips from each view. Features are processed by an AttentiveGatedProjector (AGP), which normalizes, fuses (via cross-view attention and mean pooling), and aligns the representation with a language embedding space.
Generation: The fused representation is injected as special video tokens into a compact, LoRA-adapted language model (SmolLM2-135M).
Output: The model generates a single autoregressive output containing both the proficiency label and natural-language feedback (e.g., "Proficiency Level: ; Proficiency Commentary: "). The label is parsed deterministically from the text.
Efficiency: This approach uses only 5.3M trainable parameters and requires significantly fewer training epochs.

3. Key Results

Evaluated on the Ego-Exo4D demonstrator proficiency subset, the proposed methods achieve state-of-the-art performance with drastically reduced computational costs:

Accuracy:
- SkillFormer achieves 47.5% top-1 accuracy on the combined Ego+Exos view configuration, surpassing the TimeSformer baseline (40.8%) while using 4.5× fewer trainable parameters and 3.75× fewer training epochs.
- SkillFormer + PATS further improves this to 48.0%.
- ProfVLM achieves the highest combined accuracy of 48.2% (Ego+Exos), outperforming the discriminative baselines despite using only 8 input frames and 6 training epochs.
Parameter Efficiency:
- ProfVLM uses approximately 5× fewer trainable parameters than SkillFormer and 20× fewer than the TimeSformer baselines.
Generative Quality:
- ProfVLM produces expert-style feedback with high textual similarity to ground truth, achieving BERTScore (F1) of 85.53, METEOR of 18.23, and ROUGE-L of 15.65 on the Ego+Exos configuration.
Temporal Sampling Impact:
- PATS yields the largest gains in domains requiring temporally coherent movement patterns (bouldering, music, basketball), demonstrating that local sampling density is more critical than total frame count.

4. Significance and Design Principles

The paper posits that the shift from closed-set classification to efficient, multi-view systems represents a significant step forward in AQA. The authors highlight four primary design lessons derived from their results:

Selective View Fusion: Simply adding views is insufficient; unstructured fusion can dilute cues. Effective alignment and weighted fusion (as in CrossViewFusion and AGP) are critical, as the optimal viewpoint is domain-dependent.
Temporal Sampling over Frame Count: More frames do not guarantee better performance. Effective sampling patterns that preserve local temporal density (PATS) allow models to match or surpass heavier baselines with fewer frames.
From Classification to Generation: Moving to generative models (ProfVLM) allows for the production of interpretable, expert-style feedback without sacrificing accuracy, while drastically reducing the parameter budget.
Domain-Aware Adaptation: No single temporal configuration or view setup is optimal for all skills. Future systems should likely employ shared visual encoders with lightweight, domain-specific adapters or sampling policies rather than monolithic models.

The authors conclude that these methods enable a transition toward interactive, agent-based systems capable of observing athletes across sessions and adapting feedback over time, while remaining compatible with on-device deployment through further compression techniques.