🤖 machine learning

Profy: Interpretable Visualization of Expertise-Dependent Motor Skills Toward Supporting Piano Practice

The paper introduces Profy, a weakly supervised visualization system that leverages aggregated listener ratings to generate time-aligned highlights and evidence scores from synchronized audio and key-motion data, enabling piano learners to identify and review specific passages where their performance diverges from expert standards without requiring localized ground-truth labels.

Original authors: Kazuki Kawamura, Fujiki Nakamura, Hayato Nishioka, Momoko Shioki, Shinichi Furuya, Jun Rekimoto

Published 2026-06-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kazuki Kawamura, Fujiki Nakamura, Hayato Nishioka, Momoko Shioki, Shinichi Furuya, Jun Rekimoto

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to learn a difficult piece of music on the piano. You practice for hours, but when you listen back, you can't quite figure out why it sounds "off." You know you hit the right notes, but the timing feels rushed, or the rhythm is uneven in a way you can't catch. Usually, you'd need a teacher to listen, point out the exact second where you stumbled, and say, "Stop right there, try that again."

This paper introduces Profy, a smart system designed to be that teacher for solo practice, but with a clever trick: it doesn't need a teacher to sit there and mark every single mistake in advance.

Here is how Profy works, explained through simple analogies:

1. The Problem: The "Blind Spot" in Practice

When you practice alone, you often don't know where to look. It's like trying to find a specific typo in a 10-page document without a "Find" function. You might know the whole document feels "clunky," but you don't know if the problem is on page 3, line 2, or page 7, line 5. Most piano apps just give you a final grade (like "B-"), which tells you that you made mistakes, but not where to fix them.

2. The Solution: Profy as a "Smart Highlighter"

Profy is like a highlighter pen that automatically scans your recording and marks the specific moments that sound "amateur-ish" compared to how a pro would play.

How it learns: Imagine you show Profy 1,000 recordings. You tell it, "This recording is by a Pro," and "This one is by an Amateur." You don't tell it which specific notes were wrong. You just give it the label for the whole song.
The Magic: Even without being told exactly where the mistakes are, Profy learns to spot the subtle patterns that separate the pros from the amateurs. It figures out that pros have smoother finger movements and more consistent timing, while amateurs might have tiny, invisible jerks or rushed moments.
The Output: When you play a piece, Profy analyzes the sound and the actual movement of your fingers (using special sensors that track every key at 1,000 times per second). It then draws a "highlight" on a timeline showing exactly which seconds you should scrub back to and loop.

3. The "Two Eyes" Approach

Profy doesn't just listen; it also "sees" your fingers.

The Audio Eye: It listens to the sound.
The Motion Eye: It watches your fingers move up and down on the keys with extreme precision (faster than the human eye can see).
The Safety Net: If your room is noisy and the audio is bad, Profy relies more on the motion eye. If your fingers are moving weirdly but the sound is clear, it uses the audio. It combines both to make sure it doesn't get confused by a cough or a noisy room.

4. Why This is a Big Deal

Usually, to build a system that finds specific mistakes, you need a human expert to sit down and draw boxes around every single error in hundreds of videos. That takes forever and costs a lot of money.

Profy is different. It uses Weak Supervision. Think of it like training a dog. Instead of teaching the dog, "If you see a red ball, bark," you just say, "Good boy" when it finds a ball and "No" when it doesn't. Eventually, the dog figures out what a ball looks like on its own. Profy does the same: it learns to find the "bad spots" just by knowing which whole recordings were good and which were bad.

5. Does it actually work?

The researchers tested this with 73 pianists. They asked 21 expert pianists to manually mark the parts of a recording that needed work. Then, they compared the experts' marks with Profy's highlights.

The Result: Profy's highlights matched the experts' marks surprisingly well (about 61% correlation). It successfully pointed out the same "trouble spots" that a human teacher would have found, even though it was never taught exactly where those spots were.

6. How You Would Use It

Imagine you are practicing a scale. You hit "Record." When you finish, Profy shows you a timeline.

It highlights a tiny 3-second chunk in the middle where your fingers rushed.
You click that highlight, and the music loops just that part.
You try it again.
You don't waste time listening to the parts you played perfectly; you go straight to the part that needs work.

Summary

Profy is a tool that turns a vague feeling of "I'm not playing well" into a specific, actionable target: "Try looping this 2-second section." It does this by learning from the difference between experts and beginners, using both sound and finger-motion data, all without needing a human to manually label every single mistake in the training data. It's like having a tireless, hyper-observant coach who knows exactly where to focus your attention, even if they can't explain why in words.

Technical Summary: Interpretable Visualization of Expertise-Dependent Motor Skills Toward Supporting Piano Practice

Problem Statement

Learning piano performance requires mastering nuanced motor skills—such as micro-timing, articulation, and dynamic balance—that extend beyond simple note correctness. While teachers provide contextual guidance, most practice occurs in isolation, leaving learners unaware of where, when, and how their playing deviates from pedagogical expectations. Conventional practice tools typically provide global scores or correctness metrics, which fail to identify specific passages requiring focused review (e.g., scrubbing or looping). Furthermore, obtaining dense, time-aligned annotations for nuanced techniques is expensive and impractical for everyday practice, making fully supervised localization unfeasible at scale. Existing deep learning models often remain opaque, offering raw scores rather than domain-centered explanations that musicians can act upon.

Methodology: The Profy System

Profy is a weakly supervised system designed to generate time-aligned review highlights from a single piano performance without requiring localized annotations. It leverages high-resolution sensing and multimodal fusion to decompose a binary take-level label into localized evidence.

1. Data Collection and Corpus

The authors curated a multimodal corpus comprising 1,083 valid takes from 73 pianists (ranging from amateurs to experts).

Sensors: Synchronized 1 kHz per-key motion data (vertical displacement in mm) captured via a non-contact optical system (HackKey) and 44.1 kHz stereo audio.
Tasks: Short technique studies (15 unique tasks: 9 scales, 6 arpeggios) designed to stress specific motor control challenges like thumb-under transitions and hand relocations.
Labels: Binary take-level labels (Expert-labeled vs. Amateur-labeled) derived from aggregated listener ratings (technique and musicality) thresholded at the corpus median. No localized annotations were used for training.

2. Model Architecture

The system processes synchronized sensor and audio streams through a three-stage pipeline:

Preprocessing: Sensor and audio streams are aligned via onset detection and cross-correlation, then interpolated to a fixed-length time base ( $T=1000$ frames). Audio quality indicators (non-silence ratio, spectral flatness, loudness) are extracted to inform reliability.
Multimodal Encoding & Fusion:
- Two lightweight encoders map sensor and audio sequences into a shared hidden space.
- Bidirectional Cross-Modal Attention: Four-head attention allows the motor control stream to query the acoustic stream and vice versa.
- Reliability-Aware Gating: A gating network, conditioned on acoustic quality indicators, dynamically mixes the sensor encoding, audio encoding, and cross-attended features. This ensures robustness when audio quality is poor (e.g., high noise), shifting reliance toward the sensor data.
Temporal Decomposition (The Core Innovation):
- A 2-layer BiLSTM processes the fused representation.
- Attention Head: Predicts frame importance weights ( $\alpha_t$ ).
- Evidence Head: Predicts signed local evidence ( $\ell_t$ ), where positive values support the "Expert" class and negative values support the "Amateur" class.
- Contribution Calculation: The clip-level prediction is decomposed into per-frame contributions: $c_t = \alpha_t \cdot \ell_t$ .
- Highlight Generation: The system visualizes only frames where the contribution pulls the prediction toward the "Amateur" class: $s_t = m_t \cdot \text{ReLU}(-c_t)$ . This creates a time-localized "review signal" indicating passages where the performance deviates from expert norms.

3. Training Strategy

The model is trained end-to-end using only the take-level binary label via a Binary Cross-Entropy (BCE) loss. No frame-level supervision is provided. The system learns to assign high attention weights to frames that provide strong evidence for the correct class label. Regularization terms encourage non-collapsed fusion and sparse, peaky highlight distributions without requiring ground-truth spans.

Key Contributions

Profy System: A piano practice interface that renders time-aligned review passages for scrubbing, looping, and score-linked review, derived solely from coarse take-level labels.
Weakly Supervised Multimodal Model: A novel architecture that decomposes take-level predictions into signed, time-localized evidence scores, enabling interpretable visualization without dense annotations.
Validation Against Expert Consensus: An evaluation demonstrating that weakly supervised localization aligns with human expert judgments on localized review spans.

Evaluation Results

The system was evaluated on a performer-disjoint 3-fold cross-validation setup using 1,083 takes.

Clip-Level Classification (E1): The multimodal model achieved a Macro-F1 of 0.781 and Accuracy of 0.782, outperforming unimodal baselines (Sensor-only: 0.756; Audio-only: 0.759) and a decision-level Product-of-Experts baseline. This confirms the learnability of the expertise signal from coarse labels.
Highlight Validity (E2): On a subset of 20 amateur clips annotated by 21 expert pianists, the model's highlight score showed significant alignment with expert consensus:
- Pearson Correlation ( $r$ ): 0.612
- ROC-AUC: 0.753
- Average Precision (AP): 0.567
- The multimodal model outperformed unimodal baselines in AP and ROC-AUC, suggesting improved prioritization of review spans.
Robustness (E3): Under input corruption (audio noise and sensor dropout), the multimodal model maintained higher stability in its highlight curves compared to unimodal baselines. The reliability-aware gating successfully shifted weight to the sensor stream when audio quality degraded.
Pedagogical Alignment: Analysis of expert notes revealed the system is most effective at capturing localized, time-specific issues (e.g., timing, unevenness) with capture rates of 53–63%, while capturing more global issues (e.g., phrasing, tone) less consistently.

Significance and Claims

The paper claims that Profy demonstrates a viable design pattern for transforming coarse quality labels into actionable, localized feedback for embodied skill learning.

Bridging the Gap: It addresses the bottleneck of practice awareness by identifying specific moments of deviation without requiring expensive, dense annotations.
Interpretability: By visualizing signed contributions rather than just attention weights, the system provides a "direction-aware" review signal that tells learners where to look and why (i.e., because that moment supports the amateur classification).
Practical Utility: The system functions as a "triage tool," narrowing the search space for solo practice. It allows learners to scrub, loop, and replay specific passages associated with expert-amateur differences.
Modesty in Scope: The authors explicitly state that the system is validated as a local review aid for short technique studies. It is not a definitive diagnosis of musicality or a substitute for teacher judgment. The "expertise" defined is implicit in the training cohort and may not generalize to all genres or interpretive styles. The system is designed to complement human judgment by highlighting candidates for inspection rather than prescribing corrections.

In summary, Profy proves that high-resolution multimodal sensing combined with weak supervision can yield interpretable, time-localized cues that align with expert intuition, offering a scalable path toward automated, actionable feedback for motor skill acquisition.