🤖 AI

Co-policy: Responsive Human-Robot Co-Creation for Musical Performances

The paper presents Co-policy, a framework for embodied human-robot musical co-creation that integrates a fine-tuned Qwen-vl planner for semantic intent grounding with a Gaussian-Mixture Visuomotor Policy to generate real-time, physically executable complementary musical responses.

Original authors: Xuetao Li, Wenke Huang, Mang Ye, Zijian Liu, Jinhua Xie, Jifeng Xuan, Miao Li

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xuetao Li, Wenke Huang, Mang Ye, Zijian Liu, Jinhua Xie, Jifeng Xuan, Miao Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: From "Playback" to "Jam Session"

Imagine you are playing a song on a piano.

The Old Way (Robotic Playback): You tell a robot, "Play these three notes." The robot hits the keys exactly as told, like a tape recorder. It's accurate, but it's boring. It's just a machine doing a chore.
The New Way (Co-policy): You play a few notes and say, "Hey, let's make this sound energetic!" The robot listens, understands your mood, and then plays a complementary melody that fits with yours. It's not just copying you; it's jamming with you.

This paper introduces Co-policy, a system that turns a robot from a "tape recorder" into a "musical partner."

The Three-Step Recipe for Robot Music

The authors built a system that breaks the complex task of "robot music" into three distinct jobs, like a small band with specific roles:

1. The Translator (The "F-Qwen" Planner)

The Job: This part listens to what you say, looks at the notes you played, and sees the robot's camera view of the instrument.
The Analogy: Think of this as a musical translator who speaks both "Human" and "Robot." If you say, "Make it happy," and play a sad melody, the translator doesn't just repeat the sad notes. It looks up a "cheat sheet" (called a Semantic Anchor Bank) to figure out what a "happy" version of that melody looks like. It creates a structured plan: "Okay, I will play these specific notes to match your energy, but I'll avoid hitting the bell that is currently vibrating."
Why it matters: It stops the robot from just copying you. It forces the robot to add something new that fits the vibe.

2. The Conductor (The Constrained Variation)

The Job: Once the translator has a plan, this step checks the rules.
The Analogy: Imagine a strict but helpful conductor. The translator might say, "Let's play a high note!" But the conductor checks the sheet and says, "Wait, the robot's arm can't reach that high note without hitting the table. Let's pick a lower note that sounds just as good."
Why it matters: It ensures the robot's ideas are physically possible. It balances creativity with reality.

3. The Muscle (The GMP - Gaussian-Mixture Visuomotor Policy)

The Job: This is the part that actually moves the robot's arm to hit the chime.
The Analogy: Most robots are like single-track recorders. If they need to hit a bell, they calculate one perfect path and follow it. If they miss, they fail.
- Co-policy's GMP is like a jazz drummer. When it needs to hit a bell, it doesn't just pick one way to do it. It thinks, "I could hit it from the top, or swing from the side, or tap it gently." It keeps all these options in its head at once.
- The Speed Trick: Other advanced robots (using "Diffusion Policies") are like a painter who has to slowly add layers of paint to get the right color. It takes time. Co-policy's GMP is like a photographer who snaps the perfect picture instantly. It predicts all the possible ways to hit the bell in one single "snapshot" (forward pass), making it fast enough for real-time music.

How They Tested It

The researchers didn't just simulate this on a computer; they built a real robot with a flexible, human-like hand to strike a set of musical chimes (like a xylophone).

The Test: Humans played a short melody or gave a verbal instruction (e.g., "Make it energetic").
The Result: The robot listened, figured out a complementary melody, and physically struck the chimes in time with the human.
The Score: Human experts (professional musicians) listened to the results. They rated the robot higher on "intent alignment" (did it get the mood right?) and "creativity" (did it add something new?) compared to older robot systems.
The Speed: The robot reacted much faster than the "slow painter" robots, allowing for a back-and-forth musical conversation without awkward pauses.

The Key Takeaway

The paper argues that for robots to truly "create" with humans, they can't just be smart computers that output text or audio files. They must be embodied. They need to understand that their physical body has limits (can it reach that bell?) and that their actions happen in real-time (can it hit the note before the human finishes the phrase?).

Co-policy solves this by separating the "thinking" (what to play) from the "doing" (how to move), allowing the robot to be both creative and physically precise, turning a solo performance into a duet.

Technical Summary: Co-policy – Responsive Human-Robot Co-Creation for Musical Performances

1. Problem Statement

The paper addresses the challenge of embodied human-robot musical co-creation. While generative AI has advanced in producing disembodied content (text, audio, scores), robotic systems often function merely as playback devices that execute pre-specified notes without genuine creative contribution or real-time physical adaptation.

The authors identify two unresolved scientific questions:

Semantic-to-Action Gap: How can a robot transform an incomplete human creative seed (e.g., a motif, verbal intent, or style) into a complementary musical response rather than a mechanical copy?
Physical Execution & Latency: How can this response be executed physically, accurately, and with interaction-level latency?

Existing solutions face trade-offs:

Diffusion Policies: Highly expressive but suffer from high latency due to iterative denoising, making them unsuitable for tight human-robot musical timing.
Standard Imitation Models: Often collapse toward mean actions, losing the diverse striking modes necessary for expressive performance.
Disembodied Generative Models: Lack the physical constraints (reachability, instrument dynamics) required for real-world execution.

2. Methodology: The Co-policy Framework

Co-policy is a modular, lightweight embodied AI agent that separates the pipeline into three distinct stages: Semantic Intent Grounding, Constrained Musical Variation, and Visuomotor Execution.

2.1 Semantic Intent Grounding and Planning

Instead of treating the Vision-Language Model (VLM) as a monolithic end-to-end generator, Co-policy uses it as a semantic grounding module.

Semantic Anchor Bank: A database of paired musical seeds, score images, style descriptors, and robot-playability tags is used to retrieve relevant context during inference.
Fine-tuned Qwen-vl (F-Qwen): The VLM is fine-tuned to output structured JSON plans rather than free-form text. It processes human speech, live musical seeds (extracted from audio), and visual observations (RGB) to define the robot's role (e.g., "accompaniment"), style, tempo, and reachable notes.
Constrained Variation Planner: The planner generates a robot note response ( $N^r_t$ $N_{t}^{r}$ ) that must satisfy four criteria:
1. Motif Consistency: Retaining recognizable structure from the human seed.
2. Harmonic Validity: Avoiding tonal conflicts.
3. Novelty: Preventing direct copying (ensuring creative contribution).
4. Embodied Playability: Ensuring the plan is executable within the robot's workspace and latency limits.

2.2 Gaussian-Mixture Visuomotor Policy (GMP)

To address the latency and multimodality challenges, the paper introduces GMP, a single-pass conditional mixture-density policy.

Guided Self-Attention (GSA) Encoder: To resolve the global-local imbalance in visual reasoning, the encoder uses a dual-stream architecture:
- A Swin Transformer stream captures long-range spatial dependencies (scene layout).
- A DenseNet stream extracts pixel-wise local features (contact regions).
- Cross-Stream Coupling: Queries are drawn from the global stream, while keys and values come from the local stream. This allows the model to ask "where is the target?" using global context but answer using local evidence, improving the localization of specific chime bells.
Mixture-Density Action Head: Instead of predicting a single deterministic action (which leads to mode collapse), GMP predicts a conditional Gaussian mixture distribution over full robot action vectors (or short-horizon trajectories):
$p_\theta(\tau_t | O_t, N^r_t, c_t) = \sum_{k=1}^K \pi_k \mathcal{N}(\tau_t; \mu_k, \Sigma_k)$
- Each component $k$ represents a latent action mode (e.g., striking from above, side-swing, or gentle tap) rather than a specific joint.
- This preserves multiple feasible physical responses in a single forward pass, avoiding the iterative denoising of diffusion models.
Inference: The system selects the most likely mode ( $\arg\max \pi_k$ ) or aggregates the mixture expectation, enabling low-latency execution.

3. Key Contributions

Formal Definition of Embodied Co-creation: The paper defines co-creation as a closed-loop interaction where the robot contributes a complementary response to an incomplete human seed, distinguishing it from robotic playback or disembodied generation.
Modular Embodied Agent: A framework combining F-Qwen for semantic grounding, a constrained planner for musical variation, and GMP for execution. This modular approach avoids the deployment costs of monolithic end-to-end VLA models while maintaining physical grounding.
Low-Latency GMP: A conditional mixture-density policy that predicts multimodal action distributions in a single forward pass, preserving alternative feasible action modes without the latency of diffusion-based denoising.

4. Experimental Results

The system was evaluated on real-world chime performance and the ManiSkill2 simulation platform.

4.1 Real-Robot Evaluation (Chime Performance)

Co-creation Quality: In blinded expert evaluations (10 professional chime players), Co-policy with F-Qwen and GMP achieved an average score of 74.8 in semantic music co-creation and 62.0 in concerto co-creation. This outperformed the Diffusion Policy baseline (66.3 and 55.8, respectively) and other baselines like $\pi_0.5$ and GR00T.
Execution Accuracy: Co-policy achieved an action success rate ( $Acc_a$ ) of 0.78 and tuning accuracy ( $Acc_t$ ) of 0.88, resulting in a combined accuracy of 0.69. This is a significant improvement over the Diffusion Policy (0.56 combined accuracy).
Latency: The post-command action frequency for Co-policy was 18.6 Hz, significantly higher than the Diffusion Policy (1.01 Hz) under the same hardware and action horizon settings.

4.2 Ablation Studies

Semantic Anchoring: Replacing the off-the-shelf Qwen-vl with the fine-tuned F-Qwen improved co-creation scores by 7.4%–13.3% in semantic music and 3.5%–9.6% in concerto tasks.
GMP Components: Ablation of the GMP encoder components showed that the Swin Transformer stage contributed 32% to performance, the DenseNet block 27%, the GSA mechanism 21%, and the mixture-density head 16%.
Generalization: On the ManiSkill2 simulation platform (Pour, Fill, Hang, Excavate tasks), GMP demonstrated robust generalization, validating the visuomotor policy's effectiveness beyond the specific chime task.

5. Significance and Claims

The paper claims that physically grounded action generation is a necessary component of human-AI co-creation.

Beyond Playback: The system demonstrates that a robot can transform an incomplete human seed into a semantically aligned, musically coherent, and physically executable complementary response.
Interpretability: By separating semantic grounding, planning, and execution, the system allows for independent inspection of failures (semantic vs. perceptual vs. execution), unlike monolithic end-to-end models.
Latency vs. Expressivity: The work shows that a conditional mixture-density policy can preserve multiple feasible action modes (expressivity) while meeting the strict latency requirements of musical interaction, a trade-off often difficult for diffusion models.

The authors maintain a modest scope, acknowledging that the current system is validated primarily on chime performance and relies on high-level symbolic audio cues rather than closed-loop tactile feedback. They position the work as a step toward "guided variation" in embodied AI, where creativity is constrained by physical feasibility and instrument properties.