🤖 AI

MyoInteract: A Framework for Fast Prototyping of Biomechanical HCI Tasks using Reinforcement Learning

This paper introduces MyoInteract, a novel framework that utilizes a user-friendly GUI and optimized training processes to enable interaction designers to rapidly prototype, train, and evaluate biomechanical RL simulations, thereby significantly reducing development time and lowering barriers to entry for non-experts.

Original authors: Ankit Bhattarai, Hannah Selder, Florian Fischer, Arthur Fleig, Per Ola Kristensson

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ankit Bhattarai, Hannah Selder, Florian Fischer, Arthur Fleig, Per Ola Kristensson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are an architect designing a new type of door handle. In the past, to test if your design was comfortable for people to use, you would have to build a physical prototype, hire a dozen people to try it out, and wait weeks to analyze the results. If the handle hurt their wrists, you'd have to start over.

Now, imagine you could build a "digital twin" of a human arm—a robot made of virtual muscles and bones—and test your handle design on it instantly. That is the promise of Biomechanical Reinforcement Learning (RL). It allows researchers to simulate how a human body moves and interacts with technology without needing real people.

However, until now, using this technology was like trying to fly a spaceship by manually wiring every single circuit while blindfolded. It took days, required a PhD in robotics, and if something went wrong, you often didn't know why until the simulation crashed.

MyoInteract is a new tool that turns that spaceship cockpit into a simple, user-friendly dashboard. Here is how it works, explained through everyday analogies:

1. The Problem: The "Black Box" and the "Time Trap"

Before MyoInteract, setting up a simulation was a nightmare of "execution" and "evaluation" gaps:

The Execution Gap: You couldn't just say, "Make a virtual hand press a button." You had to write complex code (XML and Python files) to tell the computer exactly how to move every muscle fiber. It was like trying to bake a cake by manually calculating the chemical reaction of every egg molecule instead of just mixing the batter.
The Evaluation Gap: Once you started the simulation, it was a "black box." You would wait 12 to 48 hours (days!) for the computer to learn. When it finally finished, if the virtual hand moved strangely, you had no idea why. Was the reward system wrong? Was the target too far? You had to guess and start the whole 2-day process over.

2. The Solution: MyoInteract

The authors built MyoInteract to fix these problems. Think of it as a "Lego set" for human movement simulations.

The "Lego" Approach (Task Decomposition): Instead of building a whole robot from scratch, MyoInteract gives you pre-made blocks. You have a "Pointing" block and a "Pressing" block. You can snap them together to create complex tasks (like "Point at a virtual screen, then press a physical button") using a simple visual menu. No coding required.
The "Turbo Button" (GPU Acceleration): The biggest breakthrough is speed. The researchers used a powerful new engine (MuJoCo-MJX) that runs on graphics cards (GPUs) instead of standard processors.
- Analogy: If the old way was like a single snail carrying a heavy load, MyoInteract is like a thousand snails working in perfect unison.
- Result: Tasks that used to take 12 to 48 hours now take under one hour (sometimes just minutes). This is a 98% reduction in time.
The "Dashboard" (Real-time Feedback): Instead of waiting days to see the results, MyoInteract gives you a live dashboard. You can watch the virtual hand learn in real-time. If it's failing to reach a target, the dashboard shows you exactly which part of the task is broken. It's like having a GPS that tells you "You're stuck in traffic" immediately, rather than waiting until you arrive at your destination to realize you took a wrong turn.

3. The Proof: The Workshop

To test if this actually works for regular people (not just robotics experts), the authors held a workshop with 12 interaction designers. Most of them had never used Reinforcement Learning before.

The Challenge: They gave these designers a simple goal: Create a simulation of a specific interaction (like tapping a phone screen or pressing a button on a public display) and train the virtual user to do it.
The Outcome: Within a single session (about two hours), every participant successfully:
1. Set up their task using the visual menu.
2. Trained the virtual user.
3. Watched the results and analyzed why the virtual hand moved the way it did.
The Verdict: The designers found the tool intuitive. They could tweak settings (like how hard the button needs to be pressed) and see the results immediately. One participant noted that the visual feedback made it easy to understand how different settings changed the outcome, even without being an expert.

4. Why This Matters (According to the Paper)

The paper claims that MyoInteract transforms biomechanical simulation from a specialized research tool (used only by experts who can wait days for results) into a practical design instrument (usable by designers in a single work session).

It lowers the barrier: You don't need to be a coder or a physicist to use it.
It speeds up iteration: You can test 10 different button layouts in an hour instead of waiting a month.
It keeps the science real: Even though it's fast and easy, the virtual movements still follow real human laws of physics (like Fitts' Law, which predicts how fast we move to a target). The paper verified that the movements generated by MyoInteract look and feel just like real human movements.

Summary

MyoInteract is a new framework that makes simulating human movement as easy as playing with a toy building set. By using super-fast computer chips and a simple visual interface, it turns a process that used to take days and require a PhD into a process that takes minutes and can be done by anyone. It allows designers to "try before they buy" in the physical world, testing how comfortable and efficient their designs are for the human body, all within the time it takes to drink a cup of coffee.

Technical Summary: MyoInteract

Problem Statement

Biomechanical Reinforcement Learning (RL) offers a powerful method for generating generative models of user behavior in Human-Computer Interaction (HCI), capable of predicting motion trajectories and physical effort for complex tasks. However, the field is currently restricted to a small group of experts due to three primary barriers, analyzed through Norman's Action Cycle:

Gulf of Execution: Configuring simulations requires extensive low-level coding (XML, Python, YAML), custom reward function design, and deep expertise in physics engines (e.g., MuJoCo) and RL algorithms. This prevents interaction designers from easily specifying what they want to test.
Gulf of Evaluation: The training process is a "black box." Feedback is typically limited to aggregate metrics (e.g., mean reward) available only after training completes. Diagnosing why a policy failed (e.g., unreachable targets vs. poor reward shaping) requires restarting the process, which is inefficient.
Temporal Barrier: Training a single policy traditionally takes 12 to 48 hours. This duration makes iterative prototyping impossible, as setting up a simulation often takes longer than conducting a human study.

These barriers prevent biomechanical RL from serving as a practical, iterative design tool for the broader HCI community.

Methodology

The authors present MyoInteract, a framework designed to collapse these barriers by integrating GPU-accelerated physics simulation with a user-friendly Graphical User Interface (GUI).

System Architecture and Acceleration

Backend: MyoInteract is built on MuJoCo-MJX, a GPU-accelerated re-implementation of the MuJoCo physics engine using the JAX framework. This allows for thousands of parallel environments to run simultaneously on GPUs/TPUs.
Algorithm: The system utilizes Proximal Policy Optimization (PPO) implemented in Brax, enabling end-to-end GPU execution without CPU-GPU transfer bottlenecks.
Performance Gain: This architecture reduces training times by up to 98% compared to previous CPU-based or less parallelized approaches (e.g., User-in-the-Box), compressing training from days to minutes.

Task Modeling and Composability

To address the Gulf of Execution, MyoInteract introduces a compositional task modeling approach:

Primitives: Complex interactions are decomposed into two fundamental primitives: Pointing (moving an end-effector to a target sphere) and Pressing (applying force to a surface).
Sequencing: Users define complex workflows (e.g., AR menus, public displays) by chaining these primitives sequentially.
Abstraction: The system automatically generates the underlying simulation logic (XML/Python) based on high-level configurations, eliminating the need for manual scripting.

Interface and Observability

The framework features a Gradio-based GUI with two modes to manage complexity:

Simple Mode: Exposes essential parameters (target size, location, dwell time, force thresholds) via sliders and a 3D renderer for pre-training visualization. It allows users to adjust reward function weights and see immediate mathematical impacts.
Advanced Mode: Provides granular control over biomechanical fidelity (noise models, muscle activation), observation spaces, and RL hyperparameters.
Multi-Level Feedback (Observability): To address the Gulf of Evaluation, the system provides real-time dashboards (via Weights & Biases) that decompose metrics into:
- Sequential Subtask Completion: Success rates for individual targets.
- Spatial Error Analysis: 3D visualizations of success rates per location.
- Reward Decomposition: Breakdown of distance, completion bonuses, and effort penalties.
- Qualitative Inspection: Automatic generation of training videos to verify biomechanical plausibility (e.g., checking for unnatural postures).

Biomechanical Model

The default model is an adaptation of MoBL-ARMS (26 muscles, fixed fingers) from the UitB framework. The system supports signal-dependent and constant motor noise to capture speed-accuracy trade-offs and randomizes initial joint states to ensure policy generalization.

Key Contributions

MyoInteract Framework: An open-source system for fast prototyping of biomechanical HCI tasks that enables configuration, simulation, and evaluation in under one hour.
GPU-Accelerated Training: A technical implementation that reduces training time by 91–98% (from ~7 hours to ~15–36 minutes for standard tasks) while preserving biomechanical validity.
Empirical Validation: A workshop study with 12 HCI researchers (mostly novices in RL) demonstrating that the framework allows non-experts to successfully setup, train, and assess goal-directed user movements in a single session.
Design Guidelines: A set of principles (Abstraction, Visibility, Progressive Disclosure) for designing accessible biomechanical simulation tools.

Results

Performance Benchmarks

Speed: Training the standard pointing task (10 targets) took an average of 6.9 hours on the authors' hardware using the existing User-in-the-Box (UitB) framework. MyoInteract (Replication) achieved convergence in 0.17 hours (approx. 10 minutes), and the Default configuration in 0.60 hours (approx. 36 minutes). This represents a 40x to 10x speedup, respectively.
Complex Tasks: Three diverse scenarios (AR Interaction, Public Display, Mobile Typing) were successfully trained in less than 30 minutes each.

Biomechanical Validity

The policies trained with MyoInteract were validated against established motor control principles:

Fitts' Law: The simulated movements adhered to Fitts' Law with an $R^2$ of 0.89.
Movement Profiles: The generated trajectories exhibited the characteristic bell-shaped velocity profiles and "settling phases" required for dwell-time tasks, consistent with previous literature.

Workshop Evaluation

Usability: Participants (8 novices in RL) rated the task setup and training process as "relatively easy" (mean Likert score 3.58/5 for setup, 3.75/5 for training).
Accessibility: Participants reported that the GUI made it possible to configure tasks without extensive RL background knowledge.
Feedback Utility: Visual feedback (videos and decomposed metrics) was highly valued for monitoring progress and diagnosing issues, though some participants requested more guidance on interpreting specific metrics.

Significance and Claims

The paper claims that MyoInteract fundamentally shifts biomechanical RL from a specialized, days-long research endeavor into an accessible, hour-scale workflow suitable for iterative design.

Lowering Barriers: By collapsing the temporal and expertise barriers, the framework allows interaction designers to test hypotheses "in silico" without waiting for human studies or requiring deep technical expertise in physics engines.
Iterative Design: The speed of the system enables a qualitative shift in workflow, allowing researchers to run multiple simulations, compare variants, and refine rewards within a single working session.
Generative Tool: The authors posit that by making the physical body computationally accessible, biomechanics can evolve from a method for explaining movement-based interaction to a generative tool for designing it.

The paper remains modest regarding its current scope, acknowledging that it currently focuses on sequential target acquisition and selection (pointing/pressing) and that future work is needed to expand task complexity, integrate cognitive models, and improve transfer learning across tasks.