Technical Summary: Proactive Conversational Assistant for Procedural Manual Tasks based on Audio and IMU

Problem Statement

Real-time conversational assistants for procedural manual tasks (e.g., furniture assembly, cooking) typically rely on video input to maintain context awareness. However, video-based approaches present significant barriers: they are computationally expensive, often requiring bulky hardware for on-device processing, and compromise user privacy when data is uploaded to the cloud. Furthermore, existing wearable assistants often provide only reactive assistance (answering questions) rather than proactive guidance (narrating steps and intervening). While some proactive systems exist, they frequently depend on Head-Mounted Devices (HMDs) for egocentric video, which suffers from low adoption due to user discomfort. There is a need for a lightweight, privacy-preserving, on-device assistant capable of providing comprehensive, step-by-step guidance without relying on video or cloud infrastructure.

Methodology

System Architecture

The proposed system is a fully on-device, real-time proactive conversational assistant. It utilizes only audio and Inertial Measurement Unit (IMU) data from a user's wearable device (specifically a smartwatch on the dominant wrist) to understand context. The system architecture (Figure 1) operates as follows:

1. Input: A custom Android application on a Snapdragon W5 Gen 1 watch streams audio (32 kHz) and IMU data (50 Hz) to an Edge AI box.
2. Activity Recognition: A neural network processes these signals to recognize specific user activities (e.g., sanding, drilling, chopping, stirring).
3. Orchestration: A task-specific orchestrator tracks recognized activities, timestamps, and user skill levels. It maintains counters to determine the current step in the procedural sequence.
4. Language Model (LM) Inference: The orchestrator prompts a language model when an activity is recognized, a user comment/question is detected, or a timer elapses. The prompt includes the task description, orchestrator state, and the five most recent dialogue turns.
5. Output: The LM generates a response, which is converted to speech via Text-to-Speech (TTS) and streamed back to the watch for low-latency feedback.

Tasks and Data Generation

The system was evaluated on two distinct procedural tasks:

• Furniture Assembly: A table assembly task involving sanding, flipping, screwing frames (vertical), and screwing legs (horizontal).
• Cooking: A soup-making task involving chopping, grating, heating, pouring, and stirring at specific intervals.

To train the conversational agent, a rule-based conversation generator created a multi-turn dataset of 1,000 conversations per task. This generator simulates user behavior based on a skill level (0–1), introducing mistakes (e.g., drilling too early, forgetting to stir) and user questions. It also simulates user challenges to corrections and verbal acknowledgments.

Activity Recognition Model

The activity recognizer is a dual-encoder neural network:

• Audio Encoder: Uses a CNN10 PANN architecture.
• IMU Encoder: Uses an Attend&Discriminate model with convolution, self-attention, recurrent, and temporal attention layers.
• Training: Data was sourced from SAMoSA, Adaimi, and BoxLift datasets, supplemented by author-collected data. The model was trained for 200 epochs with a batch size of 8.

Language Model and Finetuning

The authors experimented with three Qwen2.5 models (0.5B, 1.5B, and 3B parameters).

• Finetuning Strategy: The models were finetuned using Low Rank Adaptation (LoRA). A key innovation is User Whim Agnostic (UWA) finetuning. Standard training might wait for a user to say "Okay" before the assistant speaks. UWA adjusts the model to predict the next assistant dialogue before the user makes a trivial statement, preserving the assistant's proactive nature.
• Deployment: All models (except the activity recognizer) run as Docker microservices on a Dragonwing IQ9 processor using the Qualcomm AI Engine Direct SDK.

Key Contributions

1. On-Device Proactive Assistant: The design of a real-time assistant that provides comprehensive step-by-step guidance using only lightweight, privacy-preserving modalities (audio and IMU), eliminating the need for video or cloud processing.
2. Synthetic Conversation Dataset: A method for generating timestamped, multi-turn conversation datasets where an assistant guides a user, corrects mistakes, and answers questions, tailored to specific procedural tasks.
3. Finetuning for Proactivity and Precision: A finetuning approach (including UWA) that significantly improves the model's ability to limit unnecessary dialogue while enhancing answer accuracy.

Results

Activity Recognition

The activity recognizer achieved varying levels of precision and recall.

• Challenges: "Screwing" was sometimes misclassified as "sanding" or "drilling," and "stir-frying" was occasionally detected as "chopping."
• Impact: These errors can lead to incorrect guidance, suggesting that class-specific probability thresholds may be necessary in future iterations.

Conversational Assistant Performance

Evaluation metrics included SentenceBERT, entailment scores, and an LLM-as-a-judge framework (using GPT-5).

• Pretrained vs. Finetuned: Pretrained models performed adequately on key instructions and mistake corrections (which were explicitly in the prompt) but struggled with answering user questions and often generated verbose, irrelevant, or hallucinated messages (e.g., "Let's proceed...").
• Improvements via Finetuning:
  • Recall: A 150% increase in the recall of answers.
  • Precision: A 50% increase in precision, indicating a significant reduction in unnecessary or irrelevant dialogue.
  • Quality: A 55% improvement in LLM-judged ratings compared to pretrained counterparts.
• Comparison with GPT-5: While GPT-5 passed correctness thresholds for most messages, its responses were qualitatively too long and verbose for objective guidance. The finetuned small models provided succinct, objective guidance suitable for edge deployment.

Significance and Claims

The paper claims to present the first proactive conversational assistant that provides comprehensive, step-by-step guidance for procedural tasks using only lightweight modalities (audio and IMU) while running fully on the edge.

The authors emphasize that by moving away from video and cloud dependency, the system addresses privacy concerns and computational constraints, making it viable for widespread adoption on comfortable, everyday wearables like smartwatches. The study demonstrates that with specific finetuning strategies (like UWA), small language models can effectively replace larger, cloud-based models for this specific domain, offering a balance of proactive guidance, accuracy, and efficiency.

Limitations

The authors acknowledge several limitations:

• Wrist Assumption: The system assumes the dominant wrist is the right wrist (true for ~90% of the population).
• False Alarms: The activity recognizer may trigger false positives in non-task environments (e.g., turning a door knob detected as turning a stove knob).
• Context Limits: Compared to video, audio/IMU has limited context awareness (e.g., distinguishing between chopping radishes vs. carrots is harder).
• Safety: The system lacks safety guardrails to physically prevent user injury, only offering verbal warnings.
• Generalization: A model finetuned for one task does not perform better than a pretrained model on a different task; cross-task generalization requires multi-task finetuning.