Robust Assembly State Reasoning from Action Recognition for Human-Robot Collaboration

Technical Summary: Robust Assembly State Reasoning from Action Recognition for Human-Robot Collaboration

Problem Statement
In the context of Industry 5.0 and Human-Robot Collaboration (HRC), a critical challenge remains: enabling robot partners to accurately perceive the current stage of a collaborative assembly task based on Human Action Recognition (HAR) inputs. While HAR systems can often identify the type of action being performed (e.g., "screwing"), they frequently fail to map these actions to the specific state of a complex assembly sequence. This is particularly difficult in realistic scenarios where tasks involve repeated actions of the same primitive type, non-linear user behaviors (skipping or reordering steps), and imperfect sensor data. Current approaches often rely on rigid, handcrafted logic or specific cues that reduce interaction fluidity, while advanced methods lack robust comparisons across diverse, noisy, and variable task environments.

Methodology
The authors systematically investigate and compare five distinct methods for tracking assembly states given action recognition inputs. The study utilizes two diverse open-source datasets:

1. HA4M: An epicyclic gear train assembly with 13 action types and 19 states, featuring repeated actions.
2. IKEA ASM: Three furniture assembly tasks (Coffee Table, Side Table, TV Bench) characterized by natural variations in action order, error correction, and repeated cyclic actions.

The five evaluated approaches are:

• Baseline Logic: A deterministic algorithm that iterates through states based on matching observed actions and decrementing a time counter (initialized to the mean action duration). It advances states if the observed action matches the next expected action or if a time threshold is met.
• Probabilistic Enhanced Baseline: An extension of the baseline that incorporates a probabilistic model. Instead of a hard time threshold, it uses a Normal Cumulative Distribution Function (CDF) based on the mean and standard deviation of action durations to calculate the probability of completion, triggering a state transition when this probability exceeds 0.9.
• Hidden Markov Model (HMM): A CategoricalHMM implemented with predefined transition and emission matrices. The model assumes a known sequential structure, with transition probabilities favoring the current state (0.7) or moving to the next (0.225), and emission probabilities favoring the correct observation (0.7). This method requires no training data.
• Task-Focused LSTM: A neural network trained specifically on a single task type. It takes a sequence of one-hot encoded action inputs and outputs the current state index. It is trained on 5-second windows with 20% simulated noise.
• Action-Focused LSTM: A generic network architecture designed to be instantiated for each action in a sequence. It predicts the start and completion probabilities of specific actions based on the current action type, duration statistics, and the output of the previous action model. This aims to avoid retraining for every new task.

Experiments were conducted in two stages:

1. Simulated Inputs: The methods were tested against ground-truth action sequences with varying levels of injected noise (0% to 50%) to evaluate robustness.
2. Realistic Inputs: A pre-trained ST-GCN model was used to generate HAR predictions from skeleton data, which were then fed into the state tracking methods to simulate real-world sensor imperfections.

Key Results
The study reveals that no single method is optimal across all scenarios; performance is highly dependent on task variability and input noise levels.

• Noise Robustness: In the HA4M dataset (which has repeated actions but a fixed sequence), the Task-LSTM performed best under low-to-moderate noise (F1 score of ~0.64 at 0-0.3 noise). However, as noise increased, logic-based methods showed greater resilience.
• Variable Tasks: In the IKEA datasets (characterized by high variability and repeated cyclic actions), the Baseline and Probabilistic Baseline methods significantly outperformed the neural network and HMM approaches. The HMM, in particular, struggled to distinguish between sequential identical actions in HA4M and tended to "jump ahead" to the end of the task in IKEA scenarios.
• Realistic HAR Inputs: When using actual ST-GCN outputs (which had lower accuracy, e.g., F1 0.27 for IKEA), the Probabilistic Baseline achieved the highest F1 score for IKEA tasks (0.46), while the Task-LSTM remained superior for HA4M (0.57).
• Failure Modes: Confusion matrices indicated that neural networks and HMMs often get "stuck" in states or skip ahead when faced with repeated actions or noise. Logic-based methods were more prone to getting stuck but less prone to skipping states.

Significance and Claims
The paper claims that robust state tracking in HRC requires a nuanced approach rather than a "one-size-fits-all" solution. The authors emphasize that:

1. Context Matters: Optimal methods vary by task. Logic-based approaches are more robust in scenarios with high variability and repeated actions, while NN and HMM methods can perform well in tasks with limited variability.
2. Duration Modeling is Critical: Methods that explicitly model expected action duration (such as the Probabilistic Baseline) are essential for handling repeated actions where no additional sensing is provided to distinguish between instances.
3. Generalizability vs. Specificity: While deep learning offers high performance in controlled settings, it struggles with the noise and variance inherent in real-world HRC. The authors advocate for generalizable architectures that minimize handcrafted rules but acknowledge that current logic-based approaches offer a necessary robustness for deployment in complex, non-ideal environments.

The work concludes that accurate state prediction remains a non-trivial challenge, particularly when users deviate from ideal procedures. Future work must focus on integrating additional data modalities (such as object state tracking) to further improve robustness without sacrificing the ability to deploy systems with minimal retraining.