Personalized and Robust Proactive Robot Assistance with Uncertainty-Guided LLM Reasoning

Technical Summary: GLOBE – Personalized and Robust Proactive Robot Assistance

Problem Statement
Proactive robot assistance in household environments requires the accurate prediction of human activities and subsequent object usage under dynamic and noisy conditions. Existing approaches, such as spatio-temporal models (e.g., STOT, SLaTe-PRO), often rely on complex architectures that are computationally expensive and sensitive to environmental variability. These methods frequently struggle with real-world noise, such as object displacements caused by pets, toddlers, or incidental human actions, which are not explicitly modeled in standard datasets. Furthermore, there is a need for frameworks that balance predictive accuracy with computational efficiency and robustness for long-term deployment.

Methodology: The GLOBE Framework
The authors propose GLOBE (uncertainty-Guided LLM reasoning for routine Object prediction via Behavioral tEmporal modeling), a lightweight hybrid framework designed to predict object usage for proactive assistance. The system operates through a two-phase process: an observation (training) phase and an assistance (inference) phase.

1. Sequential Prediction via n-gram Markov Models:

   • Activity Prediction: The system trains n-gram Markov models on sequential user activity data. It calculates transition probabilities based on recent activity history ($c_{act}$). To handle limited context, the model employs a "back-off" strategy, progressively using shorter history sequences if the longest observed sequence was not encountered during training.
   • Object Prediction: Object usage is predicted conditioned on the predicted activity ($\hat{a}_t$) and recent object history ($c_{obj}$) within that specific activity context. The model restricts predictions to the set of objects co-occurring with the activity, ensuring contextual consistency.

2. Uncertainty-Guided LLM Reasoning:

   • To address sparsity, ambiguity, or unseen patterns where the Markov model's confidence is low, the framework selectively invokes a pre-trained Large Language Model (LLM).
   • A confidence threshold ($d$) determines when to trigger the LLM. If the sequential model's probability falls below this threshold, the LLM is queried with the recent activity, object sequence, and a filtered set of top-$k$ candidates from the n-gram model.
   • The LLM utilizes its semantic reasoning capabilities to refine the prediction, accounting for broader contextual knowledge without being the sole decision-maker.

3. HOMER-Noise Dataset:

   • To evaluate robustness, the authors introduce HOMER-Noise, an extension of the existing HOMER+ dataset.
   • This dataset simulates realistic environmental disturbances by injecting structured object movements caused by three distinct sources: Person (reorganizing/tidying), Pet (incidental displacement), and Toddler (unpredictable random movement).
   • These disturbances are generated using LLMs to ensure semantic consistency with the household environment, creating a benchmark that distinguishes between activity-induced changes and external noise.

Key Contributions

• GLOBE Framework: A hybrid architecture combining efficient n-gram Markov modeling for routine patterns with selective, uncertainty-guided LLM reasoning for edge cases. This design avoids the computational overhead of using LLMs for every prediction.
• HOMER-Noise Dataset: A novel dataset extension that explicitly models structured environmental noise (human, pet, toddler interactions) to better simulate real-world household variability.
• Robustness and Efficiency: The approach demonstrates that lightweight sequential models, augmented by LLMs only when necessary, can achieve competitive performance with state-of-the-art (SOTA) methods while significantly reducing training time and improving resilience to noise.

Experimental Results
The framework was evaluated on both the clean HOMER+ dataset and the noisy HOMER-Noise dataset, comparing performance against SLaTe-PRO, STOT, standalone n-gram models, and zero-shot LLM baselines.

• Predictive Performance: On the clean HOMER+ dataset, GLOBE achieved a mean F1-score of 0.73, closely matching SLaTe-PRO (0.74) and outperforming the standalone n-gram model (0.72) and the zero-shot LLM (0.29). GLOBE specifically achieved the highest activity prediction F1-score (0.80).
• Computational Efficiency: GLOBE demonstrated superior training efficiency, requiring only 8.33 seconds compared to SLaTe-PRO's 11,415 seconds. While inference latency was higher (192.49s vs. 93s for SLaTe-PRO) due to selective API calls, this cost is incurred only during low-confidence scenarios.
• Robustness to Noise: Under structured noise conditions (HOMER-Noise), GLOBE consistently outperformed SLaTe-PRO in mean F1-score across Person, Pet, and Toddler noise types. Notably, GLOBE maintained stable activity prediction performance (0.80) across all noise levels, whereas SLaTe-PRO's performance degraded significantly.
• Ablation Studies: The results indicated that the framework is robust across varying noise levels in training (0.25 to 0.75) and that a confidence threshold of 0.5 offers the best balance between sequential modeling and LLM intervention.

Significance and Claims
The paper claims that GLOBE addresses critical gaps in proactive robot assistance by offering a solution that is both computationally efficient and robust to environmental variability. By leveraging the strengths of lightweight statistical models for routine behaviors and reserving LLM reasoning for uncertain situations, the framework achieves a balance between speed and adaptability. The introduction of HOMER-Noise provides a more realistic benchmark for evaluating systems intended for real-world deployment, where object states are frequently altered by factors unrelated to the primary user activity. The authors validate the system's potential through a proof-of-concept integration with a Stretch 3 mobile manipulator, demonstrating its applicability in human-robot interaction scenarios.