Exploring Immersive Social-Physical Interaction with Virtual Characters through Coordinated Robotic Encountered-Type Contact

1. Problem Statement

Virtual Reality (VR) experiences are currently dominated by audio-visual stimuli, lacking the physical engagement necessary for truly immersive interactions. While haptic technologies exist, many rely on localized or abstract feedback (e.g., vibration) that fails to replicate the kinesthetic and social nuances of real-world touch. Specifically, Social-Physical Interactions (interactions where social meaning and physical contact are intrinsically intertwined, such as a fist bump or handover) are difficult to render convincingly in VR. Existing Encountered-Type Haptic Displays (ETHDs) often focus on static objects and lack the dynamic capabilities required for these complex social gestures. Furthermore, integrating robotics into VR often breaks immersion if the robot's physical form is visible, yet hiding the robot requires precise synchronization between physical and virtual domains to avoid dissonance.

2. Methodology

The authors developed ETHOS (Encountered-Type Haptics for On-demand Social interaction), an experimental platform designed to bridge the gap between physical robotics and immersive VR.

A. System Architecture

ETHOS integrates three subsystems:

1. Physical Subsystem: A torque-controlled 7-DoF KUKA LBR iiwa 7 R800 robotic manipulator mounted horizontally. It is equipped with interchangeable, custom-molded props: a 3D-printed baton (for handovers), a silicone fist, and a silicone open hand (for fist bumps and high fives). A six-axis force-torque sensor monitors interaction forces.
2. Virtual Subsystem: A Meta Quest 3 HMD running a Unity-based VR environment (a forest setting). An anthropomorphic virtual character provides social cues (gaze, gestures) to guide the interaction.
3. Coordination Subsystem:
   • Registration: Uses a ChArUco board and the HMD's internal cameras for fiducial-based physical-virtual registration, establishing a shared coordinate anchor.
   • Communication: Utilizes UDP unicast with Protocol Buffers for low-latency, bidirectional state exchange between the HMD (~90 Hz) and the robot controller (1000 Hz).
   • Safety: Implements continuous monitoring of user position relative to the robot's workspace to halt motion if collisions are imminent.

B. Control Strategies

The study compared two robotic control strategies against a purely virtual baseline:

• No Physicality (NP): A purely virtual baseline with no haptic feedback.
• Static Physicality (SP): The robot positions a prop at a fixed location. The virtual character aligns with the prop, and the robot remains stationary during contact.
• Dynamic Physicality (DP): The robot tracks the user's hand in real-time. It generates a trajectory that blends an initial midpoint path with the user's current hand position using an exponential weighting function. This allows the prop to move and impact the user, simulating natural touch dynamics.

C. Experimental Design

• Interactions: Three social-physical tasks: Object Handover, Fist Bump, and High Five.
• Study Design: A mixed factorial design with 55 participants. Each participant experienced one interaction type (between-subjects) under all three physicality conditions (within-subjects).
• Metrics:
  • Technical: Spatial alignment error (mm) and interaction latency (ms).
  • Experiential: Multimodal Presence Scale (MPS) covering Physical, Social, and Self-presence; plus custom ratings for Realism, Enjoyability, Comfort, and Character Connection.

3. Key Contributions

1. ETHOS Platform: The design and implementation of the first ETHD capable of supporting dynamic, social-physical interactions in VR, successfully concealing the robot behind the virtual environment.
2. Technical Characterization: Quantification of the system's spatial alignment (5.09 ± 0.94 mm) and interaction latency (28.58 ± 31.21 ms), confirming performance is generally below perceptual thresholds.
3. Empirical Evaluation: A comprehensive user study demonstrating that adding physical contact (even static) significantly enhances the user experience compared to purely virtual interactions, and analyzing the nuances of dynamic vs. static rendering.

4. Results

Technical Performance

• Spatial Alignment: The average registration error was 5.09 mm, well below the 1 cm threshold often cited as the limit for perceptible misalignment.
• Latency: The mean interaction latency was 28.58 ms. While the mean is below perceptual thresholds for most haptic tasks, the standard deviation (±31.21 ms) indicates that in some trials, latency exceeded the strict 50 ms threshold for haptic feedback, suggesting potential for occasional perceptible lag.

User Study Findings

• Impact of Physicality: Both Static (SP) and Dynamic (DP) conditions significantly outperformed the No Physicality (NP) baseline across all metrics (Presence, Realism, Enjoyability, Connection).
  • Presence: Physical contact significantly improved Total, Physical, Social, and Self-presence.
  • Realism & Connection: Participants reported the NP condition as "unsatisfying" and "disappointing," while physical conditions felt "grounded" and "real."
• Static vs. Dynamic:
  • Total Presence: DP yielded a statistically significant but modest increase over SP.
  • Other Metrics: No significant differences were found between SP and DP for Realism, Enjoyability, Comfort, or Connection.
  • Interaction Specifics: The "High Five" interaction did not show significant realism improvements in DP, likely due to the limitations of the silicone hand prop's fidelity.
• Comfort: No significant differences in comfort were found across conditions, though the study noted that comfort assessment was limited to a single item.

5. Significance and Implications

• Validation of ETHD for Social HRI: The study proves that Encountered-Type Haptic Displays can successfully mediate socially meaningful physical interactions in VR, even when the robot is hidden from the user.
• Necessity of Physicality: The results strongly suggest that purely virtual social interactions lack the "presence" and "realism" required for meaningful engagement. Physical contact is a critical driver for immersion in social contexts.
• Design Insights for Dynamic Rendering: While dynamic physicality (DP) improved total presence, it did not consistently outperform static physicality in other areas. This highlights that control fidelity (timing, trajectory smoothness, and force compliance) is as critical as the mere presence of motion. Simple dynamic tracking is insufficient; future systems require interaction-specific trajectory generation and adaptive force control.
• Future Directions: The work points to the need for better prop materials (to match virtual counterparts), more sophisticated behavioral realism in virtual characters, and personalized interaction strategies to enhance user agency and comfort.

In conclusion, ETHOS demonstrates that combining precise robotic contact with immersive VR creates a powerful medium for social-physical interaction, offering a pathway to more engaging training, rehabilitation, and entertainment applications.