Technical Summary: Immersive and Wearable Thermal Rendering for Augmented Reality

Problem Statement

Augmented Reality (AR) systems overlay digital information onto the user's physical environment, requiring users to interact simultaneously with real and virtual objects. While visual and auditory feedback in AR have advanced, haptic and thermal feedback remain underexplored. Existing thermal feedback systems, often designed for Virtual Reality (VR), face significant challenges in AR contexts:

1. Manual Dexterity: Devices covering fingertips or requiring handheld controllers interfere with grasping, manipulation, and the natural thermal sensing of physical objects.
2. Real-World Perception: Occluding the skin with a thermal device blocks access to real-world temperature cues, diminishing the user's ability to perceive the physical environment.
3. Coherence: Static thermal stimuli fail to match the dynamic nature of AR interactions, where contact locations and thermal states change over time.

The core problem is designing a thermal feedback system that preserves manual dexterity, maintains access to real-world thermal information, and provides coherent virtual temperature sensations without obstructing natural object interaction.

Methodology

Design Considerations

The authors propose three AR-specific design considerations implemented in a novel proof-of-concept prototype:

1. Indirect Feedback: Relocating thermal actuation away from the fingertips to the palm to preserve dexterity and direct thermal sensing at the contact points.
2. Active Thermal Passthrough: Using outward-facing sensors to measure the temperature of contacted physical surfaces and actively rendering that temperature to the skin via inward-facing actuators, rather than passively transmitting heat.
3. Spatiotemporal Rendering: Utilizing independently controlled thermal actuators to create thermal sensations that vary across space and time, matching dynamic AR interactions.

Prototype Device

The device is a palm-mounted wearable featuring:

• Actuators: A 3×3 grid of 6.5 mm × 6.5 mm Thermoelectric Modules (TEMs) arranged on the palm. Each module is individually controlled to provide heating or cooling.
• Sensing: Outward-facing 10 kΩ thermistors (covered by thermally conductive silicone) and reflective phototransistors for contact detection. These measure the temperature of physical surfaces and detect when the palm is in contact.
• Thermal Management: A flexible silicone base with internal water-cooling channels to manage heat buildup and maintain a neutral baseline temperature (30°C) when not active.
• Control: A software PID controller with an ADN8833 TEM driver and differential filtering to ensure stable temperature control. The system achieves a physical surface rendering delay of approximately 55 ms.

Experimental Evaluation

Five controlled human-subject experiments were conducted with 12 participants to validate the design considerations:

1. JND Testing: Measured the Just Noticeable Difference (JND) for warming and cooling stimuli on the palm, both globally (all actuators) and locally (single row).
2. Indirect Feedback (User Experience): Compared user ratings of immersion, realism, and comfort when visual contact occurred at the fingertips but thermal feedback was delivered to the palm versus direct palm contact.
3. Active Thermal Passthrough: Evaluated the accuracy of distinguishing temperatures between a physical object and a virtual replica, and between a bare hand and the device-covered hand.
4. Spatial Patterns: Tested participants' ability to discriminate between six different spatial thermal patterns (rows and columns) and their impact on immersion and realism during static virtual object contact.
5. Moving Thermal Rendering: Compared immersion and realism ratings between static thermal stimuli and dynamic stimuli that moved across the palm in correspondence with hand motion across a virtual surface.

Key Results

• Perceptual Sensitivity: Participants demonstrated high sensitivity to thermal changes on the palm. The average JND was approximately 3.88°C for localized warming and 2.51°C for global warming.
• Indirect Feedback Trade-offs: While indirect feedback (palm actuation during fingertip visual contact) significantly reduced perceived realism (average score 2.88 vs. 6.17 for palm contact), it did not significantly reduce immersion or comfort. Users remained immersed despite the sensory mismatch.
• Active Thermal Passthrough: The device successfully supported temperature discrimination. Participants achieved 95.0% accuracy when comparing virtual stimuli and 90.1% accuracy when comparing bare skin to device-mediated stimuli. Realism scores for these interactions averaged 5.75/7.
• Spatiotemporal Rendering:
  • Participants could discriminate spatial thermal patterns with high accuracy (77.8% for warm, 78.9% for cool).
  • Moving thermal rendering significantly outperformed static rendering. Moving stimuli increased average immersion from 5.13 to 6.42 and realism from 3.46 to 6.13.

Significance and Claims

The paper claims that effective AR thermal haptics require a shift from simply adding temperature cues to strategically balancing physical interaction, real-world perception, and virtual sensation. The authors assert that their three design considerations are viable strategies:

1. Indirect Feedback is a valid alternative to fingertip actuation, preserving manual dexterity and immersion even if it sacrifices some degree of sensory realism.
2. Active Thermal Passthrough is essential for wearable AR devices covering sensitive skin areas, as it mitigates the perceptual cost of occlusion by restoring access to real-world thermal information.
3. Spatiotemporal Rendering significantly enhances the quality of AR interactions. Dynamic thermal cues that align with contact location and motion are superior to static stimuli, improving both immersion and realism without requiring additional vibrotactile modalities.

The work concludes that these findings provide a framework for designing thermal haptics that support the unique "mixed" nature of AR, where users must fluidly transition between physical and virtual entities.

Limitations

The authors acknowledge that the current prototype is tethered to bench-top electronics and a water reservoir, limiting mobility to a tabletop workspace. They also note variability in individual TEM response times due to uneven contact pressure across the palm, suggesting a need for better contact pressure management or adaptive control gains in future iterations.