💻 computer science

TouchMap-OR: Multi-View 3D Mapping of Hand-Surface Contacts

TouchMap-OR is a multi-view RGB-D vision system that reconstructs identity-resolved 3D hand-surface contact histories in operating rooms by fusing articulated hand meshes, multi-person skeleton tracking, and semantic environmental modeling to infer when and where clinicians touch specific surfaces.

Original authors: Sophokles Ktistakis, Rui Wang, Bastian Grande, Hugo Sax

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sophokles Ktistakis, Rui Wang, Bastian Grande, Hugo Sax

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a busy operating room as a high-stakes dance floor. Doctors, nurses, and equipment are constantly moving, touching, and interacting. The big problem? We can't see the invisible "dance steps" that spread germs. Usually, to figure out who touched what and when, we have to hire a human observer to watch the whole thing, which is tiring, prone to mistakes, and misses a lot of details.

TouchMap-OR is a new "super-vision" system designed to solve this. Think of it as a team of six synchronized security cameras that don't just record video, but actually build a living, 3D map of the room in real-time.

Here is how it works, broken down into simple steps:

1. The "3D Puzzle" (Building the Room)

First, the system looks at the room through multiple cameras. It uses a smart AI to identify everything in the room—the patient, the monitors, the blankets, the medical tools—and builds a 3D model of them. It's like creating a digital Lego version of the operating room where every block has a label (e.g., "This is the patient's arm," "This is the IV machine").

2. The "Ghost Dancers" (Tracking People)

Next, the system tracks the doctors and nurses. It doesn't just see them as blobs; it builds a 3D skeleton for each person, following them as they move around the room. Crucially, it keeps their identities straight. If two doctors cross paths, the system knows, "That's still Dr. Smith on the left, not Dr. Jones," preventing the confusion that often happens with standard video tracking.

3. The "Magic Hands" (Reconstructing Hands)

This is the special sauce. Standard cameras often lose track of hands when they are covered by gloves, hidden behind instruments, or blocked by other people. TouchMap-OR uses a special trick: it looks at the hands from every camera angle at once and fuses them together. It reconstructs a detailed 3D model of the hand (fingers, palm, everything) even if it's partially hidden. It's like having a 3D printer that instantly builds a perfect model of a hand based on scattered clues from different angles.

4. The "Contact Detective" (Finding the Touch)

Finally, the system compares the 3D hand models against the 3D map of the room. It asks a simple question: "Is the hand close enough to the surface to be touching it?"

If a doctor's hand gets within a few inches of the patient's arm, the system logs it.
It records who touched what, where exactly, and for how long.

What Did They Find?

The researchers tested this system in a real operating room during three actual medical procedures (anesthesia inductions). They compared their system against other existing tracking methods and found:

Better Identity: It was much better at keeping track of which doctor was which person, even when they moved around each other.
Better Touch Detection: It successfully identified about 75% of the actual hand-touches that happened, which was a huge improvement over the other methods.
The "Clutter" Problem: The system is great at knowing a hand touched something (like a blanket or a machine), but sometimes it gets confused about exactly which specific object it was if the objects are very close together (like a bandage right next to a blanket).

The Bottom Line

TouchMap-OR is a tool that turns a chaotic, blurry video of a surgery into a clear, 3D story of who touched what. It doesn't just say "a hand was here"; it says "Dr. Smith touched the patient's arm for 10 seconds." This creates a detailed map of interactions that could help scientists understand how germs move through a hospital, but the paper focuses strictly on building this map, not on using it to change medical rules yet.

Technical Summary: TouchMap-OR

Problem Statement

Pathogen transmission during medical procedures is significantly driven by hand–surface interactions between clinicians, patients, and medical equipment. Current infection-prevention practices rely on manual observation, which is labor-intensive, prone to bias, and lacks the temporal resolution to reconstruct detailed contact histories. While recent automated systems can detect hand hygiene events or dispenser usage, they fail to observe the specific sequences of contact between hands, surfaces, and patients that govern transmission pathways. Furthermore, existing computer vision approaches for hand–object contact often assume visible skin, minimal occlusion, and controlled environments—conditions rarely met in operating rooms where surgical gloves obscure skin cues and hands are frequently occluded by instruments and personnel.

The core challenge addressed is identity-resolved hand–surface interaction reconstruction in real operating rooms: determining who touched what surface, where, and for how long, despite heavy occlusion, clutter, and the presence of multiple clinicians.

Methodology: TouchMap-OR

TouchMap-OR is a multi-view RGB-D vision system designed to reconstruct metric 3D representations of operating rooms that jointly model clinicians, articulated hands, and semantic environment structures. The pipeline consists of four main stages:

1. Semantic 3D Room Modeling

To localize contact events, the system constructs a unified semantic 3D model of the operating room.

Segmentation: Calibrated RGB-D cameras capture streams segmented in 2D using SAM2 with sparse, hand-annotated point prompts for key surfaces (e.g., patient, monitors, blankets, tubes).
Fusion: Segmented instances are backprojected to 3D. Points from all cameras are fused in a common world frame via voxel down-sampling (10 mm resolution) and majority voting over semantic labels to produce a unified semantic point cloud ( $S_t$ ). This cloud is indexed by a KD-tree for efficient nearest-surface queries.

2. Per-Camera Detection and Metric Alignment

Each camera independently estimates body and hand poses.

Body Pose: RTMO is used for multi-person 2D skeleton estimation.
Hand Geometry: WiLoR, a MANO-based estimator, predicts articulated hand meshes from single RGB images.
Metric Alignment: To place hand meshes in the metric camera coordinate system, the system fits the reconstruction to the corresponding depth map via a Sim(3) alignment (scale, rotation, translation) using RANSAC to reject occlusions. This yields metrically aligned hand meshes ( $H^{(i,t)}_{cam}$ ) and extracts hand anchors (palm center, fingertips) with reliability scores based on alignment residuals.

3. Multi-View Association and 3D Person Tracking

The system generates globally consistent multi-person 3D skeleton tracks with stable identities.

Association: 3D tracks are projected to image planes and matched against 2D detections using Mean Per-Joint Pixel Error (MPJPE). The Hungarian algorithm solves for one-to-one matches.
3D Reconstruction: Matched joints are reconstructed via weighted non-linear least squares triangulation across views, enforcing epipolar consistency.
Depth Lifting: Joints that cannot be triangulated are recovered from RGB-D observations, validated by anatomical plausibility checks (bone length constraints).
ID Management: A track life-cycle manages initialization, confirmation, and removal. A "stitching" procedure merges fragmented tracks caused by occlusions by accumulating re-association votes over time to establish a one-to-one mapping between transient and persistent IDs.

4. Hand Fusion and Contact Detection

Hand Fusion: Metrically aligned hand meshes from all views are transformed to the world frame and grouped using DB-SCAN on palm centers. The mesh with the lowest alignment residual is selected as the representative fused hand.
Person Association: Fused hands are greedily associated with tracked persons based on distance to side-specific body joints (wrist, elbow, shoulder), enforcing temporal persistence to maintain continuity.
Contact Inference: The system queries the semantic point cloud for the nearest surface to hand anchors. A contact is registered when the distance falls below a threshold ( $\tau_{on}$ ). To ensure temporal stability, an exponential moving average (EMA) smooths coordinates, and a hysteresis rule (using $\tau_{on}$ and a higher $\tau_{off}$ ) prevents flickering. Consecutive active frames are merged into contact episodes specifying the clinician ID, hand side, surface label, start/end times, and contact point.

Key Contributions

Identity-Resolved 3D Contact Reconstruction: The introduction of a multi-view RGB-D pipeline that reconstructs hand–surface contact episodes between multiple clinicians and semantically labeled operating room surfaces in real-world settings.
Multi-View 3D Hand Reconstruction and Person Association: A method to fuse MANO-based hand reconstructions across cameras and associate each hand with a persistent clinician identity using multi-view 3D skeleton tracking, handling occlusions and identity switches.
Semantic 3D Contact Mapping: The mapping of reconstructed hand trajectories to semantically labeled scene surfaces, enabling the reconstruction of temporally consistent contact episodes that detail which clinician touched which surface and for how long.

Experimental Results

The system was evaluated on a dataset of three real anesthesia induction procedures involving 2–5 clinicians, captured by six synchronized RGB-D cameras. The dataset includes 1,616 manually annotated hand–surface contact episodes.

Tracking Performance: TouchMap-OR achieved an IDF1 score of 0.48, comparable to the MVPose baseline (0.49), but significantly reduced identity switches to 108 (vs. 236 for MVPose and 547 for 4D Association), demonstrating superior temporal identity consistency.
Contact Detection: TouchMap-OR achieved a binary contact F1-score of 0.75 and an IoU of 0.60, outperforming baselines (MVPose: 0.56 F1; 4D Association: 0.29 F1).
Semantic Attribution: The system achieved a semantic contact F1 of 0.37, indicating improved ability to attribute contacts to specific surfaces compared to baselines (0.22 for MVPose).
Identity Accuracy: The system achieved 0.96 identity attribution accuracy, confirming reliable linking of contact events to specific clinicians.

Significance and Claims

The paper claims that TouchMap-OR provides a computational basis for analyzing transmission pathways by reconstructing detailed contact histories between clinicians, instruments, and patient surfaces. The authors emphasize that while raw tracking metrics are important, accurate modeling of articulated hands combined with a semantically structured 3D scene is more critical for interaction understanding than maximizing tracking metrics alone.

The system enables the reconstruction of contact episodes in cluttered, multi-person clinical environments where traditional methods fail due to occlusion and lack of semantic context. By identifying "who touched what, where, and for how long," the system offers a scalable foundation for quantifying hand-associated transmission networks, potentially supporting data-driven infection prevention strategies and offering new perspectives on human–environment interaction dynamics in healthcare.

The authors modestly acknowledge limitations, noting that semantic attribution can be ambiguous in cluttered scenes (e.g., distinguishing between a bandage and a nearby blanket) and that thin or transparent items (like IV lines) may be incompletely segmented. However, the system demonstrates stable performance under typical operating room conditions, including occlusions and close proximity between clinicians.