💻 computer science

Vision-Based Safe Human-Robot Collaboration with Uncertainty Guarantees

This paper proposes a vision-based framework for human-robot collaboration that ensures certifiable safety by integrating aleatoric uncertainty estimation and out-of-distribution detection to generate conformal prediction sets for human motion with guaranteed probabilistic confidence.

Original authors: Jakob Thumm, Marian Frei, Tianle Ni, Matthias Althoff, Marco Pavone

Published 2026-05-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jakob Thumm, Marian Frei, Tianle Ni, Matthias Althoff, Marco Pavone

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot working side-by-side with a human, like a dance partner. For the dance to be safe, the robot needs to know exactly where the human is and, more importantly, where they are going to be in the next few seconds. If the robot guesses wrong, it might bump into the person.

This paper presents a new "safety brain" for robots that uses cameras to watch humans and predict their moves with a very high level of confidence. Here is how it works, broken down into simple concepts:

1. The Problem: Guessing is Risky

Current robots often rely on expensive, wired suits with markers to track human movement. If they try to use regular cameras (like on a phone), they often have to guess.

The Old Way: "The human might move 1.6 meters per second in any direction." This is like a robot assuming a human could suddenly teleport or run at super-speed in any direction just to be safe. It's so cautious that the robot stops working almost all the time.
The New Way: The authors want the robot to say, "I'm 99% sure the human will be here, and I have a mathematical guarantee that I won't be wrong."

2. The Solution: A Three-Step Safety Pipeline

The authors built a system that acts like a highly skilled, cautious detective.

Step A: Seeing with "Fuzzy" Eyes (Uncertainty-Aware Vision)
Instead of just saying, "The human's hand is at point X," the system says, "The hand is at point X, but I'm a little blurry, so it could actually be anywhere inside this small bubble around X."

Analogy: Imagine looking at a friend through a slightly foggy window. You know roughly where they are, but you acknowledge the fog. The robot calculates the size of that "fog bubble" (uncertainty) for every body part.

Step B: Predicting the Future with a "Safety Net" (Conformal Prediction)
The robot looks at the history of where the human has been and predicts where they will go next. But instead of drawing a single line for the future path, it draws a safety bubble around that path.

The Magic Trick: They use a statistical method called "Conformal Prediction." Think of this as a "confidence guarantee." The system calibrates itself so that it can promise: "I guarantee that 99% of the time, the human's actual hand will be inside this bubble."
The Result: These bubbles are much smaller and tighter than the giant, conservative "run anywhere" bubbles used by older robots. This means the robot can move closer and work faster without being dangerous. In their tests, these new safety bubbles were 11 times smaller than the old standard, making the robot much more efficient.

Step C: Handling the "Weird Stuff" (OOD Detection)
What happens if the camera sees something it has never seen before? Maybe the lighting changes, or a person walks in wearing a strange costume.

The Old Problem: If a robot's vision gets confused, it might freeze or crash because it doesn't know what to do.
The New Fix: The system has a "lie detector" (Out-of-Distribution detection). If the camera sees something weird, the system says, "I don't trust this image." Instead of panicking, it smoothly reuses its last known good prediction to keep the robot moving safely until the vision clears up.
The Result: In real-world tests, this trick reduced the number of times the robot had to stop or fail by 36%.

3. Real-World Testing

The team tested this on a real Franka robot arm working with a human.

They used a standard camera (Intel RealSense).
The robot successfully stopped before the human could reach it, proving it was safe.
The system handled "bad" camera inputs gracefully without crashing the whole operation.

Summary

This paper introduces a way for robots to "see" humans and predict their moves with a mathematical guarantee of safety.

It admits when it's unsure (calculating "fog bubbles").
It creates tight, guaranteed safety zones (conformal sets) that are much smaller than old methods.
It keeps working smoothly even when the camera gets confused (handling "weird" inputs).

The result is a robot that is safer, less likely to stop unnecessarily, and ready to work alongside humans in factories or homes.

Technical Summary: Vision-Based Safe Human-Robot Collaboration with Uncertainty Guarantees

Problem Statement

As autonomous robots increasingly operate in human-centered environments (industry, healthcare, households), ensuring safety requires accurate perception of human poses, reliable motion prediction, and provable safety guarantees against collisions. Current safe Human-Robot Collaboration (HRC) approaches face three critical limitations:

Dependency on Marker-Based Systems: Provably safe methods often rely on marker-based tracking, which limits deployment flexibility.
Inadequate Uncertainty Handling: Vision-based methods using RGB-D cameras often assume fixed maximum estimation errors or provide uncertainty estimates lacking formal guarantees. Furthermore, they frequently fail catastrophically on Out-of-Distribution (OOD) inputs, a risk not captured by standard aleatoric uncertainty.
Conservative Motion Models: Traditional safety frameworks use simple, non-data-driven motion models (e.g., assuming a maximum velocity of 1.6 m/s in any direction per ISO 13855:2010) to define reachable sets. These models are highly conservative, leading to inefficient robot operation. Conversely, state-of-the-art motion predictors lack heteroscedastic uncertainty estimates, end-to-end uncertainty propagation, and conformal prediction guarantees.

Methodology

The authors propose a unified framework for vision-based human pose estimation and motion prediction that integrates uncertainty estimation with conformal prediction to provide certifiable safety guarantees. The pipeline consists of four main stages:

1. Uncertainty-Aware 3D Pose Estimation

2D Estimation: The system adapts YOLO26 to output not only the 2D mean joint positions but also the covariance matrix for each joint in two calibrated camera frames. This covariance is derived from the scale of the Gaussian distribution in the flow-based regression model.
Triangulation: 3D joint positions are computed via standard linear triangulation.
3D Covariance Propagation: Cross-covariances are estimated from residual correlations over a calibration set. The final 3D covariances are obtained through first-order propagation, augmented with a small isotropic term to account for systematic reconstruction errors (e.g., calibration imperfections).

2. Uncertainty-Aware 3D Motion Prediction

Architecture: The model extends the Discrete Cosine Transform (DCT) Transformer architecture. It processes pose and covariance inputs along the temporal axis.
Frequency Separation: Low-frequency components (smooth global trends) and high-frequency components (rapid, fine-scale variations) are processed via dedicated Multi-Layer Perceptron (MLP) paths after the attention module before being recombined via inverse DCT.
Uncertainty Integration: Uncertainty inputs are scaled by a learnable factor and added to pose features.
Covariance Prediction: To ensure stability and validity, the model predicts covariance matrices using Cholesky factorization.
Training Strategy: A stable multi-phase optimization is employed: (i) training a deterministic base predictor, (ii) decoupled learning of the uncertainty head with a frozen backbone, and (iii) end-to-end fine-tuning with a gradually reduced weighting factor ( $\lambda$ ) to balance accuracy and calibration. The loss function combines multivariate Gaussian Negative Log-Likelihood (NLL) and an $L_1$ pose loss.

3. Conformal Prediction Sets

To convert uncertainty estimates into certifiable safety bounds, the framework constructs conformal prediction sets:

Non-conformity Measure: A score is defined based on the Euclidean distance between the predicted and ground-truth positions, normalized by the square root of the maximal eigenvalue of the predicted covariance.
Set Construction: Using a calibration set, a threshold is determined such that the true joint position falls within the predicted sphere with probability $1-\epsilon$ .
Temporal Extension: To cover continuous time, the radius of the prediction sphere is extended by $\Delta t \cdot v_{max}$ (where $v_{max} = 1.6$ m/s) for time steps between discrete predictions.
Reachable Occupancies: The SARA tool is used to expand these joint-level spheres into full-body reachable occupancies for safety shielding.

4. OOD Detection and Handling

To address the failure of models on unseen data:

Detection: The Sketching Lanczos Uncertainty (SLU) method is applied to both the 2D pose estimation and motion prediction networks to detect OOD inputs. Reduced models (predicting only head and hand positions) are used for efficiency.
Graceful Degradation: If an input is flagged as OOD, the pipeline does not halt. Instead, it reuses the previous valid motion prediction to fill the pose buffer. The system only resumes accepting new image-based predictions after a sequence of $N_{req}$ consecutive valid observations, preventing feedback loops of invalid data.

Key Contributions

The paper presents a framework for safe HRC featuring:

End-to-End Uncertainty Propagation: A pipeline that estimates 3D human motion with heteroscedastic aleatoric uncertainty propagated from 2D estimation through triangulation to motion prediction.
Conformal Prediction Sets: The generation of prediction sets for human poses that offer valid, high-confidence guarantees ( $1-\epsilon$ ), directly integrable with safety shields like SARA.
Continuous OOD Handling: A method to handle Out-of-Distribution inputs in a continuous prediction setting by reusing past valid predictions, ensuring pipeline continuity without catastrophic failure.

Experimental Results

The framework was evaluated on the Human3.6M dataset and in a real-world HRC setting with a Franka Emika robot.

Prediction Accuracy: On the Human3.6M benchmark, the model's stage 1 (deterministic) performance outperformed state-of-the-art baselines (HisRep, ST-DGCN, etc.). However, the final model (with uncertainty prediction) showed a slight increase in Mean Per Joint Position Error (MPJPE) compared to stage 1, a trade-off for providing uncertainty estimates.
Conformal Set Efficiency: Compared to the standard ISO 13855:2010 constant velocity model, the proposed conformal prediction sets achieved higher coverage (98.25% vs. 97.93%) while reducing the volume of the predicted reachable set by a factor of 11.
OOD Handling Efficacy: In the full pipeline evaluation, the proposed OOD handling mechanism reduced the rate of invalid pose buffers by 36.0% (increasing valid motion predictions) with only a marginal increase in MPJPE (2.6%).
Real-World Deployment: Integrated with the SARA shield, the system successfully operated on a Franka Emika robot. In speed and separation monitoring mode, the robot consistently stopped before the human operator could reach it, demonstrating the practical viability of the certifiable safety guarantees.

Significance and Claims

The authors claim that their work bridges the gap between high-accuracy data-driven motion prediction and rigorous safety requirements. By combining conformal prediction with OOD detection, the framework allows robots to operate with certifiable safety guarantees in unseen situations without relying on the overly conservative assumptions of traditional motion models. The reduction in prediction volume by a factor of 11 suggests that this approach can significantly improve robot efficiency and fluidity in collaborative tasks while maintaining safety. The graceful degradation strategy ensures robustness in real-world deployments where sensor data may occasionally fall outside the training distribution.