💻 computer science

HOPE: Hand-Object Pressure Estimation from Monocular Videos

The paper proposes HOPE, a novel framework that estimates temporally evolving per-vertex pressure and contact on a hand mesh from monocular videos by unifying diverse tactile and contact data into a shared vertex space and employing a vertex-anchored video transformer, thereby enabling robust pressure estimation for dynamic hand-object interactions beyond the limitations of prior planar or single-image methods.

Original authors: Subin Jeon, Byungjun Kim, Hanbyul Joo

Published 2026-08-07

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Subin Jeon, Byungjun Kim, Hanbyul Joo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a magic show. The magician pulls a rabbit out of a hat, and you see the movement, the color of the fur, and the way the hat tilts. But you can't feel the rabbit. You don't know if the magician is gently stroking it or squeezing it tight. In the world of robots and computers, this is a huge problem. We have cameras that can "see" hands grabbing objects, but they are blind to the invisible force of a squeeze. This field, called computer vision, tries to teach machines to understand the physical world, not just its appearance. For a long time, scientists could only guess how hard a hand was pressing if the hand was touching a flat, boring surface like a table, and even then, it was tricky. But real life is messy. We grab round apples, squish soft pillows, and use tools. To build robots that can cook, clean, or help us in virtual reality, they need to understand not just where a hand touches something, but how hard it is pressing at every single point on the skin.

Enter HOPE (Hand-Object Pressure Estimation), a new method that acts like a super-powered X-ray for the invisible force of a handshake. Instead of needing special gloves with sensors or flat pressure mats, HOPE looks at a regular video of a hand and guesses the pressure map right on the skin itself. Think of it like this: if your hand were a 3D mesh made of 778 tiny, invisible dots, HOPE tells you exactly how hard each dot is pushing against an object. The researchers found that by combining videos of hands wearing sensor gloves (which give exact pressure numbers) with videos of bare hands just touching things (which give contact clues), they could teach a computer to "feel" pressure without ever touching anything. They proved that this system works on gloves, on flat tables, and even on wild, everyday videos of people grabbing random objects, predicting both where the contact happens and how strong the squeeze is.

The Problem: Seeing is Not Feeling

Imagine trying to learn how to play the piano just by watching a video of someone else's hands. You can see their fingers moving, but you can't feel the weight of the keys or the pressure needed to make a sound. That's the challenge computers face today. They are great at spotting that a hand is near a cup, but they are terrible at knowing if the hand is gently holding it or about to crush it. Previous attempts to solve this were like trying to measure the weight of a fish by looking at a flat shadow on the wall. They worked okay if the fish was lying on a flat table, but as soon as the fish jumped into a 3D world with curves and angles, the measurements fell apart. Also, most of these methods needed the hand to be wearing a special, bulky glove with sensors, which changes how the hand looks and feels, making it hard to apply to real, bare-handed humans.

The Solution: A "Universal Translator" for Touch

The team behind HOPE came up with a clever idea: stop trying to measure pressure on the object or the table, and measure it directly on the hand itself. They treated the hand like a digital map with 778 specific "checkpoints" (called vertices).

To teach the computer, they used a "universal translator" approach. They took data from three different sources and forced them to speak the same language:

The Gloved Data: Videos of hands wearing sensor gloves that measure exact pressure (in kilopascals, or kPa). This is like having a teacher who knows the exact math.
The Flat Data: Videos of hands pressing on flat, sensor-covered tables. This is like a teacher who only knows how to press a button.
The Bare-Hand Data: Videos of people grabbing things without gloves, where we only know where they touched, but not how hard. This is like a student who knows where to put their fingers but not how much strength to use.

The magic trick was "lifting" all these different types of data onto the same 778-point map of the hand. Even though the glove data only covered the palm and fingertips, and the bare-hand data covered the whole hand, they could combine them. The bare-hand data acted as a "safety net" or a structural guide, teaching the computer that pressure can only happen where there is contact. If the computer sees a hand touching a ball, it knows pressure must be there, even if it doesn't have a specific number for how hard.

How It Works: The Time-Traveling Detective

The brain of HOPE is a special AI model called a "Vertex-anchored Video Transformer." Imagine this model as a detective who doesn't just look at a single photo, but watches the whole movie.

The Persistent Tokens: The model treats each of the 778 points on the hand as a character that stays the same throughout the video. Even if the hand moves, the "thumb-tip" character is always the thumb-tip.
The Movie Watcher: It watches the video frame by frame. It knows that pressure isn't instant; it builds up. When a hand approaches a cup, there is no pressure. When it touches, pressure starts. When it squeezes, pressure goes up. By watching the sequence of events, the model can guess the pressure better than if it just looked at one frozen picture.
The "No Contact, No Pressure" Rule: The model has a built-in rule: if the hand isn't touching the object, the pressure must be zero. This helps it ignore fake signals and focus on real interactions.

What They Found

The researchers tested HOPE on several datasets, including the "OpenTouch" dataset (gloved hands), "PressureVisionDB" (flat surfaces), and "MOW" (bare hands in the wild).

Better than the Old Ways: On the glove datasets, HOPE predicted the pressure much more accurately than previous methods that tried to guess pressure from flat images. It was especially good at pinpointing exactly which part of the hand was doing the work.
The Bare-Hand Surprise: Even though the model was mostly trained on data from gloved hands, it could successfully predict pressure on bare hands in everyday videos. This is a big deal because it means the model learned the physics of touching, not just the look of a glove.
The Power of Mixing Data: When they tested the model with and without the "bare-hand contact" data, they found that adding the bare-hand videos made the model much better at generalizing to new, unseen situations. The contact data acted as a guide, helping the model understand the shape of interactions even when it didn't have exact pressure numbers.

The Limits and the Future

The paper is careful to note that HOPE isn't perfect. It relies on a separate tool to figure out the hand's shape first; if that tool gets confused (like when the hand is hidden behind an object), HOPE might get confused too. Also, the model currently only predicts the force pushing straight into the object (normal pressure), not the sliding or twisting forces. And while it works well, it still needs more diverse data to handle every possible object in the world.

However, the results suggest a promising path forward. By treating pressure as a property of the hand itself, rather than the object, and by mixing different types of learning data, HOPE brings us one step closer to robots that can truly "feel" the world, not just see it. It's a step toward machines that can pick up a ripe tomato without squishing it, or help a human with a delicate task, all by watching a simple video.

Technical Summary: HOPE – Hand-Object Pressure Estimation from Monocular Videos

Problem Statement

Estimating physical pressure from visual observations is critical for understanding contact-rich hand-object interactions, with applications in egocentric perception, VR/AR, and robot learning. However, existing vision-based pressure estimation methods face significant limitations:

Constrained Geometries: Prior approaches (e.g., PressureVision, EgoPressure) largely rely on calibrated settings with instrumented planar surfaces, representing pressure as 2D maps tied to specific sensor layouts.
Lack of Generalization: These methods struggle to generalize to dynamic, diverse hand-object interactions involving arbitrary 3D objects, self-occlusion, and varied contact modes (e.g., grasping, tool use).
Data Modality Mismatch: Recent tactile datasets (e.g., OpenTouch) provide metric pressure on the hand but require tactile gloves, obscuring visual cues (skin deformation, wrinkles) and creating a domain gap between gloved training data and natural bare-hand interactions.

The core challenge is to estimate metric pressure and contact directly on the hand surface from monocular RGB video, generalizing across diverse objects and interaction types without relying on fixed sensor geometries.

Methodology: HOPE Framework

The authors propose HOPE, a framework that formulates pressure estimation as a hand-centric video prediction problem. The system predicts temporally evolving per-vertex normal pressure and contact directly on a MANO hand mesh.

1. Unified Annotation Pipeline

To address the heterogeneity of available data, HOPE lifts three distinct supervision sources into a shared MANO vertex space ( $V=778$ ):

Tactile-Glove Pressure (OpenTouch): Pressure from a 16×16 taxel grid on a glove is mapped to MANO vertices. Since taxels cover multiple vertices, pressures are aggregated via averaging and smoothed using geodesic Gaussian filtering.
Planar-Sensor Pressure (PressureVisionDB): Pressure from a planar sensel pad is projected onto the palm vertices of a reconstructed MANO mesh using calibrated camera-to-pad extrinsics.
Distance-Based Contact (HOI Datasets): To supervise regions not covered by sensors (e.g., dorsal hand) and to bridge the gap to bare-hand interactions, the authors incorporate binary contact labels from datasets like DexYCB and ARCTIC. A vertex is labeled as "in contact" if its distance to the object surface is below a threshold ( $\delta_c = 10$ mm).

This unification allows metric pressure labels and broad bare-hand contact data to be used jointly, treating contact as a structural prior for pressure.

2. VertexFormer Architecture

The core model is a vertex-anchored video transformer designed to reason over both the hand surface and time:

Tokenization:
- Patch Tokens: Input video frames are encoded using a frozen DINOv3 backbone.
- Vertex Tokens: Each MANO vertex is represented as a persistent token. These tokens are enriched with visual features, hand pose (from HaWoR), global orientation, and translation. Crucially, each token includes a learned identity embedding ( $e_v$ ) to anchor it to a specific vertex index, ensuring a fixed output space.
Attention Mechanisms:
- Spatial Attention: Interleaved Vertex Self-Attention (propagating contact signals across the hand topology) and Vertex-to-Patch Cross-Attention (refining visual evidence from the image grid).
- Temporal Attention: A layer that aggregates information across $T$ frames for each vertex, helping to disambiguate loading states and handle short occlusions.
Contact-Gated Pressure Head: The network outputs two heads: a contact probability ( $\hat{c}_t$ ) and a raw pressure value ( $\tilde{p}_t$ ). The final pressure prediction is gated by the contact probability: $\hat{p}_t = \hat{c}_t \odot \tilde{p}_t$ . This enforces the physical prior that pressure must vanish in the absence of contact, allowing contact-only supervision to regularize pressure prediction.

3. Training Objective

The model is trained with a unified loss function combining binary cross-entropy for contact and L1 loss for pressure. For samples with only contact labels (from HOI datasets), the pressure loss is masked out, enabling joint training on heterogeneous data.

Key Contributions

Hand-Centric Formulation: A novel approach representing pressure and contact as per-vertex fields on the MANO mesh, decoupling estimation from specific object shapes or sensor layouts.
Unified Supervision: A method to lift tactile-glove pressure, planar-sensor pressure, and distance-based contact annotations into a single vertex space, utilizing bare-hand contact data as a structural prior for pressure learning.
Vertex-Anchored Transformer: A temporal model that treats vertices as persistent tokens, aggregating visual and pose evidence over time, and employing a contact-gated head to enforce physical consistency.

Experimental Results

The framework was evaluated on three benchmarks: OpenTouch (hand-object pressure), PressureVisionDB (hand-surface pressure), and MOW/ARCTIC (dense contact).

Hand-Object Pressure (OpenTouch): HOPE outperformed baselines (PressureVision, HACO) in both contact and force metrics. Notably, it achieved a vertex-level F1 of 0.660 compared to near-zero for planar methods and 0.673 precision for contact, demonstrating superior spatial localization.
Hand-Surface Pressure (PressureVisionDB): Despite predicting in a hand-centric space rather than the sensor's native 2D space, HOPE achieved the lowest vertex-level force errors (MAE: 0.449 kPa, RMSE: 2.314 kPa) compared to planar methods, while maintaining competitive frame-level contact accuracy.
Generalization to Bare-Hand: The model successfully generalized to bare-hand egocentric and in-the-wild videos, producing dense contact and pressure predictions despite being trained primarily on gloved-hand data.
Ablation Studies:
- Data Composition: Combining OpenTouch (gloved force), PVDB (planar force), and HOI (bare-hand contact) yielded the best cross-domain generalization.
- Input Modalities: The full model utilizing visual features, hand pose, and temporal context achieved the best performance, confirming the complementary nature of these signals.
- Robustness: The system showed marginal sensitivity to MANO reconstruction noise, maintaining stable performance even with significant pose perturbations.

Significance and Claims

The paper claims that HOPE represents a shift from constrained, sensor-specific pressure estimation to a unified, hand-centric approach. By unifying heterogeneous supervision sources, the framework enables metric pressure estimation in scenarios where prior methods are inapplicable (e.g., arbitrary 3D objects, non-planar surfaces).

The authors emphasize that their method produces joint contact and pressure predictions that extend beyond the scope of contact-only or planar-pressure baselines. While the pressure supervision is dominated by a single tactile-glove dataset, the integration of bare-hand contact data allows the model to generalize to natural interactions. The work establishes a foundation for vision-based physical reasoning in hand-object manipulation, though the authors note limitations regarding heavy occlusion, the need for broader metric-pressure sources, and the current focus on normal pressure rather than full contact wrenches.