💻 computer science

Monocular Vision Based Control Framework for Grasping

This paper presents a unified monocular vision-based control framework that enables a position-controlled robotic gripper to successfully grasp both soft, deformable objects and rigid items in unstructured environments by leveraging open-vocabulary detection, language-guided stiffness estimation, and real-time visual tracking to adapt grasping strategies without tactile sensors.

Original authors: Shail Jadav, Dongheui Lee

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shail Jadav, Dongheui Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot arm trying to pick up a grocery bag. Inside, there's a hard plastic water bottle and a squishy head of lettuce. For a human, grabbing both is easy: you squeeze the bottle firmly, but you gently cup the lettuce so it doesn't turn into green mush. For a robot, this has been a nightmare. Usually, robots need special "feelers" (tactile sensors) on their fingers to know when they are squeezing too hard, or they need a different, squishy hand for soft things and a hard hand for rigid things.

But this paper suggests a different way: a robot that uses only one regular camera (like the one on your phone) and a standard, stiff robot hand to grab both the squishy lettuce and the hard bottle. It does this by acting like a very observant, language-savvy detective.

The "Magic Eye" and the "Word Guess"

First, the robot looks at the object and asks a simple question: "What is that?" It uses a language trick called StiffNET. Think of this like a robot that has read a million cookbooks and knows that "lettuce" is soft and "plastic bottle" is hard, just because of the words used to describe them. Before the robot even touches the object, this language guess tells it, "Okay, be gentle," or "Okay, you can squeeze hard."

Once the robot knows what it's dealing with, it starts watching the object like a hawk. It doesn't just look at the whole picture; it picks out four tiny, invisible dots on the object's surface and tracks them as the robot moves.

The Two Different Dance Moves

Here is where the robot gets clever. It performs two completely different dances depending on what it's holding:

1. The "Squishy" Dance (For Lettuce, Cheese, Croissants)
When the robot grabs something soft, it watches those four dots. If the object is a rigid bottle, the dots stay in the same shape relative to each other. But if it's a head of lettuce, the dots will squish closer together or stretch apart as the robot squeezes. The robot measures this "squishiness" (called dissimilarity).

The Analogy: Imagine you are holding a stress ball. If you squeeze it too hard, it changes shape. The robot watches the shape change. If the shape changes too much, the robot knows, "Whoa, I'm squeezing too hard!" and it loosens its grip. If the shape isn't changing enough, it knows, "I need to squeeze a bit more to hold on." It keeps adjusting its grip width in real-time until the shape is just right.

2. The "Zoom" Dance (For Bottles, Hard Plastic)
When the robot grabs something hard, the dots won't squish. So, the robot ignores the shape and watches the distance instead. As the robot arm moves up to lift the bottle, the camera gets closer to the object. The robot notices the object getting "bigger" in the camera view (a change in a scaling factor).

The Analogy: Think of holding a hard rock. You don't need to feel it squish; you just need to know when your fingers have closed enough to stop the rock from falling. The robot watches the object get closer in the camera view. As it gets closer, the robot automatically narrows its fingers until it's holding the object tight.

The Proof: Real-World Trials

The authors didn't just run this on a computer screen; they built a real robot arm (a Franka Emika Research 3) with a standard gripper and tested it in the real world. They tried it on:

Squishy stuff: Fresh mozzarella cheese, lettuce, croissants, and paper towels.
Hard stuff: A plastic water bottle.

The results showed that the robot could successfully pick up and move all these items without any special soft fingers or touch sensors. For the soft items, the robot adjusted its grip based on how much the object deformed. For the hard bottle, it adjusted based on how close the camera got.

What This Is NOT

It's important to know what this robot can't do yet. The paper explicitly rules out the need for expensive, fragile "vision-based tactile sensors" (special fingertips that act like cameras) and complex physics models that try to calculate exactly how much force is needed. The authors argue that relying on those is often too expensive or breaks down over time. Instead, they suggest that looking at the object with a regular camera and using language to guess the material is enough.

They also note that this method works best when the robot moves slowly and steadily. If the robot were to jerk the object around very fast, the system might get confused, so they suggest adding a "safety factor" (like a buffer zone) to make sure the robot doesn't drop things if it moves too quickly.

In short, this paper suggests that a robot doesn't need to "feel" to know how to hold something. It just needs to see, know the name of the object, and watch how the object moves or changes shape. It's a simpler, cheaper way to teach robots to handle the messy, squishy, and hard world of everyday objects.

Technical Summary: Monocular Vision Based Control Framework for Grasping

Problem Statement

Robotic manipulation in unstructured environments faces significant challenges when handling objects with widely varying mechanical properties, ranging from soft, deformable items (e.g., food, fabrics) to rigid everyday objects. Existing approaches typically address these categories in isolation, often relying on tactile sensing, object-specific mechanical models, or specialized end-effectors (such as soft grippers). However, tactile sensors struggle with objects softer than the sensor's elastomer, degrade over time, and are costly. Conversely, specialized soft grippers lack the force generation required for rigid objects, limiting their versatility. Furthermore, precise mechanical modeling of deformable objects is difficult due to time-varying and nonlinear material behaviors (e.g., changing moisture content in food). The paper posits that a unified, sensor-efficient approach is needed that can handle both compliant and rigid objects using only visual feedback, without requiring explicit mechanics models or tactile instrumentation.

Methodology

The proposed framework is a unified, monocular vision-based control pipeline designed for a standard position-controlled gripper (specifically a Franka Emika Hand). It operates using only RGB input and integrates several computer vision components with a language-driven control strategy.

1. Visual Perception and State Estimation

The system processes visual data through a sequential pipeline:

Object Detection & Segmentation: An open-vocabulary object detection model (YOLOv8x-worldv2) identifies the target object based on a text prompt. The detected bounding box is refined by a segmentation network (Meta SAM2) to generate a binary mask of the object.
Boundary-Aware Point Assignment: To ensure robust tracking, especially on low-texture or deformable objects, the system selects query points via a boundary-weighted clustering algorithm. This method assigns higher weights to points near the object's contour, ensuring a denser distribution of tracking points where deformation is most observable.
Real-Time Tracking & Depth Estimation: The selected points are tracked in real-time using TAPIR (Tracking Any Point with per-frame Initialization and Temporal Refinement). A monocular depth estimation model (Depth Anything) converts these 2D tracked points into 3D coordinates.
Geometric State Analysis:
- Deformable Objects: The system computes a Procrustes-based dissimilarity measure ( $D_t$ ) by comparing the current configuration of tracked points against their initial state. This metric serves as a visual proxy for object deformation and, by extension, the grasp force.
- Rigid Objects: For rigid items, the system utilizes a scaling factor ( $sf_t$ ) derived from the ratio of pairwise distances between tracked points. This factor correlates with the object's distance from the camera and is used to regulate gripper width.

2. Language-Driven Stiffness Estimation (StiffNET)

A key innovation is the StiffNET model, which infers an object's expected compliance directly from its semantic description (text prompt) before physical contact.

Architecture: It maps text embeddings (via a frozen CLIP encoder) to a scalar log-stiffness value using a neural network.
Training: The model is trained on two complementary sources: (1) pairwise hardness comparisons generated by an LLM (Gemma 3) to learn relative ordering, and (2) sparse ground-truth stiffness measurements to anchor the absolute scale.
Function: The estimated stiffness ( $\hat{k}$ ) acts as a prior to select the control mode: objects below a threshold $k_{th}$ are treated as deformable, while those above are treated as rigid.

3. Adaptive Grasping Control

The controller dynamically adjusts the gripper width ( $W_t$ ) based on the selected mode:

Deformable Mode: The gripper width is adjusted to minimize the difference between the current dissimilarity ( $D_t$ ) and a reference dissimilarity ( $D_{ref}$ ). $D_{ref}$ is dynamically updated based on the scaling factor to compensate for slippage or object movement.
Rigid Mode: The gripper width is regulated directly by the scaling factor ( $sf_t$ ), narrowing the gripper as the object appears to move closer (or the camera moves away) until a stable grasp is achieved.

Key Contributions

Unified Framework: The paper presents the first system to handle both soft/deformable and rigid objects within a single control pipeline using only monocular RGB input and a standard position-controlled gripper.
Semantic Priors for Compliance: It demonstrates that language-based semantic knowledge (via StiffNET) can provide effective object-level priors regarding material stiffness, enabling the system to select an appropriate grasping strategy before contact without tactile sensing.
Visual-Only Adaptation: The work shows that visual feedback alone—specifically through Procrustes-based dissimilarity for deformables and scaling factors for rigid objects—can support stable, adaptive grasping across a diverse range of everyday items.

Experimental Results

The framework was validated on a Franka Emika Research 3 arm with a modified Franka Hand (equipped with 3D-printed foam-cushioned fingers) using a generic USB camera. Experiments were conducted on a dataset of objects with distinct mechanical properties:

Deformable/Soft: Lettuce, fresh mozzarella cheese, croissants, and paper towels.
Rigid: Hard plastic bottles.

Performance:

The system successfully completed pick-and-place operations for all tested objects.
For deformable objects (e.g., lettuce), the controller successfully adapted the gripper width to maintain a stable grasp as the object deformed, evidenced by the stabilization of the dissimilarity metric and scaling factor.
For rigid objects (e.g., plastic bottles), the system effectively regulated the gripper width based on the scaling factor, achieving a secure grasp once the object was lifted.
The algorithm ran at 30 frames per second on an Nvidia RTX A4000 GPU with four grid points ( $k=4$ ).
StiffNET demonstrated the ability to estimate stiffness values comparable to ground truth and other LLM-based estimates for various materials (e.g., tofu, wood, steel, PVC).

Significance and Claims

The authors claim that this work offers a practical, sensor-efficient, and generalizable approach for robotic manipulation in food handling and household automation. By eliminating the need for specialized tactile sensors, complex mechanical models, or custom soft grippers, the framework lowers the barrier to deploying robots in unstructured environments.

The paper emphasizes that while human manipulation relies on a seamless integration of visual and tactile feedback, this system proves that visual feedback alone is sufficient to achieve stable grasping across a broad spectrum of object compliances. The integration of semantic priors allows the robot to "anticipate" object behavior, bridging the gap between high-level language understanding and low-level motor control. The authors position this as a step toward more versatile robotic assistants capable of handling the diverse physical properties of real-world objects without requiring extensive re-calibration or specialized hardware for each new task.