Towards Fine-Grained Object Manipulation: SAM3-Guided Visuomotor Policy with Persistent Memory Learning and Focused Visual Conditioning
This paper presents a SAM3-guided visuomotor framework featuring FOM-SAM3 for persistent object memory and FSAE for focused visual conditioning, enabling robots to robustly distinguish and manipulate fine-grained objects amidst distractors and visual similarities.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Robots have long been masters of broad, sweeping tasks: picking up a cup, moving a box, or stacking blocks. For these jobs, a robot's "eyes" usually scan the whole room, looking for any object that fits a general description like "cup" or "box." This approach works well when the goal is simply to grab a cup. But the real world is far more specific. Imagine a human asked to pick up a specific red can of strawberry juice from a shelf crowded with red cans of soda, blue cans of water, and green cans of tea. A human instantly recognizes the brand, the flavor, and the subtle differences in the label. A standard robot, however, often sees only "red can" and grabs the wrong one, or gets confused by the clutter. This gap between general object recognition and the ability to distinguish fine details—like a specific model, color pattern, or brand—is the frontier of fine-grained manipulation. The challenge is not just seeing the object, but seeing the right object among many that look nearly identical, and then acting on it with precision.
A team of researchers has developed a new system that teaches robots to make this distinction, allowing them to handle specific items even when visually similar "twins" are nearby. Their approach, detailed in a recent study, moves beyond the idea of teaching a robot a new skill from scratch every time it encounters a new object. Instead, they created a way for the robot to build a persistent memory of specific items it has learned to recognize. The system is built on top of a powerful visual foundation model known as SAM3, which is excellent at finding and outlining objects in images. However, the standard version of this model is limited; it can be told to find "a cup," but it struggles to find "the specific blue cup with the chip on the rim" when a nearly identical blue cup is right next to it. The researchers solved this by creating a "memory bank" where the robot stores unique digital fingerprints for each specific item it needs to learn.
To build this memory, the researchers did not retrain the entire massive visual system, which would be slow and computationally expensive. Instead, they kept the core visual engine frozen and taught it a new trick: how to associate a specific set of internal "tokens" with a specific object. They showed the robot a few dozen photos of a target item, such as a red Wontae strawberry juice can, against a plain background. Through a process of learning what makes that specific can different from other red cans, the system generated a compact set of memory tokens. These tokens act like a persistent ID card for that specific object. Once stored, the robot can retrieve this ID card whenever it needs to find that specific can again, even if the can is in a different room, under different lighting, or surrounded by confusing look-alikes. This allows the robot to switch between tasks simply by swapping the memory token it is looking for, rather than relearning the entire task.
The second major innovation is how the robot uses this memory to decide what to do. In many robotic systems, the visual information is a blurry mix of the whole scene, which can distract the robot's decision-making. The researchers introduced a method called focused spatial-appearance encoding. This technique forces the robot to ignore everything outside the target object. It takes the exact shape and location of the target, as identified by the memory tokens, and extracts only the visual details from inside that shape. It combines this with the precise coordinates of where the object is sitting. By feeding the robot's action planner only this focused, high-quality data, the system ensures the robot is reacting to the specific item it was asked to find, not the background clutter or similar-looking neighbors. This focused view is then passed to the robot's control software, which calculates the smooth movements needed to grab or push the object.
The researchers tested this system on a real robot arm equipped with two cameras, one providing a third-person view of the table and another mounted on the arm for a first-person perspective. They created a dataset of thirty different physical objects, ranging from cans and cups to lids and boxes, many of which had visually similar counterparts. In one set of tests, the robot was asked to pick up a specific can while other cans of the same color but different brands were placed nearby. Standard robotic policies, which rely on general scene descriptions, failed frequently in these scenarios, often grabbing the wrong can or getting confused by the distractors. The new system, however, successfully identified and manipulated the correct target in nearly all trials. When the researchers swapped the target for a different object of the same type but a different brand, the robot simply retrieved the new memory token and performed the task correctly without any new training or demonstrations.
The results showed that the system could distinguish between objects that were visually almost identical, a task that stumped other methods. In tests where the robot had to pick a specific item from a group of three or four similar objects, the new approach maintained a high success rate, while other methods saw their performance drop significantly. The system also proved robust when the robot had to move the object to a new location, as the focused visual encoding helped it track the item's changing position and orientation. The researchers noted that the system is not perfect; if the unique features of an object are hidden, such as the label on a can being turned away from the camera, the robot cannot identify it. Additionally, by focusing so strictly on the target, the robot might miss other obstacles in the scene, which could be a limitation for navigating complex environments. Nevertheless, the work demonstrates a significant step forward in giving robots the ability to handle the subtle, specific demands of real-world tasks, moving them from general grabbers to precise operators capable of distinguishing the familiar from the confusing.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.