💻 computer science

Kitchen Robotic Manipulation utilizing Foundation Models

This paper presents a modular perception pipeline for kitchen robotic manipulation that integrates multiple foundation models to achieve robust 6D pose estimation and grasp planning, demonstrating an 89.12% ADI on a cluttered benchmark and successful zero-shot deployment on physical robots for tasks like dish transfer and cup stacking.

Original authors: Myung-Hwan Jeon, Sankalp Yamsani, Joohyung Kim

Published 2026-08-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Myung-Hwan Jeon, Sankalp Yamsani, Joohyung Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to teach a robot to help out in a messy kitchen. It sounds simple, but for a machine, a kitchen is a nightmare of chaos. Unlike a factory where every tool sits in the exact same spot, a kitchen is full of dishes piled up, hidden behind others, and sitting on slippery, shiny surfaces. For a robot to pick up a coffee mug, it first needs to "see" it clearly, figure out exactly where it is in 3D space, and know how to grab it without knocking over the whole stack. This is the world of robotic perception: the science of giving machines eyes and brains that can understand a messy, changing world.

To do this, scientists have been developing Foundation Models. Think of these as super-smart students who have read almost every book and looked at almost every picture on the internet before they ever start a specific job. Because they've seen so much, they don't need to be re-taught from scratch every time they enter a new room; they can just use their general knowledge to recognize a "cup" even if they've never seen that specific cup before. The big question researchers are asking is: Can we take these powerful, pre-trained "brains" and plug them directly into a robot's body so it can handle household chores without needing months of custom training for every single house?

The Kitchen Robot's New "Swiss Army Knife" Brain

In this paper, a team of researchers builds a robotic "perception pipeline" designed to help a robot arm navigate a cluttered kitchen sink and move dishes around. Instead of building one giant, rigid brain that only works for one specific kitchen, they created a modular system. Imagine a high-end camera where you can swap out the lens, the sensor, and the processor depending on what you are shooting. This robot system works the same way: it allows researchers to mix and match different "Foundation Models" to see which combination works best for finding and grabbing dishes.

The robot's job is to look at a sink full of plates, cups, and bowls, figure out exactly where each item is (its 6D pose, which means its location and its angle in 3D space), and then plan a safe way to pick it up. The team tested this system on a custom dataset of 20 real-world kitchen scenes, complete with messy piles and hidden objects.

The Recipe for Success

The researchers didn't just guess which models to use; they systematically tested 24 different combinations of visual and geometric models. They treated the system like a recipe, swapping ingredients to find the perfect flavor.

The Eyes (Visual Models): These models look at the 2D pictures to recognize what objects are there.
The Hands (Geometric Models): These models look at the 3D shape of the objects to understand their structure.
The Glue (Feature Fusion): They found that simply using the eyes or the hands wasn't enough. The secret sauce was fusing the 2D image features with the 3D point-cloud features. It's like having a detective who not only recognizes a face from a photo but also understands the shape of the person's body to know exactly how to shake their hand.

After running the numbers, the team discovered a "champion" configuration: LLMDet (for spotting objects), SAMv2 (for cutting out the object from the background), DINOv2 (for recognizing fine details), and GeoTransformer (for understanding 3D geometry). When these four worked together, the robot achieved an ADI (Average Distance of Indistinguishable views) of 89.12%. In plain English, this means the robot could estimate the position and angle of a dish with incredible accuracy, even when the dish was partially hidden or surrounded by clutter.

Real-World Proof

The best part? They didn't just stop at computer simulations. They took this "champion" configuration and put it on a physical robot arm in a real kitchen. They watched it successfully:

Transfer dishes from a sink to a dishwasher.
Stack cups on a counter.
Pick up items from a messy sink.

The robot did all this without any retraining for the specific kitchen it was in. It didn't need to learn what a "cup" looked like in that specific house; it just used its pre-trained foundation knowledge. In a series of real-world tests, the robot succeeded in 87.5% of its attempts (259 successes out of 296 trials).

Where It Stumbles

The paper is honest about where the system isn't perfect yet. The main reason for failure wasn't the robot's "brain" failing to see the object; it was the robot's "hand" slipping. The robot uses a compliant gripper (a soft, adaptive hand) to hold things gently, but sometimes the object would slip out during transport, especially when stacking cups in a tight, crowded sink. The robot also occasionally tried to grab the wrong spot if the detection was slightly off. However, the authors note that these are physical challenges with the gripper, not failures of the perception system itself.

Why This Matters

This work suggests that we don't need to build a new, custom robot brain for every single house. Instead, we can use a flexible, modular system that swaps in the best available "foundation models" to handle the job. It proves that by combining the right mix of visual and geometric intelligence, robots can adapt to the messy, unpredictable reality of human homes without needing a teacher to show them every single task. While there is still work to be done on making the robot's grip stronger and its movements smoother, this pipeline offers a practical, scalable path toward robots that can actually help us with the dishes.

Technical Summary: Kitchen Robotic Manipulation utilizing Foundation Models

Problem Statement

Deploying robotic systems in everyday human environments, particularly kitchens, presents significant challenges due to the unstructured, dynamic, and cluttered nature of these spaces. Traditional vision-based robotic systems rely heavily on task-specific datasets and supervised learning, which often fail to generalize beyond predefined environments. Consequently, these systems lack the adaptability required to handle unfamiliar surroundings or varying object arrangements without extensive manual intervention or retraining. While recent foundation models have demonstrated remarkable generalization capabilities across diverse domains, a critical gap remains in systematically integrating these models into robust perception pipelines that can bridge high-level perception with the concrete demands of physical manipulation (e.g., 6D pose estimation and grasp planning) without environment-specific fine-tuning.

Methodology

The authors propose a modular perception pipeline designed for household manipulation tasks, specifically focusing on dishware handling in kitchen environments. The system operates on the assumption that kitchen inventories are relatively stable and that 3D CAD models of target objects are available. The pipeline consists of four primary stages:

Multi-View Instance Segmentation: The system utilizes Open-Vocabulary Object Detection (OVOD) to generate bounding box prompts for the Segment Anything Model (SAMv2). To ensure consistency across multiple views, a cross-view verification mechanism projects masks using depth data and camera poses, assigning consistent object IDs based on Intersection over Union (IoU) thresholds.
Instance-Aware 3D Map Reconstruction: The pipeline constructs a 3D semantic map by fusing RGB-D frames. It employs a coarse 3D point cloud registration based on feature matching, followed by Iterated Closest Point (ICP) refinement to correct pose errors. Instance IDs from the segmentation masks are assigned to the point cloud to create an instance-aware map.
2D-3D Feature Fusion and Pose Estimation: This is the core innovation of the pipeline. For each segmented instance, the system extracts features using a visual foundation model ( $\Phi$ $Φ$ ) and a geometric foundation model ( $\Psi$ $Ψ$ ).
- Visual Features: Extracted from cropped object regions and projected into 3D space via depth and camera poses, then aggregated via voxel pooling.
- Geometric Features: Computed directly on the object point cloud.
- Fusion: The image and point features are L2-normalized and concatenated to form a unified per-point feature representation.
- Registration: The system performs object candidate proposal by comparing query features against a database of rendered CAD model features. It then executes object registration using correspondence-based alignment in the fused feature space, followed by ICP refinement.
Grasp Estimation: Using the estimated 6D poses and predefined CAD models, the system generates grasp candidates. These are filtered through kinematic feasibility checks (inverse kinematics), collision avoidance (using FCL), and reachability constraints to ensure physically executable grasps.

The architecture is explicitly designed to be modular, allowing the substitution of different visual and geometric foundation models to identify optimal configurations.

Key Contributions

The paper presents three primary contributions:

Unified Pipeline Integration: A system-level integration of multiple foundation models into a unified 6D-pose perception pipeline that operates across diverse kitchen environments without environment-specific training or fine-tuning.
Modular Architecture and Systematic Evaluation: A flexible framework where each component can be swapped. The authors systematically evaluated 24 combinations of visual and geometric foundation models on a custom real-world kitchen dataset to identify the best-performing configuration.
Real-World Validation: Demonstration of the integrated system on physical robotic platforms performing complex kitchen tasks (sink-to-dishwasher transfer, cup stacking) without further tuning, validating scalability and practical utility.

Experimental Results

The authors evaluated the pipeline on a custom dataset comprising 20 real-world kitchen scenes with varying clutter, occlusion, and object arrangements.

Best Configuration: The combination of LLMDet (OVOD), SAMv2 (Segmentation), DINOv2 (Visual Foundation), and GeoTransformer (Geometric Foundation) achieved the highest performance.
Quantitative Performance: This configuration achieved an Average Distance of model points for Indistinguishable views (ADI) of 88.92% on the 20-scene benchmark without ICP refinement. When ICP refinement was applied, the accuracy improved to 89.12%. For comparison, the state-of-the-art FoundationPose method achieved 87.32% ADI even when provided with ground-truth masks, whereas the proposed best configuration outperformed it without requiring such ideal inputs.
Feature Fusion Impact: The study confirmed that fusing 2D image features with 3D geometric features consistently improved performance across all configurations. While GeoTransformer performed poorly in isolation due to its reliance on global context, it achieved the highest accuracy when fused with discriminative image features from DINOv2.
Real-World Success: In physical demonstrations involving 296 trials across two different kitchen environments, the system achieved a success rate of 87.50% for tasks including simple grasping and cup stacking.
Failure Analysis: The majority of failures were attributed to physical interaction issues, such as gripper slip due to the compliant nature of the gripper or accidental displacement of neighboring objects in highly cluttered scenarios, rather than perception errors.

Significance and Claims

The paper claims that the proposed modular pipeline offers a practical framework for household robotics by demonstrating that foundation models can be effectively integrated to handle manipulation tasks without the need for environment-specific retraining. The authors highlight that their approach:

Enables Scalability: The system adapts to varied robot systems and new environments without additional tuning.
Provides Flexibility: The modular design allows practitioners to tailor model combinations to specific computational constraints or task difficulties.
Bridges Perception and Control: Unlike end-to-end vision-language-action policies that fold perception and control into a single learned model, this approach produces explicit 6D poses compatible with classical motion planners, ensuring reusability across different sensor configurations.

The authors acknowledge limitations, including reliance on CAD models for registration (limiting applicability to unknown objects), occasional grasp failures due to gripper constraints, and the need for more advanced manipulation strategies for tightly stacked objects. They suggest future work could involve learned trajectory generators and multi-view reconstruction methods for on-boarding new objects.