Technical Summary: Embodied Robot Manipulation in the Era of Foundation Models

1. Problem Definition

Robotic manipulation remains one of the most challenging tasks in embodied intelligence. Unlike navigation or locomotion, which primarily regulate the robot's own motion, manipulation involves contact-rich interactions that intentionally alter object states. This complexity arises from the need to close the loop across multiple levels: perception, semantic understanding, task reasoning, physically grounded action generation, and reliable execution. The difficulty is further compounded by the diversity of manipulation settings, ranging from basic single-arm tasks to dexterous multi-fingered control, deformable object manipulation, and mobile or humanoid embodiments.

While recent advances in vision, language, and multimodal learning have accelerated the development of robotic foundation models, a systematic understanding of how these models reshape the design space of manipulation is needed. Existing surveys often organize literature by specific model families (e.g., Vision-Language-Action models, diffusion policies), which can obscure the functional roles of different components and their interactions across the planning-control hierarchy.

2. Methodology and Taxonomy

This paper proposes a unified abstraction that decomposes robotic manipulation into three distinct layers: High-Level Planning, Low-Level Action Modeling, and Actuation-Level Control. The survey focuses primarily on the first two layers, where foundation models exert the most direct influence.

A. High-Level Planning

This layer generates structured guidance (artifacts) rather than executable motor commands. It determines action intent, temporal organization, and attentional focus. The paper categorizes high-level planning approaches into six main types:

• LLM-based Task Planning: Utilizes Large Language Models for semantic reasoning and task decomposition (e.g., SayCan, LLM-Planner). While strong in reasoning, these plans often lack physical grounding and may fail geometric constraints.
• MLLM-based Task Planning: Employs Multimodal Large Language Models to jointly reason over vision and language, enabling end-to-end task reasoning or improved planning without separate vision modules (e.g., PaLM-E, RoboBrain).
• Programmatic Planning: Uses code generation as an intermediate abstraction to bridge high-level reasoning and low-level control, offering explicit structure and conditional logic (e.g., Code as Policies, ProgPrompt).
• Geometric Constraint-based Planning: Translates language and perception into structured spatial objectives or relational constraints (e.g., VoxPoser, ReKep) that are optimized by classical motion planners.
• Affordance-based Planning: Infers action possibilities (what actions an object enables) from geometric, visual, semantic, or multimodal cues to bridge perception and manipulation (e.g., CLIPort, Ditto).
• 3D Representation-based Planning: Uses 3D representations (e.g., Gaussian Splatting, Neural Descriptor Fields) as mid-level interfaces to transform perception into structured action-relevant representations like grasp candidates or spatial correspondences.

B. Low-Level Action Modeling

This layer transforms observations and task specifications into executable actions. The paper organizes learning-based approaches in this layer by their learning pipeline:

1. Learning Strategies:
   • Reinforcement Learning (RL): Includes model-free (trial-and-error) and model-based (using dynamics models) approaches.
   • Imitation Learning (IL): Includes Behavior Cloning (from action) and Imitation from Observation (IfO).
   • Learning with Auxiliary Tasks: Enhances policies via self-supervised objectives like world modeling or goal extraction.
2. Input Modeling: Defines how multimodal inputs are encoded.
   • Vision-Action Models: Map visual observations directly to motor commands (2D or 3D inputs).
   • Vision-Language-Action (VLA) Models: Integrate semantic grounding, spatial reasoning, and sequential action generation. The survey details trends in 2D vs. 3D inputs, model-oriented vs. model-agnostic strategies, and the integration of tactile, force, and audio modalities.
   • Tactile-based Models: Specifically addresses the incorporation of tactile feedback for contact-rich manipulation.
3. Latent Learning: Focuses on acquiring robust representations.
   • Pretrained Latent Learning: Learning transferable visual representations from general, human-egocentric, or robotic datasets.
   • Latent Action Learning: Learning intermediate representations that compress task-relevant motion, including discretized (vector quantization) and continuous latent actions.
4. Policy Learning: Specifies how representations are decoded into actions.
   • Paradigms: Includes MLP-based, Transformer-based (e.g., ACT), Diffusion Policies (iterative denoising), Flow Matching (deterministic transport), and other paradigms like State-Space Models.

3. Key Contributions

• Unified Taxonomy: The paper introduces a structured framework that categorizes robotic manipulation methods based on their functional role (High-Level Planning vs. Low-Level Action Modeling) rather than just model architecture. This reveals common design patterns across different model families.
• Abstraction of Foundation Model Roles: It clarifies that foundation models in manipulation serve two primary roles: (1) generating structured planning artifacts (constraints, code, affordances) for downstream action generation, or (2) directly modeling executable actions and trajectories.
• Comprehensive Coverage: The survey systematically reviews the integration of diverse modalities (vision, language, touch, audio, force) and the evolution from classical planning to learning-based and foundation-model-driven approaches.
• Identification of Gaps: It synthesizes emerging trends and identifies specific limitations in current approaches, such as the lack of physical grounding in LLM planners, the ambiguity of affordance predictions, and the challenges of 3D perception in VLAs.

4. Results and Findings

The paper does not present new experimental results but synthesizes findings from the existing literature to draw several conclusions:

• High-Level Planning: While LLMs and MLLMs provide strong semantic reasoning, purely language-based plans often fail to satisfy geometric or kinematic constraints. Hybrid approaches that generate geometric constraints or code are more robust for execution.
• Low-Level Action Modeling: VLA models have become a dominant paradigm, successfully integrating semantic grounding with action generation. However, a gap remains between high-level semantic reasoning and precise continuous motor execution.
• Multimodality: While vision and language are standard, the integration of tactile, force, and audio modalities is critical for robust, contact-rich manipulation but remains an area of active development.
• Latent Representations: The use of latent spaces (both discrete and continuous) is emerging as a key strategy to improve transferability, efficiency, and the handling of long-horizon tasks.

5. Significance and Future Directions

The paper argues that the field is moving toward more general-purpose robotic foundation models but faces significant hurdles. It identifies four core challenges for future research:

1. Building a True Robot Brain: Moving beyond task-specific fitting to systems that generalize across unseen tasks, objects, and embodiments through flexible architectures and lifelong learning.
2. Data and Evaluation Bottlenecks: Addressing the scarcity of real-world data and the lack of standardized, reproducible evaluation frameworks that cover diverse environments and interaction complexities.
3. Multimodal Physical Interaction: Developing unified models that adaptively integrate diverse sensory streams (touch, audio, thermal) and handle complex material manipulation (deformable objects).
4. Safety and Collaboration: Ensuring safety as a first-class design constraint, including intrinsic safety mechanisms, fault detection, and safe multi-robot collaboration in open-world environments.

The survey concludes that while substantial progress has been made, achieving robust, general-purpose robotic manipulation requires overcoming data, evaluation, and perception bottlenecks, and ensuring safety in open-world settings. The proposed abstraction provides a structured view to guide future research in designing more adaptive and trustworthy robotic systems.