Technical Summary: CLIFT – Turning Gemini Robotics On-Device into Humanoid Specialists via Non-Invasive Closed-Loop Iterative Fine-Tuning

1. Problem Statement

The robotics community faces a fundamental tension: the most capable Vision-Language-Action (VLA) foundation models are typically trained on proprietary data and remain closed-source, limiting downstream users' ability to adapt them. While an emerging paradigm offers Managed Supervised Fine-Tuning (SFT) APIs (e.g., Gemini Robotics On-Device or GROD) that allow users to submit data and receive tuned policies without accessing weights or gradients, this regime presents significant limitations for agile, contact-rich humanoid manipulation.

Current API-based adaptation relies primarily on demonstration-only SFT. This approach hits a performance ceiling because:

1. Distribution Mismatch: It trains on a fixed distribution of human teleoperation data, whereas deployment occurs under the policy's own closed-loop observation distribution.
2. Execution Quality: Task completion alone does not guarantee success; factors like smoothness, contact quality, and safety are critical.
3. Controller Coupling: VLA policies are tightly coupled with downstream whole-body controllers. The same action chunk can yield different physical outcomes due to latency, compliance, and tracking dynamics, creating a gap between policy output and deployed behavior.

Existing closed-loop improvement methods (e.g., PPO, residual RL) require access to internal signals like gradients, likelihoods, or critic networks, which are unavailable in restricted-access API regimes. The central question is: How effective is the managed-API regime for humanoid adaptation, and how can closed-loop improvement be realized within it to achieve task mastery?

2. Methodology: CLIFT

The authors propose CLIFT (Closed-Loop Iterative Fine-Tuning), a non-invasive pipeline that bridges managed SFT interfaces with closed-loop policy improvement. CLIFT operates as a "flywheel" that converts deployment-time reward feedback into API-compatible supervised data without ever touching the model's internal weights or losses.

The pipeline consists of three iterative steps:

A. Preference-Calibrated Dense Reward Modeling

To score deployment rollouts, CLIFT employs a two-stage reward generation process:

1. Candidate Generation: A zero-shot VLM generates multiple candidate per-step reward sequences for a given rollout.
2. Human Selection & Distillation: Human operators provide pairwise preferences on rollout quality. The candidates whose cumulative return rankings best match human preferences are selected. These selected sequences are then distilled into a generative reward model ($R_\theta$) via fine-tuning. This model provides dense, preference-calibrated rewards that capture execution quality beyond coarse task progress.

B. Retrieval-Based Advantage Conditioning

Instead of discarding failed rollouts or treating episodes as binary success/failure, CLIFT performs chunk-level credit assignment:

1. Retrieval: For each action chunk starting at state $o_t$, the system retrieves a set of comparison chunks from other rollouts that start from visually similar states.
2. Scoring: The chunk is scored based on its discounted return over a look-ahead window using the reward model $R_\theta$.
3. Labeling: The chunk is labeled with a binary Advantage Token ($I \in \{positive, negative\}$). If the chunk's return lies in the top 30% of the retrieved set (auto-calibrating to state difficulty), it receives a positive label; otherwise, it is negative.
4. Data Conversion: Each rollout chunk is converted into an SFT tuple: (observation, instruction, action_chunk, advantage_token).

C. Iterative Self-Improvement

The loop proceeds as follows:

1. Deployment: The current policy $\pi_k$ is deployed on the robot to collect closed-loop rollouts.
2. Relabeling: Rollouts are scored by $R_\theta$, and chunks are labeled with advantage tokens.
3. Dataset Construction: A cumulative dataset $D_k$ is formed by combining the original human demonstrations (always labeled positive) and the newly relabeled rollout chunks.
4. Fine-Tuning: The base model is fine-tuned from scratch on $D_k$ via the managed SFT API to produce $\pi_{k+1}$.
5. Inference: At deployment, the policy is conditioned on the positive advantage token to steer behavior toward high-quality actions.

3. Experimental Setup

• Platform: Unitree G1 humanoid robot with dual arms, dexterous hands, and head-mounted RGB cameras.
• Control: Hierarchical whole-body control where the VLA outputs upper-body joint angles and lower-body velocities, tracked by an RL-learned controller.
• Tasks: Three contact-rich, agile manipulation tasks:
  1. Box Packing: Precise grasp and placement.
  2. Cup Insertion: Asymmetric bimanual coordination (stabilizing a carrier while inserting a bottle).
  3. Bimanual Plate Handover: Coordinated transfer between hands.
• Models Compared:
  • GROD: Closed-weight model via managed SFT API.
  • $\pi_{0.5}$: Open-weight VLA (Physical Intelligence) with direct access to weights.
• Protocol: Both models were initialized with 2 hours of teleoperated demonstrations and subjected to two cycles of CLIFT.

4. Key Results

Performance Gains

• GROD (Closed-Weight): CLIFT successfully pushed GROD to near-perfect task mastery after two flywheel cycles, despite API-only access.
  • Box Packing: 93% $\to$ 100%
  • Cup Insertion: 70% $\to$ 98%
  • Bimanual Plate Handover: 53% $\to$ 96%
• $\pi_{0.5}$ (Open-Weight): CLIFT also improved the open-weight model, but it plateaued at a significantly lower ceiling (e.g., 5% $\to$ 30% on Bimanual Plate Handover).
• Comparison: Direct SFT on GROD already outperformed $\pi_{0.5}$ trained on the same data. Even when $\pi_{0.5}$ was granted "invasive" adaptation (FiLM-style conditioning to inject preference signals directly into the network), it still trailed the API-adapted GROD.

Dense vs. Episode Selection

The authors compared two variants of CLIFT:

1. Dense Advantage-Conditioned: Labels every chunk in every rollout (positive or negative).
2. High-Return Episode Selection: Only keeps the top 30% of successful episodes (standard DAgger-style).

• Finding: On easy tasks, both performed similarly. On the hardest task (Bimanual Plate Handover), the Dense variant significantly outperformed episode selection (96% vs. ~84%). This demonstrates that retaining failed rollouts and relabeling well-executed chunks as positive examples (and poor chunks as negative) is crucial for learning from failure.

Emergent Behaviors

CLIFT enabled the policy to acquire behaviors not present in the original demonstrations:

• Pre-manipulation: Reorienting a box with fingers/wrist to facilitate grasping.
• Recovery: Attempting a second insertion after a failed first attempt, rather than stalling.

5. Significance and Claims

The paper claims that CLIFT demonstrates that managed SFT APIs can serve not just as customization interfaces, but as practical mechanisms for closed-loop policy improvement.

• Non-Invasive Mastery: It is possible to push a closed-weight foundation model to near-perfect success on complex, contact-rich tasks without "opening the model box" (accessing weights, gradients, or losses).
• Foundation Model Importance: The choice of the base model is critical. A stronger closed-weight foundation model (GROD) adapted via a narrow API interface outperforms a weaker open-weight model ($\pi_{0.5}$) even when the latter is granted full internal access. This suggests that for closed-loop adaptation, the strength of the pretrained prior may matter more than the level of training access.
• Data Efficiency: By converting deployment failures into dense, chunk-level training signals, CLIFT extracts maximum value from real-robot rollouts, turning failure into a learning signal rather than discarding it.

The authors conclude that while real-robot rollouts are costly, this approach offers a viable path toward task mastery for humanoid robots in the era of proprietary foundation models.