💻 computer science

R2S-Eval: Robot Evaluation with Real-to-Sim Calibration via Vision-Language Models

The paper proposes R2S-Eval, an automated evaluation pipeline that combines real-to-sim calibration with vision-language model preference assessment to generate stable, quality-aware rankings of robot manipulation policies, thereby overcoming the labor-intensive and limited nature of conventional real-world success-rate metrics.

Original authors: Yidi Wang, Feixiang Ruan, Ruoqu Chen, Jie Yin, Yang Yu, Mengdi Xu, Kaifeng Zhang

Published 2026-09-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yidi Wang, Feixiang Ruan, Ruoqu Chen, Jie Yin, Yang Yu, Mengdi Xu, Kaifeng Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the quiet corners of modern robotics, a new generation of machines is learning to perform delicate tasks, from stacking blocks to pouring liquids, guided by models that can see the world and understand language. These systems, often called vision-language-action models, are designed to take a command like "pick up the cup" and translate it into a series of physical movements. However, teaching a robot to move is only half the battle; the other half is figuring out how well it actually performs. Traditionally, scientists have judged these robots by watching them try a task over and over again on a physical table, counting how many times they succeed and how many times they fail. This method is slow, exhausting for the human operators who must reset the scene after every attempt, and often too blunt to catch the subtle differences between a clumsy success and a graceful one. If a robot drops a cup but manages to pick it up again, or if it wobbles violently before placing an object, a simple "success" or "failure" label misses the story entirely.

A team of researchers has proposed a different way to judge these machines, one that moves away from counting heads and toward watching the whole performance. Their approach, called R2S-Eval, combines two powerful ideas to create a more efficient and insightful evaluation system. First, they build a digital twin of the real-world testing environment, a simulation that is carefully calibrated to match the physical robot's movements and the objects it interacts with. Instead of forcing the robot to run thousands of trials on a real table, which would take days of human labor, the team runs these trials inside this high-fidelity computer world. This allows them to generate hundreds of video clips of the robot attempting tasks in a fraction of the time. The second part of their method involves using an artificial intelligence system trained to understand both images and language to watch these videos. Rather than just checking if the task was completed, this AI acts like a human judge, comparing pairs of video clips to decide which performance looked better, smoother, or more controlled. By aggregating these pairwise comparisons, the system produces a ranking of the robot's policies that reflects the quality of the behavior, not just the final outcome.

The researchers tested this pipeline on a dual-arm robot platform equipped with dexterous hands and multiple cameras, asking it to perform seven different tabletop tasks, such as picking up a ball and placing it in a box, or stacking blocks. They compared six different robot control models, ranging from large, complex systems to smaller, more efficient ones. In the traditional setup, evaluating these models would require the robot to attempt each task hundreds of times in the real world, with a human operator constantly resetting the objects and monitoring the hardware. The team found that their new method could generate the necessary video data in a simulated environment that was tuned to match the real world so closely that the robot's behavior in the computer was nearly identical to its behavior on the table. When they ran the robot in the real world, the success rates of the different models were very close to the rates observed in the simulation, with an average difference of only about two percentage points. This confirmed that the digital twin was a reliable stand-in for the physical machine, allowing the researchers to skip the tedious hardware trials without losing accuracy.

Once the videos were collected, the team turned to the artificial intelligence judge to rank the models. They showed the AI pairs of videos from different robots performing the same task and asked it to choose which one looked better. The AI considered factors like how smoothly the robot moved, whether it hesitated, and how much progress it made even if it ultimately failed. The results showed that the AI's rankings agreed with human preferences about ninety-two percent of the time, a significant improvement over simple success counting. More importantly, the system revealed nuances that a binary pass-or-fail test would have missed. For instance, in one experiment, two robots both successfully completed a task, but one did so with a single, fluid motion while the other dropped the object and had to try again. A traditional evaluation would have marked both as successful, but the new system correctly identified the smoother performance as superior. Similarly, when two robots both failed, the system could distinguish between one that made a genuine attempt and got stuck versus one that barely moved at all.

The study also quantified the human effort saved by this approach. In a conventional evaluation, the time required from a human operator grows linearly with the number of trials; if a researcher wants to test a robot twenty times on a task, they must spend twenty times the effort setting up and resetting the scene. The researchers estimated that running a full evaluation of six models across seven tasks with twenty trials each would require nearly twenty-seven hours of continuous human labor. By shifting the heavy lifting to the calibrated simulation and the automated AI judge, the R2S-Eval pipeline eliminated the need for this repeated hardware operation entirely. The system produced stable rankings that did not fluctuate wildly as more data was added, suggesting that the method is robust enough to be used as a standard tool for comparing future robot designs.

The findings suggest that the future of robot evaluation may lie less in counting successes and more in understanding the quality of the journey. By using a simulation that mirrors reality and an AI that can appreciate the subtleties of motion, researchers can now assess robot policies with a level of detail that was previously impossible without an army of human observers. The work does not claim to have solved every problem in robotics, nor does it suggest that simulations are perfect replacements for the physical world in all contexts. However, it demonstrates that for the specific goal of ranking and improving robot behaviors, a carefully calibrated digital environment combined with intelligent video analysis can provide reliable, detailed, and human-aligned insights. This shift allows scientists to move beyond the coarse metric of "did it work?" to the more meaningful question of "how well did it work?", paving the way for robots that are not just functional, but truly skilled.

Technical Summary: R2S-Eval

Problem Statement

Evaluating robot manipulation policies, particularly generalist Vision-Language-Action (VLA) models, is increasingly critical yet remains hindered by the limitations of conventional real-world evaluation. Current protocols rely on repeated hardware trials, manual scene resets, and continuous operator monitoring, making the process labor-intensive and unstable. Furthermore, these methods typically reduce performance to binary success rates, which fail to capture nuances in execution quality, such as motion smoothness, corrective behaviors, or progress toward a goal before failure. This coarse metric often yields inconsistent policy rankings across repeated evaluations and obscures behavioral differences between policies with similar success rates.

Methodology: The R2S-Eval Pipeline

The authors propose R2S-Eval, an evaluation pipeline that shifts the paradigm from binary success counting to automated, quality-aware preference estimation. The pipeline integrates three core components:

1. Real-to-Sim Calibration

To address the inefficiency of collecting rollout videos on physical hardware, R2S-Eval employs a calibrated simulation environment.

Calibration: The simulation is calibrated to match real-world geometry, kinematics, joint limits, camera viewpoints, and task initializations. It is not intended to be a perfect photorealistic digital twin but rather a faithful reproduction of factors determining policy behavior.
Adaptive Training: To bridge the residual sim-to-real gap (e.g., friction, compliance), policies are adapted using action replay, motion planning, or teleoperation data collected in the real world.
Video Generation: Candidate policies are deployed in closed-loop within this calibrated simulator to efficiently generate rollout videos ( $V_i$ ) for evaluation, significantly reducing the need for repeated hardware trials.

2. VLM Preference Evaluation

Instead of using hand-designed scorers, the pipeline utilizes Vision-Language Models (VLMs) as automated judges to assess execution quality.

Pairwise Sampling: The system samples pairs of rollout videos from different policies under identical task and initial configurations.
Structured Judgment: The VLM first generates structured behavior descriptions (covering motion smoothness, temporal continuity, and task progress) and then compares anonymized video pairs to produce binary preferences ( $i \succ j$ ).
Aggregation: These pairwise preferences are aggregated into a preference matrix and processed using a Bradley-Terry (BT) model. This statistical model infers latent quality scores ( $\theta$ ) for each policy, producing a final ranking based on execution quality rather than success counts.

3. Validation Protocol

The authors introduce a multi-axis validation protocol to ensure the pipeline's efficacy:

Reliability: Consistency with reference rankings reflecting actual capabilities.
Stability: Convergence of scores and low variance as evidence accumulates.
Human Agreement: Alignment between VLM preferences and human annotations.
Human Effort: Reduction in manual hardware operation time.
Behavioral Informativeness: Ability to distinguish quality differences between rollouts with identical binary outcomes (success/failure).

Key Contributions

Formulation of Preference Estimation: The work reframes manipulation policy evaluation as estimating preferences over rollout behaviors, generating rankings that reflect execution quality rather than binary success counts.
R2S-Eval Pipeline: The introduction of a novel pipeline combining real-to-sim calibration for efficient video acquisition with VLM-based preference evaluation for automated ranking.
Multi-Axis Validation: A comprehensive protocol covering ranking reliability, stability, human agreement, hardware effort reduction, and behavioral informativeness.

Experimental Results

Experiments were conducted on the SharpaNorth dual-arm humanoid platform across seven tabletop manipulation tasks (e.g., stacking blocks, picking and placing objects) and six candidate VLA policies (including $\pi_0$ , OpenVLA, and X-VLA).

Real-to-Sim Consistency: The calibrated simulation preserved comparative policy conclusions. While absolute success rates showed minor discrepancies (mean absolute difference of 2.13 percentage points), the overall policy rankings in simulation closely matched those in the real world.
VLM Performance: VLM-based preference evaluation demonstrated high agreement with reference performance metrics (Spearman's $\rho \approx 0.957$ in real-to-sim settings) and strong alignment with human preferences (Human Agreement Rate $\approx 91.9\%$ ).
Stability and Efficiency: The pipeline produced stable rankings with narrow confidence intervals. Crucially, it eliminated the linear increase in operator hours associated with increasing trials or tasks in conventional hardware evaluation.
Behavioral Informativeness: Case studies demonstrated that VLMs could distinguish between rollouts with identical binary labels (e.g., two successful attempts where one was smoother, or two failures where one showed more progress), capturing nuances invisible to success-rate metrics.

Significance and Claims

The paper claims that R2S-Eval advances robot evaluation from manual, unstable, and coarse success counting toward automated, statistically stable, and quality-aware evaluation. By leveraging real-to-sim calibration and VLMs, the method provides a scalable framework that:

Substantially reduces the manual effort and hardware wear associated with repeated trials.
Reveals behavioral quality differences that binary metrics miss.
Produces policy rankings that align with human assessments of execution quality.

The authors position this work as a step toward scalable evaluation for generalist manipulation policies, noting that future work will focus on extending the pipeline to broader tasks and robots while systematically analyzing real-to-sim discrepancies.