Technical Summary: VISOR – A Vision-Language Model-based Test Oracle for Testing Robots

1. Problem Statement

Testing robotic systems, particularly in safety-critical and autonomous domains, faces a significant bottleneck known as the test oracle problem: determining whether a robot has performed its intended task correctly and with adequate quality. Traditional approaches rely on:

• Symbolic Oracles: These are task-specific, often providing only binary pass/fail judgments without quantifying the quality of the execution. They lack reusability across different tasks.
• Human Manual Evaluation: While humans can assess task quality (e.g., distinguishing between high, medium, and low execution standards), this process is time-consuming, subjective, error-prone, and difficult to scale.

The paper argues that scaling robotic testing requires an automated oracle that can assess both task correctness (success vs. failure) and task quality (graded performance) without continuous human intervention.

2. Methodology: VISOR

The authors propose VISOR, a framework that utilizes Vision-Language Models (VLMs) as automated test oracles. VISOR analyzes video recordings of robot executions to determine task outcomes and quality levels.

Core Components

• Input: The system processes videos generated by a simulator. The simulator takes environmental configurations (object positions, rotations, model IDs) and task instructions (e.g., "Pick up the orange") as input, executes a control policy, and outputs a video.
• VLM as Judge: VISOR employs a "VLM-as-a-judge" approach using zero-shot prompting. The authors evaluated several prompting styles (zero-shot vs. few-shot) and determined that few-shot prompting was computationally prohibitive due to token limits (64K vs. 7K) and inference time.
• Prompt Design: Two distinct prompt templates were designed:
  1. Correctness Assessment: Asks the VLM to classify the execution as "Successful" or "Failure" based on the entire video sequence.
  2. Quality Assessment: Asks the VLM to classify successful executions into three ordinal categories: "High," "Medium," or "Low," based on predefined decision rules adapted from prior human evaluation studies.
• Uncertainty Quantification: Recognizing the inherent uncertainty in VLMs, VISOR explicitly quantifies the model's confidence during assessment. It calculates Entropy, Maximum Softmax Probability (MSP), and DeepGini to measure the dispersion of predicted probabilities.

Experimental Setup

• Dataset: A curated subset of over 1,000 videos (1,032 total) derived from the dataset by Valle et al. [48]. The dataset covers four robotic manipulation tasks: MoveNear, PickUp, PutIn, and PutOn. The data was balanced to include equal numbers of successful and failing executions, with successful cases further balanced across quality levels (High, Medium, Low).
• Models Evaluated: The authors initially screened six VLMs (GPT-4.1, Gemini-2.5-Flash, LLaVA-Next, SmolVLM2, Qwen, Eagle). Open-source models were excluded from the main study because they exhibited a bias toward predicting "Successful" for all inputs, failing to distinguish between outcomes. The final evaluation focused on GPT-4.1 (referred to as GPT) and Gemini-2.5-Flash (referred to as Gemini).
• Metrics:
  • Correctness: Precision, Recall, F1-score.
  • Quality: Micro and Macro versions of Precision, Recall, and F1.
  • Distance Metric: A novel metric ($d$) ranging from 0 to 5, combining correctness errors (weight 2) and quality ordinal errors (weight 1) to quantify the severity of misclassification.
  • Uncertainty: Correlation between uncertainty metrics and the distance metric using Spearman's rank correlation.

3. Key Results

Task Correctness (RQ1.a)

• Gemini demonstrated higher Recall (e.g., 0.730 for MoveNear, 0.836 for PickUp), meaning it successfully identified most true positive cases but produced more false positives.
• GPT demonstrated higher Precision (e.g., 0.844 for MoveNear, 0.980 for PutIn), meaning it made fewer false positives but missed more true positive cases (lower recall).
• Statistical tests (Mann-Whitney U) confirmed these differences were significant with large effect sizes.
• Qualitative Errors: Both models struggled with "near-complete" executions (classifying them as success), interacting with the wrong object, and interpreting task-like motions as successful completions. Gemini was more prone to classifying failures as successes.

Task Quality (RQ1.b)

• Overall performance for multi-class quality assessment was modest (Micro F1 scores ranged roughly from 0.15 to 0.48).
• GPT generally outperformed Gemini in quality assessment metrics and achieved significantly lower Distance scores (closer to ground truth), particularly on PutIn and PutOn.
• Gemini performed relatively better on the MoveNear task but struggled with fine-grained spatial gap detection.
• The primary challenge identified was the VLMs' difficulty in reasoning about subtle spatial changes and fine-grained execution defects (e.g., hesitation, vibration, slight collisions).

Uncertainty and Correlation (RQ2 & RQ3)

• Low Uncertainty: Both models exhibited extremely low uncertainty (Entropy and DeepGini near 0, MSP near 1.0) across all tasks and runs.
• Confidence vs. Correctness: Low uncertainty did not correlate with high correctness. The models were often highly confident even when their predictions deviated significantly from the ground truth.
• Correlation Analysis:
  • Gemini: Showed statistically significant positive correlations between uncertainty metrics (DeepGini, Entropy) and the distance metric. Higher uncertainty generally indicated a larger deviation from the ground truth.
  • GPT: Showed weaker and less consistent correlations. For some tasks (e.g., PutIn), the correlation was not statistically significant or reversed.
• Conclusion on Uncertainty: While uncertainty metrics can serve as a signal for reliability (especially for Gemini), they are insufficient on their own to guarantee correctness.

4. Key Contributions

1. VISOR Framework: A novel approach for automating test oracle assessment in robotics using VLMs, capable of evaluating both binary correctness and graded task quality.
2. Large-Scale Empirical Study: Evaluation of VLMs on over 1,000 robotic task videos across four distinct manipulation tasks, providing a comprehensive benchmark for automated testing.
3. Novel Distance Metric: Introduction of a weighted distance metric that accounts for the severity of errors in both correctness and ordinal quality assessment.
4. Uncertainty Analysis: A detailed investigation into the relationship between VLM confidence and assessment accuracy, revealing that high confidence does not equate to correctness and that uncertainty metrics vary in utility between models.

5. Significance and Claims

The paper claims that VISOR offers a scalable, automated alternative to human manual evaluation for robotic testing.

• Cost and Efficiency: The authors estimate the cost at approximately $0.03 per assessment with an inference time of ~7 seconds. While slower and more expensive than human evaluation for single videos, VISOR offers significant advantages in parallel processing and continuous operation, making it viable for large-scale testing in CI/CD pipelines.
• Limitations: The authors explicitly state that their claims are limited to simulation-based datasets. They acknowledge that real-world factors like sensor noise and motion blur were not tested.
• Future Work: The paper suggests that fine-tuning open-source VLMs on robotic data could reduce reliance on proprietary models and improve performance. It also proposes exploring "World Models" (e.g., Nvidia Cosmos) for richer video analytics and developing voting mechanisms to consolidate assessments from multiple VLMs.

The authors conclude that while VLMs show promise as test oracles, their current limitations in spatial reasoning and the disconnect between confidence and correctness necessitate further research into uncertainty metrics and model adaptation before full deployment in safety-critical real-world scenarios.