🤖 AI

PInVerify: An Offline Embodied Benchmark for Active Instance Verification

This paper introduces PInVerify, an offline embodied benchmark designed to evaluate Active Instance Verification (AIV) tasks where agents must perform multi-view inspections to distinguish fine-grained object attributes, revealing that while LoRA-fine-tuned agents achieve strong performance, current next-best-view strategies do not yet provide reliable gains over static baselines.

Original authors: Yuhang Jiang

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuhang Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a robot waiter in a busy, cluttered restaurant. Your boss gives you a very specific order: "Bring me the white mug with the red floral pattern and a tiny blue dot."

You scan the room, spot a white mug, and walk right up to it. In many current robot tests, the moment you arrive at the mug, the test says, "Great job! You found a mug!" and you get a gold star.

But here's the problem: That mug might actually have white stripes, or a green logo, or no dot at all. You got the category right (it's a mug), but you got the specific instance wrong. If you served it to your boss, they'd be annoyed.

This paper, PInVerify, introduces a new way to test robots that fixes this mistake. It calls this task Active Instance Verification (AIV).

The Core Idea: "Look Before You Leap"

Instead of just stopping when you get close, the robot is now asked to play a game of "Is this the one?" before it commits.

Think of the robot as a detective inspecting a suspect. The suspect (the object) is standing in a room. The detective (the robot) has a description of the real culprit. The detective can't just look from one angle; they need to walk around the suspect, peek behind them, and check their shoes to be 100% sure it's the right person.

The New Test: PInVerify

The authors built a digital playground called PInVerify to test this. Here's how it works:

The Setup: They placed 18 different types of everyday objects (like backpacks, mugs, and teddy bears) in 3D virtual rooms.
The Trap: They didn't just give the robot a clear view. They created a "6-sector" map around each object. Some spots are great for seeing the object. Others are "Trap Views"—places where the robot can physically walk, but the object is hidden behind a chair or looks blurry.
The Challenge: The robot gets a detailed description (e.g., "a red backpack with a gold zipper") and a candidate object. It must decide: Is this the right backpack, or just a red backpack that looks similar?

How They Tested It

The researchers tried two main approaches to see which robot "brain" was best at this detective work:

1. The "No-Training" Detective (Training-Free)
This is like giving a smart human a magnifying glass and a checklist but not letting them practice. They use a pre-existing AI model (Qwen3-VL) that already knows how to see and read.

The Strategy: The robot breaks the description down into small clues (color, pattern, logo). It looks at the object, checks one clue, then moves to a new angle to check the next.
The Result: It was pretty good at saying "No, that's not it" when the object was wrong. But it was sometimes too quick to say "No" even when the object was right, just because it couldn't see a specific detail from one angle.

2. The "Trained" Detective (LoRA-Fine-Tuned)
This is like taking that same smart AI and giving it a crash course specifically on this game. They showed it thousands of examples of "Right Mug" vs. "Wrong Mug" and let it learn the patterns.

The Result: This trained robot got much better at saying "Yes, this is the one!" (improving from a 14% success rate to over 75%). It learned to be more confident when it saw enough evidence.
The Trade-off: Because it became so good at saying "Yes," it sometimes made mistakes by saying "Yes" to the wrong object (like confusing a black laptop for a blue one). It became a bit over-confident.

Key Findings (The "Aha!" Moments)

Geometry isn't enough: Just because a robot can navigate to an object doesn't mean it understands what the object is. Getting close is easy; verifying the details is hard.
The "Trap" is real: The paper found that robots often get stuck looking at "Trap Views" (angles where the object is hidden). If the robot doesn't realize it's looking at a bad angle, it makes a wrong decision.
Bigger isn't always better: They tested different sizes of AI brains. Surprisingly, the slightly smaller, well-trained brain (8 billion parameters) performed better than some larger, untrained ones.
Detection is the bottleneck: The biggest reason robots failed wasn't that they couldn't "think" hard enough; it was that they couldn't see the object clearly in the first place. If the robot's "eyes" (the detector) were blurry, the brain couldn't fix it.
Walking around helps, but only a little: The robots tried different strategies for which angle to look at next (like "go to the farthest spot" vs. "ask the AI where to look"). Surprisingly, the fancy strategies didn't help much more than just picking a random new angle. The real magic was in the thinking after the look, not the walking itself.

The Bottom Line

This paper doesn't just say "robots are getting smarter at walking." It says, "Robots need to learn to double-check their work."

They created a new benchmark (PInVerify) that forces robots to stop, look around, and verify details before acting. They found that while current AI is getting better at this, it still struggles with tiny details and getting tricked by bad angles. The best results came from training the AI specifically on this "verification" task, proving that for robots to be truly helpful, they need to be as careful as a detective, not just as fast as a delivery driver.

Technical Summary: PInVerify: An Offline Embodied Benchmark for Active Instance Verification

1. Problem Definition

Current embodied AI benchmarks (e.g., ObjectNav, OVON, PIN) primarily evaluate an agent's ability to navigate to the vicinity of a target object category or a language-specified instance. However, reaching the goal vicinity does not guarantee that the agent has identified the correct instance. Subtle attribute differences (e.g., "white floral" vs. "white striped" mugs) often require close-range, multi-view inspection to resolve. Existing navigation metrics treat arrival at the goal vicinity as success, masking a critical failure mode where an agent delivers the wrong object despite reaching the correct location.

The paper identifies a gap between geometric success (arriving near the target) and semantic success (verifying the instance identity against a fine-grained description). While some recent works insert "target verification" steps into navigation chains, verification remains embedded rather than isolated as a standalone decision problem. The authors argue for Active Instance Verification (AIV): a task where an agent, given a candidate object and a fine-grained natural language description, actively selects a sequence of viewpoints to decide whether the object matches the description.

2. Methodology and Benchmark Design

2.1. The AIV Task Formulation

AIV is formalized as a finite-horizon decision process. An episode is defined by a tuple $(Q, o_{query}, o_{target}, G)$ :

$Q$ : A set of natural language descriptions characterizing the query instance.
$o_{target}$ : The candidate object physically placed in the scene.
$G$ : A navigation graph around $o_{target}$ consisting of pre-captured viewpoints (nodes) and navigable transitions (edges).

State and Action:

State ( $S_t$ ): Includes the current viewpoint ( $p_t$ ), the observation (RGB image + visibility warnings), and an internal belief ( $B_t$ ) about the verification decision.
Action Space ( $A$ ): Includes navigation actions ( $NAV_{d_k}$ ) to move to one of $K=6$ azimuthal sectors, and terminal verification decisions ($YES, NO$).
Horizon: $T=6$ steps (each sector visited at most once).

Pair Types:
To distinguish between calibration bias and recognition ability, the evaluation set is balanced across three pair types:

Positive: $o_{query} = o_{target}$ (Same instance).
Neg Same: Same category, different fine-grained attributes (e.g., two backpacks with different colors).
Neg Diff: Different categories (e.g., backpack query vs. hat target).

2.2. The PInVerify Benchmark

PInVerify is an offline embodied benchmark built upon the PInNED dataset (HM3D scenes with Objaverse-XL objects). Key innovations include:

Data Scale: 3,000 evaluation episodes across 18 object categories (e.g., backpack, mug, keys).
Multi-View Topology: A 6-sector navigation graph with two observation rings (Far: 1.4–1.7m; Near: 0.9–1.2m).
Protocol-Level Annotations:
- Trap Views: Geometrically navigable sectors where the target's projected mask falls below a visibility threshold (uninformative due to occlusion or angle).
- Unreachable Sectors: Sectors blocked by walls or furniture where navigation fails.
- Stratified Pairs: Explicit separation of positive, neg-same, and neg-diff pairs to analyze failure modes.
Offline Nature: Pre-captured observations eliminate rendering stochasticity and isolate the verification decision from locomotion control, allowing controlled ablations on detection and reasoning.

2.3. Reference Agents

The authors provide two baseline agents to map the difficulty of PInVerify:

Training-Free Modular Pipeline:
- Attribute Decomposition: Uses an MLLM to break descriptions into structured attributes.
- Per-View Verification: Localizes the object (via Grounding DINO or GT boxes) and verifies attributes one by one.
- Confidence-Weighted Tracker: Accumulates evidence across views, down-weighting low-visibility (trap) observations to prevent spurious rejections.
- Next-Best-View (NBV): Compares Random, Angular Farthest-Point Sampling (FPS), and LLM-guided strategies.
LoRA-Fine-Tuned End-to-End Agent:
- Based on Qwen3-VL-4B, fine-tuned with LoRA.
- Trained via Supervised Fine-Tuning (SFT) on chain-of-thought traces, followed by alignment strategies (DPO, GRPO, GSPO).
- Performs verification and navigation in a single forward pass.

3. Key Results

3.1. Baseline Performance

MLLM vs. Embeddings: The best MLLM-based baseline (Qwen3-VL-4B with attribute decomposition and multi-view tracking) achieves 85.0% overall accuracy, exceeding the best embedding baseline (SigLIP2, 80.1%) by 4.9 percentage points (pp). The gap is significantly larger on positive pairs (59.6% vs. 42.9%).
Detection Bottleneck: Ablations using Ground Truth (GT) bounding boxes instead of Grounding DINO show a +3.1 pp gain in overall accuracy (and +10.6 pp on positive pairs), identifying detection quality as a major bottleneck, particularly for small objects (e.g., keys, watches).
NBV Strategies: Within the tested scale, different NBV strategies (Random, FPS, LLM-guided) yield results within the 95% confidence interval ( $\pm 1.3$ pp). No strategy shows reliable gains from active selection over random sampling in the training-free setting.

3.2. Trained Agent Performance

SFT Impact: A LoRA-fine-tuned agent (SFT) improves overall accuracy from 70.6% (base) to 84.8%, with a massive jump in positive pair accuracy (0.146 $\to$ 0.759).
Alignment: Post-SFT alignment with GSPO (Group Sequence Policy Optimization) further improves performance to 85.6% (88.9% with GT boxes).
Trade-offs: Trained agents exhibit a shift in calibration: they significantly improve positive confirmation but show reduced accuracy on "neg same" rejection compared to the training-free pipeline. They also demonstrate higher efficiency, reducing Average Steps to Decision (ASD) from 2.34 to 1.62 and navigation failure rates from 28% to 9%.

4. Significance and Claims

The paper positions PInVerify as a controlled testbed that complements, rather than replaces, existing navigation benchmarks. Its significance lies in:

Isolating Semantic Verification: By abstracting locomotion, it allows researchers to study the specific challenge of verifying fine-grained instance identity under occlusion and viewpoint uncertainty.
Revealing Calibration Issues: The results highlight that for on-device MLLMs ( $\le$ 8B parameters), calibration (the balance between confirming matches and rejecting mismatches) is as critical as raw capability. The best query strategy depends on the model's specific calibration profile.
Identifying Bottlenecks: The benchmark explicitly surfaces detection quality and the handling of "trap views" as primary failure modes, suggesting that future work must address robust open-vocabulary detection and uncertainty-aware view selection.
Offline Reproducibility: The use of pre-captured data ensures that evaluation results are not confounded by rendering variations or control stochasticity, facilitating fair comparison of verification algorithms.

The authors conclude that while current MLLMs can perform active verification, they are not yet robust enough to reliably select the "best" view in all scenarios, and detection quality remains a limiting factor for fine-grained semantic verification in embodied AI.