Technical Summary: Scaling Diverse Language Generation for 3D Visual Grounding (ViGiL3D++)

Problem Statement

3D Visual Grounding (3DVG) involves localizing objects in a 3D scene based on natural language descriptions. While essential for embodied agents in robotics and augmented reality, current 3DVG models struggle to generalize beyond simple linguistic patterns. Existing large-scale datasets suffer from a lack of linguistic diversity and often rely on manual annotation, which limits scalability. Recent attempts to scale datasets using Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) have produced queries that are either overly verbose, lack precise contrast between objects, or fail to synthesize scene information into cohesive sentences. Specifically, methods generating queries from images, captions, or scene graphs often fail to uniquely distinguish target objects from distractors or produce valid, diverse constraint sets.

Methodology: ViGiL3D++

The authors propose ViGiL3D++, a fully automated, scene-agnostic pipeline designed to generate diverse and valid 3DVG queries at scale. The method combines constraint sampling from scene graphs with the language generation capabilities of LLMs. The pipeline consists of three primary stages:

1. Scene Graph Extraction

The system constructs a semantic scene graph from 3D point clouds and RGB images.

• Object Labeling: It augments initial object labels with additional classes using an LLM to ensure varying granularity.
• Attribute Extraction via Cross-Referencing: To ensure consistency, the system identifies groups of similar objects (e.g., identical chairs) and prompts a Vision-Language Model (VLM) to extract attributes (color, material, function, etc.) for a representative object. These attributes are then consistently assigned to all objects in the group, preventing terminology mismatches that could confuse grounding models.
• Relationship Extraction: The system extracts both query-independent relationships (e.g., "on," "near") using geometric analysis and query-dependent relationships (e.g., "left of the agent," "tallest among") using VLMs and geometric reasoning. It also estimates room layouts to ground objects within specific spatial contexts.

2. Constraint Sampling

Instead of relying on simple templates, ViGiL3D++ employs a constraint sampler to select attributes and relationships that uniquely identify target objects.

• Target Selection: The system selects targets based on query type: zero-target (describing non-existent objects), single-target, or multi-target.
• Iterative Sampling: Constraints are iteratively sampled from the scene graph to reduce the pool of potential targets until only the desired targets remain. The sampler enforces priors on the distribution of constraint types (e.g., balancing attributes vs. relationships, including negation) and prioritizes constraints that significantly reduce the distractor pool.
• Logical Consistency: This stage ensures that the generated constraints are logically valid and traceable within the scene graph, addressing the "hallucination" issues common in direct LLM generation.

3. Rephrasing

Once a set of valid constraints is sampled, an LLM rephrases them into natural language queries.

• Diversity Enforcement: The LLM is prompted with in-context examples representing a wide range of linguistic patterns to ensure diversity in sentence structure and phrasing.
• Dense Annotation: The system preserves ID tags from the constraints, allowing for dense annotations of referenced objects (targets and anchors).
• Grammar Refinement: A final pass corrects grammatical errors and ambiguous phrasing while maintaining semantic fidelity to the original constraints.

Key Contributions

1. ViGiL3D++ Pipeline: A scalable method for generating diverse 3DVG queries that outperforms existing VLM-based generation methods in both validity and linguistic diversity.
2. Cross-Referencing Mechanism: A novel approach to attribute assignment that ensures consistent terminology across similar objects, thereby improving the precision of object contrast in generated descriptions.
3. Constraint Sampling: A strategy that enforces logical consistency and diversity in constraint types (including negation and complex relationships) before language generation, reducing the burden on the LLM to infer spatial logic.
4. Dataset and Benchmarks: The creation of a large-scale dataset (105K descriptions across 1,140 scenes) and the demonstration that training on this data improves model performance on diverse 3DVG benchmarks.

Results

The authors evaluated ViGiL3D++ through query validation, diversity analysis, and model training experiments.

• Query Validity: In a user study, ViGiL3D++ achieved a 57.1% validity rate (correctly describing only the intended targets), significantly outperforming baseline methods like SG2Query (52.7%) and 3D-GRAND (38.3% in the authors' re-evaluation). Baselines often failed to distinguish targets from distractors or generated ambiguous referents.
• Linguistic Diversity: ViGiL3D++ demonstrated superior representation of linguistic patterns compared to other scaled datasets (e.g., SceneVerse, 3D-GRAND). It showed higher frequencies of negation, ordinal relationships, and complex sentence structures, with an average constraint count and sentence length closer to human descriptions than other automated methods.
• Model Performance: Training a baseline model (V3DM) on ViGiL3D++ data resulted in competitive or superior performance across multiple benchmarks (ScanRefer, Multi3DRefer, and the diverse ViGiL3D benchmark) compared to models trained on other scaled datasets like ScanScribe or 3D-GRAND. Notably, the performance gains were attributed to the data diversity rather than architectural changes.

Significance and Claims

The paper claims that ViGiL3D++ addresses the critical bottleneck of scaling diverse 3DVG datasets without manual annotation. By decoupling logical constraint sampling from language generation, the method produces queries that are both valid and linguistically rich, enabling models to learn more robust spatial reasoning.

The authors highlight that while their method improves model performance, it also illuminates outstanding limitations in current VLMs, particularly in precise spatial reasoning and handling long lists of constraints. They argue that the key contribution is not just the dataset size, but the diversity and validity of the generated descriptions, which are essential for training agents to handle the full spectrum of referring expressions in 3D environments. The work suggests that future progress in 3D vision-language tasks depends on moving beyond simple template-based or verbose captioning methods toward structured, constraint-driven generation.