OSMa-Bench++: Toward Open-Ended Benchmarking of Semantic Mapping for Manipulation with Prompt-Generated Synthetic Scenes

Technical Summary of OSMa-Bench++

Problem Statement
Semantic mapping methods are increasingly utilized as intermediate scene representations for downstream robotic reasoning and manipulation. However, current evaluation frameworks are largely constrained by fixed benchmark datasets that fail to adequately cover manipulation-relevant corner cases, such as cluttered small objects, partial occlusions, and restricted access to target objects. While recent open-vocabulary and object-centric scene representation methods (e.g., OpenScene, OpenMask3D, ConceptGraphs, BBQ) have improved expressiveness, their assessment remains limited by closed scene collections. Furthermore, standard evaluation often relies on view-dependent observations, making it difficult to robustly assess scene graph completeness regarding object counts and inter-object relations when relevant objects are occluded or not jointly observable from a specific camera trajectory.

Methodology
The authors propose OSMa-Bench++, an extension of the original OSMa-Bench that introduces a controllable, prompt-driven pipeline for generating synthetic indoor scenes tailored for manipulation-oriented evaluation. The workflow consists of four primary stages:

1. Prompt Generation and Filtering: A Large Language Model (LLM) generates an initial pool of candidate prompts describing indoor scenes with object layouts and relations relevant to manipulation. These prompts are embedded into a semantic feature space. Using cosine similarity, near-duplicate prompts are removed, and a diverse subset is selected to ensure broad coverage of layouts and interaction scenarios.
2. Scene Synthesis and Conversion: The selected prompts are used to synthesize indoor environments using SceneSmith. Since SceneSmith outputs are in the Drake simulation format, a nontrivial intermediate layer is implemented to convert these assets into a Habitat-compatible format. This adaptation involves semantic normalization, material and texture repair, shader fallback policies, floor handling, navigation setup, and controlled lighting configuration.
3. Data Sequence Generation: The converted scenes are processed by the HaDaGe generator to produce RGB-D observation sequences along camera trajectories, mirroring the data generation process of the original OSMa-Bench.
4. Evaluation Protocol: The generated sequences are processed by semantic mapping approaches (specifically ConceptGraphs and BBQ in the experiments). Evaluation occurs via two metrics:
   • Segmentation Accuracy: Standard segmentation metrics.
   • Prompt-Grounded VQA: A novel evaluation component where the original scene-generation prompt serves as an auxiliary semantic specification. This allows for the generation of questions regarding object counts and relations that are independent of trajectory-dependent visibility. This "Prompt-Grounded" (PromptGT) protocol specifically targets the Measurements and Relations categories, which are typically unstable in standard view-dependent VQA.

Key Contributions

• Controllable Benchmarking Pipeline: The paper introduces a fully automated pipeline that extends OSMa-Bench to include prompt-generated synthetic scenes, enabling the targeted stress-testing of semantic representations under specific conditions (e.g., clutter, lighting variations, small objects).
• Prompt-Grounded Evaluation: A significant methodological contribution is the use of the generation prompt as a ground truth source for question answering. This decouples the evaluation of scene graph completeness (specifically object counts and spatial relations) from the limitations of camera trajectory visibility.
• Adaptation Layer: The work details the technical challenges and solutions required to bridge the gap between SceneSmith (Drake) and Habitat-based simulation, including asset repair and semantic normalization.

Experimental Results
The framework was evaluated on a dataset of 40 scenes (24 furniture-focused, 16 manipuland-focused) under four lighting conditions: Baseline, Camera Light, Dynamic Lights, and Nominal Lights.

• Segmentation Performance: Results indicate that Camera Light conditions cause the most significant degradation in segmentation performance for both ConceptGraphs and BBQ. Dynamic Lighting showed a mild effect, with some methods showing slight improvements in mean Accuracy (mAcc).
• Method Comparison: The two methods exhibited complementary failure modes. BBQ produced fewer but geometrically precise masks, resulting in higher frequency-weighted Intersection over Union (f-mIoU) but lower scene-graph completeness (filtering out objects). ConceptGraphs recovered more objects overall, yielding higher completeness but less precise masks.
• Prompt-Grounded QA: Both methods scored below 30% on prompt-grounded questions for Measurements and Relations, highlighting the difficulty of the task. However, distinct robustness profiles emerged:
  • BBQ generally outperformed ConceptGraphs on Relations and the Manipuland subset but suffered notable performance drops under Dynamic Lighting. This is attributed to BBQ's strategy of generating per-object embeddings from single images, making it sensitive to unfavorable frame conditions.
  • ConceptGraphs aggregated evidence across multiple views, stabilizing its embeddings under Dynamic Lighting and showing gains in that condition, particularly for Furniture categories.
• Standard vs. Prompt-Grounded: The study found that standard image-derived VQA often yielded higher accuracy scores than Prompt-Grounded QA for the same categories, suggesting that standard metrics may be inflated by trajectory-dependent biases where only visible objects are queried.

Significance
The paper posits that OSMa-Bench++ transforms synthetic scene generation into a practical tool for manipulation-oriented evaluation. Its primary significance lies in two areas:

1. Extensibility: It moves beyond fixed datasets, allowing researchers to generate diverse, challenging evaluation scenarios (e.g., specific clutter levels or lighting conditions) that are difficult to obtain in real-world fixed benchmarks.
2. Evaluation Stability: The prompt-grounded protocol provides a more stable mechanism for assessing scene graph completeness regarding object counts and inter-object relations, addressing a critical gap where standard view-dependent evaluation fails to capture the full intended scene structure.

The authors conclude that the framework effectively highlights how graph-level semantic completeness and segmentation robustness capture complementary aspects of representation quality, as evidenced by the divergent responses of BBQ and ConceptGraphs to lighting changes and scene composition.