Technical Summary: HOME-KGQA

Problem Statement

Current Knowledge Graph Question Answering (KGQA) benchmarks are predominantly biased toward encyclopedic knowledge (e.g., DBpedia, Freebase, Wikidata) and rely on single-modality, static facts. These datasets fail to address the needs of Embodied AI and real-world scenarios, such as household environments, where questions require fine-grained spatiotemporal reasoning, multimodal grounding, and the analysis of episodic events (e.g., "When did the agent put the glass in the kitchen?"). Existing benchmarks lack the complexity to evaluate models on multi-hop reasoning involving time, space, and aggregate functions within dynamic, multimodal contexts. Furthermore, current KGQA systems often assume access to entity IDs, a condition rarely met in natural human-robot interactions.

Methodology

The authors introduce HOME-KGQA, a novel benchmark dataset designed to evaluate KGQA performance on daily household activities. The construction process involves three main stages:

1. Episodic Multimodal Knowledge Graph (MMKG) Construction:

   • Source: The dataset is built upon VHAKG, an MMKG derived from synthetic daily activity videos generated by the VirtualHome simulator.
   • Schema: The KG integrates five distinct ontologies to handle heterogeneous data:
     • Event-centric structure for daily activities.
     • MSSN and SOSA for multimodal sensor data (images, videos).
     • Time Ontology for temporal points, intervals, and durations.
     • HomeOntology for activity categories (e.g., HouseCleaning).
     • X3D Ontology for 3D spatial coordinates and bounding boxes.
   • Scale: The authors probabilistically generated 100 daily-life episodes (each containing 18 activities) using a first-order Markov chain based on crowdsourced sequences. This resulted in an episodic KG containing over 150 million triples, with fine-grained temporal and spatial granularity.

2. Question and Query Generation:

   • Logic: Questions are generated by combining five elements: Agent, Action, Object, Space, and Time. Since the dataset simulates a single-person household, the "Agent" is fixed and excluded from query conditions.
   • Categories: The dataset covers diverse question types including Object properties, Actions, Spatial locations, Temporal intervals, Activity sequences, Video data, and Aggregation (count, min, max, average, sum).
   • Process: SPARQL queries are generated from templates and converted to natural language using an LLM (gpt-4o-mini) via few-shot prompting.
   • Paraphrasing: To improve naturalness, a retrieval-augmented generation (RAG) inspired approach is used. A gold dataset of 22 paraphrased pairs per question type is created, and the LLM uses the top-5 most similar examples to paraphrase raw questions, correcting grammatical errors and naturalizing time/attribute expressions.

3. Data Splitting for Generalization:

   • I.I.D. Generalization: Test sets contain unseen entities and literals, but all relations, classes, and logical operators are seen during training.
   • Compositional Generalization: Test sets contain unseen logical form constructs (specifically operators: COUNT, MIN, AVG, SUM, <, >), requiring the model to generalize to new query structures.

Key Contributions

1. First Episodic KGQA Benchmark for Home Environments: HOME-KGQA is the first dataset targeting episodic KGs in household settings, featuring fine-grained spatiotemporal structures and multimodal grounding.
2. Open Data Generation Pipeline: The authors provide the full code for episodic KG population, question/SPARQL generation, and paraphrasing, allowing for flexible dataset augmentation.
3. Comprehensive Experimental Analysis: The paper evaluates state-of-the-art methods (Text-to-SPARQL and Interactive Agent-based) on HOME-KGQA, revealing significant performance gaps compared to existing benchmarks.

Experimental Results

The authors evaluated two approaches: Text-to-SPARQL (direct generation using LLMs like GPT-4o-mini and Llama-3.1-8B) and Interactive-KBQA (an agent-based method using tools for entity linking and subgraph extraction).

• Performance Gap: Both approaches struggled significantly on HOME-KGQA compared to existing datasets (KQA Pro, WebQSP, CWQ, MetaQA).
  • In the Text-to-SPARQL setting, even with fine-tuning, accuracy on HOME-KGQA (Paraphrased) was the lowest among all tested datasets (0.148 for GPT-4o-mini vs. 0.628 on WebQSP).
  • Interactive-KBQA performed even worse, failing to answer any Video questions correctly and often hitting the maximum turn limit (20 turns) before finding an answer.
• Error Analysis:
  • Fine-tuned Text-to-SPARQL models produced 19.1% syntax errors in generated SPARQL.
  • Interactive-KBQA produced no syntax errors but failed to complete 73.8% of queries within the turn limit, indicating that the complex schema (integrating five ontologies) and lack of explicit entity labels require too many interaction steps.
• Generalization: Text-to-SPARQL showed better capability in acquiring compositional generalization than Interactive-KBQA in this specific environment, though absolute performance remained low.

Significance and Claims

The paper claims that HOME-KGQA highlights a critical challenge for the real-world deployment of KGQA systems. While LLMs excel at encyclopedic facts, they currently fail to handle the complex, multi-level spatiotemporal reasoning and multimodal grounding required for daily life applications. The authors assert that the significant performance drop on HOME-KGQA compared to traditional benchmarks underscores the need for new methods capable of navigating heterogeneous, fine-grained knowledge graphs without relying on pre-known entity IDs. They position HOME-KGQA as a foundational resource for advancing KGQA in Embodied AI and robotic navigation contexts.

The authors remain modest regarding limitations, noting that the dataset is based on synthetic single-person simulations and may lack the linguistic diversity of human-authored text. They acknowledge that the questions are template-generated and more complex than typical daily communication, but argue this complexity is necessary to evaluate the structural understanding of knowledge graphs in daily-life environments.