From "Strings" to "Things" for Personal Knowledge Graphs: Evaluating LLM Triple Extraction for Recommendation Systems

Technical Summary: From "Strings" to "Things" for Personal Knowledge Graphs

Problem Statement
Personal Knowledge Graphs (PKGs) offer a privacy-preserving framework for modeling user preferences, yet constructing them from unstructured, decentralized conversational data remains a significant challenge. Traditional recommendation systems relying on structured user-item relationships often depend on centrally logged interaction traces, which are frequently unavailable due to privacy concerns or infrastructural constraints. While Large Language Models (LLMs) excel at open-domain knowledge extraction, their capacity to construct PKGs directly from conversational interactions—bridging the gap between unstructured "strings" and semantic "things"—is underexplored. Furthermore, existing approaches often struggle with the inefficiency of processing raw conversational history on-device and the difficulty of merging unstructured summaries without redundancy.

Methodology
The authors propose a reproducible pipeline for extracting structured user-preference triples from natural language dialogues to construct PKGs. The methodology involves the following components:

• Pipeline Architecture: The system takes multi-turn dialogues between a user and an agent as input. An LLM processes the conversation to extract (User, Relation, Item) triples. These triples are normalized, aligned to external knowledge graph identifiers (specifically Wikidata IRIs), and integrated into a PKG.
• Semantic Schema: The pipeline utilizes an explicit User Movie Preference Ontology grounded in RDF. It defines two core entity types (pkg:User and pkg:Movie) and five object properties (pkg:likes, pkg:dislikes, pkg:hasSeen, pkg:hasNotSeen, pkg:wasSuggested). Entities are linked to stable, dereferenceable IRIs to ensure global interoperability.
• Dataset: The evaluation utilizes the Recommendation Dialogues (ReDial) dataset, which contains over 11,000 human-to-human conversations regarding movie recommendations. The dataset provides structured metadata annotations (ground truth) describing user preferences, allowing for a principled comparison between human-annotated data and LLM-extracted structures.
• Experimental Setup: The study evaluates open-weight LLMs from the Qwen3 and Gemma-3 families across various parameter sizes (ranging from 0.6B to 12B) and prompting regimes (0-shot to 10-shot). The extracted triples are used to augment the original dataset, replacing ground-truth annotations to create a controlled environment for testing downstream utility.
• Evaluation Metrics:
  • Upstream: Standard information extraction metrics (Precision, Recall, F1-score) are calculated by comparing extracted triples against ground-truth annotations.
  • Downstream: The utility of the extracted PKGs is assessed using the FedTREK-LM recommendation system. Models are trained on datasets augmented with LLM-extracted triples and evaluated on a separate test set to measure recommendation performance (F1-score) relative to models trained on original ground-truth data.

Key Contributions

1. Pipeline Development: A prompt-based PKG construction pipeline that converts conversational recommendation dialogues into structured RDF-compliant PKGs using an explicit semantic schema with IRIs.
2. Systematic Evaluation: The first large-scale assessment of conversational PKG extractability, benchmarking open-weight LLMs (Qwen3 and Gemma-3 variants) against the ReDial dataset's structured annotations.
3. Downstream Utility Analysis: An analysis demonstrating how extraction fidelity influences recommendation performance, showing that LLM-derived PKGs can serve as a foundation for decentralized, privacy-preserving personalized services.

Results

• Extraction Performance: The Gemma-3 Instruct family generally outperformed Qwen3 models. The best-performing model for extraction was Gemma-12B Instruct. Performance generally scaled with model size and shot count up to a point, but a significant drop in performance was observed at the 10-shot setting, suggesting overfitting to prompt patterns.
• Relation Specificity: The "liked" relation yielded the highest F1-scores across all models, as preference sentiment is often explicitly expressed. The "seen" relation was moderately successful, while "suggested" proved the most difficult, requiring long-range conversational memory.
• Downstream Impact:
  • There was a notable resilience in downstream performance relative to upstream extraction scores for smaller models. For instance, the Qwen3-0.6B model maintained over 75% of its ground-truth baseline performance in downstream tasks despite a low upstream extraction F1-score (0.3577).
  • Larger models (e.g., Qwen3-8B) experienced a "collapse" in downstream performance when trained on augmented data from their own extractions. This was attributed to the sparse and weak contextual foundation provided by their poor upstream extraction (specifically low recall), which bottlenecked their inherent reasoning capabilities.
  • Inverse Scaling Observation: In a cross-model evaluation using the Qwen3-0.6B downstream model, triples extracted by the smallest model (Gemma-1B) yielded the highest downstream F1-score (0.184), outperforming extractions from the larger Gemma-4B and Gemma-12B models. The authors attribute this to the 1B model's more balanced profile, capturing higher recall for critical "Liked" relations and better consistency in conversational flow, providing a richer foundation for learning than a sparse, high-precision graph.

Significance and Claims
The paper claims to validate a reproducible pipeline for building PKGs directly from dialogue, offering a scalable alternative to traditional approaches that rely on explicit, centralized logging. The core significance lies in demonstrating that:

1. Decentralized Construction is Feasible: Robust preference signals can be extracted using lightweight, open-weight LLMs, making on-device, privacy-preserving PKG construction viable without massive, computationally expensive models.
2. Extraction Quality vs. Downstream Utility: There is a complex relationship between extraction fidelity and downstream utility. High-precision but low-recall extractions from larger models can be detrimental to downstream tasks, whereas balanced, recall-oriented extractions from smaller models can provide a more effective foundation for learning.
3. Foundation for Adaptive AI: The work provides a diagnostic lens on current LLM capabilities in extracting user-specific knowledge, a critical step toward adaptive, conversational AI grounded in structured personalization. It highlights the potential for PKGs to support retrieval-augmented generation (RAG) and dynamic adaptation via knowledge editing, addressing challenges like the cold-start problem in personalized services.