Are a Thousand Words Better Than a Single Picture? Beyond Images -- A Framework for Multi-Modal Knowledge Graph Dataset Enrichment

1. Problem Statement

Multi-Modal Knowledge Graphs (MMKGs) integrate structured triples with unstructured data (text, images, audio) to improve entity representation and link prediction. However, existing MMKG datasets suffer from two critical limitations:

• Lack of Scalability: Current image collection relies heavily on manual curation by domain experts to ensure quality. This process is slow, expensive, and difficult to scale to large graphs, often resulting in datasets with sparse visual coverage.
• Exclusion of Ambiguous but Relevant Visuals: Curators typically discard images that are visually ambiguous (e.g., logos, symbols, abstract art) because they are hard to interpret or may introduce noise. However, these images often contain crucial semantic information (e.g., a city's coat of arms) that is lost when excluded. Directly feeding these ambiguous images into models often degrades performance due to poor visual embeddings.

The core challenge is how to scale image collection while simultaneously converting ambiguous visual signals into usable semantic representations without requiring manual intervention or modifying downstream model architectures.

2. Methodology: The "Beyond Images" Framework

The authors propose Beyond Images, an automatic, data-centric enrichment pipeline that operates in three stages. Crucially, this framework enriches the dataset rather than the model, meaning it can be applied to any standard MMKG model without architectural changes.

Stage 1: Modality Extension Module (Retrieval)

• Goal: Expand visual coverage beyond the original dataset.
• Process: The system automatically retrieves additional entity-related images from external sources (e.g., web crawlers, Wikipedia pages) using entity names and IDs (Wikidata QIDs).
• Outcome: A significant increase in the number of images per entity, capturing diverse visual perspectives including logos and abstract scenes that were previously excluded.

Stage 2: Semantic Generation Module (Image-to-Text)

• Goal: Convert all visual inputs (original + new) into textual descriptions to mitigate the "noise" of ambiguous images.
• Process: The framework employs state-of-the-art Image-to-Text (I2T) models (specifically blip2-flan-t5-xxl, git-large-coco, and llava-v1.5-7b) to generate detailed captions for every image.
• Rationale: By converting visuals to text, the framework ensures that even sparse-semantic images (like logos) or complex abstract scenes contribute explicit, usable semantics rather than relying on potentially weak visual embeddings.

Stage 3: LLM-based Semantic Fusion Module

• Goal: Consolidate multiple image descriptions into a single, coherent, entity-aligned summary.
• Process: A Large Language Model (LLM) (e.g., Mistral-7b, LLaMA-3.1, Flan-T5) takes all generated captions for a specific entity and fuses them into a single, rich paragraph.
• Function: The LLM filters out task-irrelevant content, removes redundancy, and aligns the descriptions with the entity's core semantics. This "Fusion" summary replaces or augments the standard text modality in MMKG models.

Quality Control

The authors also release a Text-Image Consistency Check Interface, a lightweight tool allowing human annotators to perform targeted audits on the generated summaries to ensure accuracy, though the pipeline is designed to function primarily automatically.

3. Key Contributions

1. Beyond Images Framework: A reusable, model-agnostic pipeline that scales MMKG datasets by retrieving new images and converting them into entity-aligned text summaries.
2. Handling Ambiguity: A novel approach to treating "ambiguous yet relevant" images (logos, symbols) not as noise, but as valuable data sources by converting them into explicit textual descriptions.
3. Tooling & Resources: Release of enriched datasets, code, and a human-in-the-loop auditing interface for quality assurance.
4. Comprehensive Evaluation: Systematic testing across three public datasets (MKG-W, MKG-Y, DB15K) and four distinct MMKG models (MMRNS, MyGO, NativE, AdaMF).

4. Experimental Results

The framework was evaluated on link prediction tasks using Mean Reciprocal Rank (MRR) and Hits@K metrics.

• Overall Performance: The "Fusion" approach (LLM summary of all image descriptions) consistently outperformed baselines across all models and datasets.
  • Gains: Up to +7% improvement in Hits@1 overall.
  • Consistency: Improvements were observed across MMRNS, MyGO, NativE, and AdaMF, demonstrating the method's generalizability.
• Impact on Ambiguous Images (The "Logo" Subset):
  • On a challenging subset of entities featuring logos and symbols, converting images to text yielded massive improvements.
  • Results: +201.35% increase in MRR and +333.33% increase in Hits@1. This proves that for sparse-semantic visuals, text representation is far superior to raw visual embeddings.
• Modality Ablation:
  • Replacing visual embeddings with image-generated text (T+G) performed comparably or better than using original images (I+T) for certain models (MyGO, NativE), suggesting that explicit text can effectively substitute for visual features in specific architectures.
  • The combination of all modalities (I+T+G) with LLM fusion yielded the best results, confirming the complementary nature of the data.
• Model Sensitivity: The blip2-flan-t5-xxl model for captioning and Mistral-7b for fusion produced the best results, highlighting the importance of generating semantically aligned, concise descriptions over verbose or generic ones.

5. Significance and Conclusion

The paper demonstrates that scaling image coverage and converting ambiguous visuals into text is a highly effective strategy for enhancing MMKGs.

• Paradigm Shift: It moves the focus from designing complex multi-modal model architectures to improving the quality and quantity of the data fed into existing models.
• Practicality: The approach is cost-effective (minimal training time increase) and does not require retraining models from scratch; it simply requires updating the input text modality.
• Future Impact: By making "ambiguous" data usable, this framework unlocks a vast amount of previously discarded web data, potentially leading to more robust and diverse knowledge graphs capable of handling complex real-world entities.

In summary, Beyond Images proves that "a thousand words" (generated from a thousand images) are indeed better than a single picture when the goal is to maximize the semantic utility of multi-modal knowledge graphs.