💬 NLP

Dual-Path LLM Reasoning for Multimodal Few-Shot Knowledge Graph Completion

The paper proposes DuPLeR, a dual-path LLM reasoning framework that integrates multimodal LLM-derived priors with factual graph structures to achieve robust few-shot knowledge graph completion by mitigating noise and enhancing entity representations through calibrated relation graphs and dual-level structural reasoning.

Original authors: Jinlan Liu, Zhiying Tu, Yongchao Xing, Yicheng Liu, Bolin Zhang, Dianbo Sui, Dianhui Chu, Hongliang Sun

Published 2026-07-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jinlan Liu, Zhiying Tu, Yongchao Xing, Yicheng Liu, Bolin Zhang, Dianbo Sui, Dianhui Chu, Hongliang Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to finish a giant, ever-growing puzzle of the world's facts, where every piece is a connection between two things, like "Elvis Presley" and "King of Rock and Roll." This is what scientists call a Knowledge Graph. It's a digital map of how everything relates to everything else, powering smart assistants and search engines. But here's the catch: the real world is messy. New facts pop up every day—new celebrities, new scientific discoveries, new slang words—and the map often doesn't have enough pieces to connect them yet. This is the problem of Knowledge Graph Completion: trying to guess the missing links.

Usually, computers are great at this if they've seen the pieces before. But what if you hand them a brand-new puzzle piece they've never seen, with only one or two other pieces nearby to give a hint? That's the few-shot or zero-shot challenge. It's like being asked to guess the ending of a story you've never read, based on just the first sentence. To help, researchers are now borrowing a trick from Large Language Models (LLMs)—the super-smart AI chatbots that have read almost everything on the internet. These AIs have a "gut feeling" about how things usually connect, even if they haven't seen the specific puzzle before. However, these AIs can sometimes be too confident, making up facts (hallucinations) or getting confused by pictures that don't quite match the text. The big question is: How do we use this AI "gut feeling" to fill in the blanks without letting it make up nonsense?

Enter DuPLeR, a new framework proposed by researchers Jinlan Liu and colleagues that acts like a super-organized detective for these missing puzzle pieces. Instead of just asking the AI "What do you think?" and hoping for the best, DuPLeR uses a clever dual-path strategy to cross-check the AI's guesses against real evidence.

First, the system asks the AI to predict what types of things might be connected (for example, guessing that a "music artist" is likely connected to a "song"). But the AI isn't trusted blindly. The system builds a "calibrated" map, taking the AI's suggestions and scrubbing away the parts that don't match the actual, hard facts already known. It's like a teacher reviewing a student's essay: they keep the good ideas but cross out the made-up details that don't fit the evidence.

Then, the real magic happens with the dual-path multimodal enhancement. Imagine you are trying to identify a suspect in a lineup.

Path One (The Detective's Focus): As the system looks at the specific clues for the current question, it uses a "query-specific" filter. It asks the AI, "Hey, looking at this specific relationship, which parts of the photo or text description actually matter?" It ignores the background noise and focuses only on the relevant visual or textual hints.
Path Two (The General Knowledge): After the detective has done their specific work, the system takes a step back and looks at the whole picture again. It grabs the full, unfiltered description and image of the suspect to make sure it hasn't missed any big, general details that the specific filter might have thrown away.

By combining these two paths—one that is laser-focused on the specific question and one that keeps the big picture in mind—the system creates a much stronger, more accurate guess about the missing link.

The researchers tested this on eight different versions of two major knowledge graph benchmarks, simulating scenarios where the computer had to learn from just one or three examples (few-shot) or even zero examples (zero-shot). The results were promising: DuPLeR consistently outperformed older methods and even other AI-assisted approaches. For instance, on the FB15K-IMG-R dataset, DuPLeR achieved a Hits@10 score of 71.77% in a 1-shot setting, significantly beating the previous best of 61.20%. In zero-shot scenarios, where no examples were provided at all, the system still managed to improve predictions, suggesting that this dual-path method helps the AI stay grounded in reality even when it's flying blind.

The study suggests that while AI can provide a helpful starting point for understanding new facts, it needs a "reality check" from actual data and a smart way to balance specific clues with general knowledge. DuPLeR doesn't claim to have solved the puzzle of the universe, but it offers a robust new way to fill in the blanks when information is scarce, ensuring that our digital maps of the world stay complete and reliable.

Technical Summary: DuPLeR – Dual-Path LLM Reasoning for Multimodal Few-Shot Knowledge Graph Completion

1. Problem Definition

The paper addresses Inductive Knowledge Graph Completion (KGC) in few-shot and zero-shot settings, specifically within Multimodal Knowledge Graphs (MMKGs).

The Challenge: Traditional transductive KGC methods fail when encountering new entities and relations not seen during training. In real-world scenarios, emerging entities often have extremely sparse local neighborhoods (few-shot) or no support triples at all (zero-shot).
The Multimodal Constraint: While multimodal information (textual descriptions, images) can supplement sparse structural data, indiscriminate fusion introduces noise, cross-modal heterogeneity, and hallucinations. This is particularly detrimental in few-shot settings where noisy modalities can distort entity representations.
The LLM Gap: Large Language Models (LLMs) offer rich world knowledge to enrich sparse contexts, but existing methods often fail to explicitly handle modality-specific reliability or the granularity mismatch between LLM-derived semantic priors and factual multimodal evidence in MMKGs.

2. Methodology: DuPLeR Framework

The authors propose DuPLeR (Dual-Path LLM Reasoning), a framework designed to integrate multimodal LLM-derived priors with structural reasoning while mitigating noise. The architecture consists of four core components:

2.1 Dual-Level Structural Reasoning

To exploit limited supervision, DuPLeR performs reasoning at two complementary levels:

Relation-Level (CCoRGE): A Context-Enhanced Co-occurrence Relation Graph Encoder constructs a high-order relation graph. It captures transferable interaction patterns among relations based on topological co-occurrence (e.g., how relations share head/tail entities). This level is less dependent on specific entity identities.
Entity-Level (MAPNet): A Multimodal-Aware Prediction Network uses the relation embeddings from CCoRGE to guide message passing over the entity graph. It identifies relation-aware structural paths and scores candidate entities.

2.2 Prior Knowledge Injection (LLM Integration)

To address the scarcity of relation-associated triples, DuPLeR leverages multimodal LLMs to infer plausible head and tail entity types for unseen relations.

Prompting Strategy: The framework uses carefully constructed prompts that treat images as optional evidence. The LLM is instructed to use visual information only when relevant to the relation and consistent with text, otherwise ignoring it to reduce visual noise.
Topology Refinement: To mitigate LLM hallucinations, a factual-triple-based refinement step is introduced. It constructs a preliminary relation graph by merging the LLM-derived semantic graph ( $G_{sem}$ ) with the structural graph ( $G_{ins}$ ). An indirect noise filtering strategy then removes edges connecting the target relation to neighbors with high "noise degrees" (discrepancies between the merged and structural graphs), resulting in a calibrated relation graph ( $G_R$ ).

2.3 Dual-Pathway Multimodal Enhancement

Recognizing that neither relation-conditioned filtering nor query-agnostic fusion alone is sufficient, DuPLeR employs two complementary pathways to inject multimodal signals:

Query-Specific Regulation (Process Injection Path): During structural message passing, text and visual features are dynamically fused based on the query relation. A relation-guided low-rank attention mechanism and modality-specific low-rank gates control the contribution of visual signals. This acts as a soft regulator, ensuring structural reasoning remains dominant while being guided by query-relevant multimodal cues.
Query-Agnostic LMF-Max (Post Injection Path): After graph propagation, a Low-Rank Fusion with Max pooling (LMF-Max) module compensates for information loss. It captures salient visual evidence (using masked max pooling to exclude invalid images) and assigns learnable weights to different low-rank semantic subspaces. This pathway provides holistic, query-independent entity-level semantics to the final prediction.

2.4 Pre-training and Inference

DuPLeR is pre-trained on multiple MMKGs with shared trainable components. During inference on unseen graphs, it derives representations for unseen relations via LLM outputs, injects them into the graph, and performs link prediction without retraining on the target graph.

3. Key Contributions

The paper claims the following contributions:

DuPLeR Framework: A novel approach integrating multimodal LLM-derived relational priors with structural reasoning to mitigate representation distortion caused by sparse and noisy supervision in inductive KGC.
Dual-Pathway Mechanism: A design that separates multimodal integration into query-conditioned regulation during message passing and query-independent supplementation after propagation, enabling robust multimodal fusion.
Empirical Validation: Extensive experiments on eight inductive variants of two MMKG benchmarks (FB15K-IMG-R and OpenBG-IMG-R) demonstrating consistent improvements in inductive generalization and robustness.

4. Experimental Results

The authors evaluated DuPLeR against three categories of baselines: traditional KG embeddings (DistMult, RotatE), unimodal inductive models (InGram, ULTRA, ProLINK), and multimodal models (MMSN).

Performance: DuPLeR consistently outperformed all baselines across 1-shot and 3-shot settings on both FB15K-IMG-R and OpenBG-IMG-R.
- On FB15K-IMG-R (v1, 1-shot), DuPLeR (using Qwen3-8B) achieved 71.63% Hits@10, significantly surpassing ProLINK (61.40%) and ULTRA (27.00%).
- On OpenBG-IMG-R (v1, 1-shot), DuPLeR achieved 25.73% Hits@10, compared to ProLINK's 22.37% and MMSN's 11.17%.
Zero-Shot Capability: In zero-shot settings, DuPLeR showed clear improvements over ProLINK, particularly with weaker LLM backbones (e.g., Llama2-7B), validating its ability to mitigate information scarcity.
Ablation Studies: Removing the multimodal LLM prior caused a ~28.2% relative performance drop in zero-shot settings, highlighting its critical role. Removing the dual-path multimodal enhancement also significantly degraded performance, confirming the necessity of both injection pathways.
Robustness: The model maintained high performance across different LLM backbones (Llama2, Qwen, DeepSeek), demonstrating that the dual-path multimodal module effectively complements the relational context even when the LLM's prior knowledge is insufficient.

5. Significance and Claims

The paper positions DuPLeR as a principled solution to the inductive few-shot KGC problem in multimodal environments. Its significance lies in:

Bridging the Gap: It effectively aligns LLM semantic priors with multimodal cues, addressing the reliability and granularity mismatch issues inherent in MMKGs.
Noise Mitigation: By introducing a topology refinement module and a dual-pathway injection mechanism, it specifically targets the noise and hallucination risks associated with using LLMs and raw multimodal data in data-scarce scenarios.
Scalability: The framework enables precise link prediction for emerging entities without requiring retraining, making it suitable for scalable intelligence applications where annotation costs are high.

The authors conclude that while LLM-derived priors provide essential relational context, the dual-path multimodal enhancement is crucial for grounding these priors in factual, entity-level semantics, thereby achieving robust inductive generalization.