🤖 machine learning

Recipe-Controlled Decoder Audit for Structural Knowledge-Graph Completion

This paper introduces a Recipe-Controlled Decoder Audit (RCDA) protocol to systematically evaluate the impact of decoder choices in structural knowledge-graph completion, revealing that decoder performance is highly sensitive to training recipes and dataset provenance, particularly on small knowledge graphs, and advocating for matched decoder comparisons before attributing gains to specific encoders.

Original authors: Xihang Shan, Ye Luo

Published 2026-06-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xihang Shan, Ye Luo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to figure out why a specific dish tastes better. You have three main levers you can pull:

The Recipe: The cooking instructions (heat, timing, spices).
The Chef: The person doing the cooking (their skill, the tools they use).
The Pan: The specific vessel the food is cooked in (a cast-iron skillet vs. a non-stick pan).

In the world of Artificial Intelligence, specifically for "Knowledge Graph Completion" (which is basically a computer trying to guess missing facts about the world), researchers have been obsessed with improving the Recipe and the Chef. But they often treat the Pan (called the "decoder") as a boring, fixed detail they don't need to talk about.

This paper, titled "Recipe-Controlled Decoder Audit," says: "Wait a minute. Let's swap the pan and see what happens."

Here is the breakdown of their findings using simple analogies:

1. The Core Experiment: The "Pan Swap"

The authors created a strict testing protocol (RCDA). They kept the Recipe (training instructions) exactly the same. They kept the Chef (the neural network structure) mostly the same. The only thing they changed was the Pan (the mathematical model used to score the answers, specifically swapping between two types called ComplEx and DistMult).

They asked a simple question: If I keep everything else identical, does changing the pan change the final taste (the score)?

2. The Findings: Big Pans vs. Small Pans

The "Big Kitchen" (Standard Datasets)
On large, complex datasets (like FB15k-237 or YAGO3-10), swapping the pan made a small but consistent difference.

The Result: One pan (ComplEx) was slightly better than the other (DistMult), but only by a tiny margin (about 0.5% to 1% better).
The Lesson: On big datasets, the pan matters, but it's not the most important thing. It's a subtle flavor note, not the main ingredient.

The "Tiny Kitchen" (Small Datasets)
On very small datasets (like Kinship or UMLS), the results were shocking.

The Result: Swapping the pan here didn't just change the flavor; it changed the winner. On the Kinship dataset, one pan was vastly superior. On UMLS, the winner even flipped depending on exactly how the experiment was set up (the "provenance").
The Lesson: On small datasets, the choice of pan is critical. If you pick the wrong one, your whole conclusion about whether your "Chef" is good or bad could be wrong.

3. The "Depth" Interaction

The authors also looked at how deep the "Chef" should be (how many layers of thinking the AI does).

The Twist: They found that the "best depth" depends entirely on which Pan you are using.
The Analogy: Imagine a deep pan works best with a slow-cook recipe, while a shallow pan works best with a quick-fry recipe. You can't just say "Deep is always better." If you use the wrong pan, the depth of your cooking might actually hurt the dish.

4. The "Saturation" Surprise

They looked at a specific dataset called YAGO3-10. Previous research suggested you needed a massive amount of computing power (a huge "pan") to get good results.

The Discovery: The authors found that with their specific recipe, you actually hit a "saturation point" very quickly. Once you reached a certain size, making the pan bigger didn't help much. It was like realizing you don't need a 10-gallon pot to boil an egg; a 2-gallon pot is already perfect.

5. The Main Takeaway (The "Audit")

The authors aren't trying to invent a new super-model. Instead, they are acting as auditors. They are telling the scientific community:

"Stop hiding the choice of the 'Pan' (decoder) in the fine print. If you claim your new 'Chef' (encoder) is amazing, you must prove it works with both pans. If you only test one, you might be fooling yourself, especially if you are working with small data."

In summary:

Don't ignore the tool: The mathematical model you choose to score your answers matters.
Context is key: It matters a little on big data, but it matters a lot on small data.
Be honest: When you publish your results, show the numbers for both types of pans so people know if your success is real or just a lucky choice of equipment.

The paper concludes with a simple checklist for researchers: Report the matched results, log the details of small datasets, and don't claim your method is the best until you've checked how it behaves with different "pans."

Technical Summary: Recipe-Controlled Decoder Audit for Structural Knowledge-Graph Completion

1. Problem Statement

Structural Knowledge Graph Completion (KGC) models rely on three independent components: the decoder (e.g., ComplEx, DistMult, RotatE), the encoder (e.g., CompGCN message passing), and the training recipe (loss functions, optimizers, hyperparameters). Prior research has extensively audited the impact of training recipes and encoder architectures. However, the decoder choice has largely been treated as an implementation detail or a fixed architectural choice rather than a controlled variable.

The paper identifies a critical gap: without a controlled comparison of decoders under a fixed recipe, it is difficult to determine whether performance gains are due to the encoder, the training recipe, or simply the decoder choice. This ambiguity acts as a "reporting confound," potentially leading to incorrect conclusions about the efficacy of new encoder architectures or training strategies.

2. Methodology: Recipe-Controlled Decoder Audit (RCDA)

The authors propose RCDA, a protocol designed to isolate the effect of the decoder by holding the training recipe and encoder architecture constant while varying the decoder.

Controlled Variables:
- Decoder: Primary comparison between ComplEx and DistMult, with targeted spot-checks for RotatE and TransE.
- Encoder: CompGCN-style message passing with varying depths ( $L \in \{0, 2, 3\}$ ), where $L=0$ represents pure embedding lookup.
- Recipe: A fixed, rigorous training recipe including 1-vs-all Cross-Entropy loss, label smoothing ( $\epsilon=0.3$ ), AdamW optimizer, MultiStepLR scheduling, and specific dropout/batch sizes.
Benchmarks: The audit covers seven datasets:
- Standard Large KGs: FB15k-237, WN18RR, YAGO3-10, CoDEx-M, CoDEx-L.
- Small KGs: UMLS and Kinship (selected for their low edges-per-relation ratios).
Evaluation: Standard filtered Mean Reciprocal Rank (MRR) and Hits@{1, 3, 10} with average-rank tie-breaking. Results are reported with multiple random seeds (3 for standard, 6 for small KGs) to ensure statistical robustness.

3. Key Findings and Results

A. Decoder Impact on Standard vs. Small KGs

Standard KGs: On the five standard benchmarks (FB15k-237, WN18RR, etc.), the performance gap between ComplEx and DistMult is modest but consistent. Under the fixed recipe, ComplEx outperforms DistMult by +0.005 to +0.012 MRR.
Small KGs: On UMLS and Kinship, the decoder choice becomes a decisive factor.
- Kinship: Shows a stable, large advantage for ComplEx (+0.143 MRR).
- UMLS: Demonstrates provenance sensitivity. In a clean server rerun, ComplEx leads by +0.022 MRR, but in an earlier run bundle with slightly different conditions, DistMult was the winner. This indicates that small-KG conclusions are highly sensitive to recipe variants and should not be treated as fixed dataset properties.

B. Encoder Depth and Decoder Interaction

The paper challenges the assumption that optimal encoder depth is a static property of a dataset.

WN18RR Interaction: The optimal encoder depth ( $L$ $L$ ) depends on the decoder.
- ComplEx peaks at $L=0$ (pure embeddings).
- DistMult peaks at $L=3$ (deeper message passing).
- Both decoders share a performance dip at $L=1$ .
Implication: Claims about encoder utility (e.g., "message passing hurts performance") are incomplete without specifying the decoder. On FB15k-237, the encoder helps ComplEx, but on WN18RR, it is neutral or harmful.

C. Capacity and Saturation (YAGO3-10)

Dimension Scan: Under the RCDA recipe, pure ComplEx ( $L=0$ ) on YAGO3-10 reaches near-saturation at embedding dimension $d=128$ , achieving 0.6971 ± 0.0048 MRR.
Diminishing Returns: Increasing dimensions to $d=256$ yields only a marginal gain (+0.008 MRR) in a single-seed check.
Recipe vs. Capacity: The significant performance jump from previous literature (e.g., NBFNet's 0.587) is attributed primarily to the training recipe (specifically the shift from sampled negative logistic loss to 1-vs-all CE and the use of AdamW) rather than simply increasing model capacity.

D. Descriptors for Decoder Sensitivity

Symmetry: While symmetry is often cited as the reason DistMult performs well on certain datasets, the authors find that dataset-level symmetry scores do not fully explain the performance gaps (e.g., UMLS is less symmetric than Kinship, yet Kinship shows a larger decoder gap).
Edges per Relation ( $e/r$ ): The authors propose $e/r$ as a useful audit descriptor. Small KGs with low $e/r$ (UMLS: 113, Kinship: 342) exhibit the largest decoder sensitivity, whereas standard KGs with high $e/r$ show consistent, smaller gaps.

4. Contributions

The paper outlines five specific contributions:

RCDA Protocol: A formalized audit protocol for structural KGC that isolates decoder effects under a fixed recipe.
Seven-Benchmark Diagnostic: A comprehensive evaluation separating shared-grid evidence (standard KGs) from small-KG decoder-only sensitivity.
Small-KG Insights: Evidence that Kinship has a stable ComplEx advantage, while UMLS is sensitive to provenance/recipe variants, challenging the notion of fixed dataset winners.
Decoder-Encoder Interaction: Demonstration that optimal encoder depth is decoder-dependent (specifically on WN18RR).
Reference Grid: Provision of a recipe-controlled structural reference, including a new state-of-the-art result for structural KGC on YAGO3-10 (0.6971 MRR) and CoDEx-M (0.5379 MRR).

5. Significance and Claims

The paper does not claim to introduce a new KGC architecture or to universally rank decoders. Instead, its significance lies in methodological rigor and reporting standards:

Decoder Choice is a Confound: The authors argue that decoder choice is a "reporting confound" that can alter the diagnostic conclusion of an experiment.
Actionable Recommendations:
- Papers should report a matched DistMult-vs-ComplEx row to contextualize encoder or recipe claims.
- Small-KG results should include provenance and recipe details due to their sensitivity.
- Encoder depth claims should be made only after sweeping the decoder $\times$ depth grid, rather than assuming a fixed optimal depth.
Limitations: The authors explicitly state their conclusions are limited to structural transductive KGC with the specific decoder/encoder families tested. They do not extend claims to inductive settings, text-augmented models, LLMs, or path-based reasoning (noting that path-based models like NBFNet still outperform structural models on WN18RR).

In summary, the paper advocates for a shift in how structural KGC experiments are reported, urging researchers to treat decoder selection as a critical experimental variable rather than a hidden implementation detail.