Technical Summary: LLM Features Can Hurt GNNs

Problem Statement

While recent literature widely reports that integrating Large Language Model (LLM)-generated node features into Graph Neural Networks (GNNs) improves accuracy on standard benchmarks (e.g., TAPE, GLEM), this paper identifies a contrasting failure mode. Specifically, when LLM features are introduced via pure input concatenation—without joint training, distillation, or prompt-conditioning—they can systematically degrade performance on homophilous benchmarks where end-to-end LLM pipelines typically succeed.

The authors isolate the "pure concatenation" sub-step to demonstrate that, under specific conditions (notably small-sample, high-dimensional settings), adding LLM features acts as a source of interference rather than signal, leading to significant accuracy drops.

Methodology

The study employs a rigorous experimental design across nine node classification datasets (spanning homophily levels $h \in [0.05, 0.81]$) to isolate the effects of LLM features.

• Experimental Setup:

  • Features: Original features ($F_{orig}$, e.g., BoW, Word2Vec) are concatenated with LLM-generated features ($F_{LLM}$, SBERT-encoded GPT-4o-mini TAPE explanations).
  • Backbones: Experiments utilize an MLP backbone (primary) and GCN/GCNII/GAT backbones (secondary) to test architectural dependency.
  • Splits: Primary results use the standard Planetoid public split (low-label regime: 60–140 training labels), with random splits used to verify regime dependence.
  • Metrics: The core metric is $\Delta_{concat}$, the test accuracy difference between $F_{orig} \parallel F_{LLM}$ and $F_{orig}$ alone.

• Analytical Frameworks:

  • Direct Contrast: Comparing specific coalitions (e.g., $\{F_{orig}, F_{LLM}\}$ vs. $\{F_{orig}\}$) to avoid diluting interference effects.
  • Shapley Decomposition: A 4-factor decomposition (Structure, $F_{orig}$, $F_{LLM}$, Depth) over 12 effective coalitions to attribute marginal contributions, acknowledging that averaging can mask specific failure modes.
  • Ablation Studies: Controlled experiments replacing $F_{LLM}$ with zeros, PCA projections of $F_{orig}$, or Gaussian noise to distinguish between dimensionality penalties, weight decay artifacts, and informational content effects.
  • Scaling Analysis: Fitting a power-law profile to the relationship between concatenation cost and the ratio of feature dimension to training samples ($\sqrt{d_l/n}$).

Key Contributions & Findings

F1: The Regime of Interference

In the low-label, high-dimensional regime (specifically the Planetoid public split), pure concatenation of LLM features degrades performance.

• Observation: On PubMed, adding LLM features drops MLP accuracy by $-17.0 \pm 0.3$ percentage points (pp). Cora drops by $-4.3 \pm 0.6$ pp. CiteSeer shows a negligible drop within noise.
• Mechanism: This is not a generic "curse of dimensionality." Ablation shows that while pure Gaussian noise causes a $-37.3$ pp drop, real LLM features cause a $-17.0$ pp drop. The LLM features provide a "content rescue" of $\sim 20$ pp over pure noise, but the net result remains a significant penalty due to overfitting in the small-sample regime.
• Regime Dependence: When using larger random splits (50/25/25), the magnitude of the drop compresses to $\le 1.2$ pp, confirming the issue is driven by small-sample overfitting rather than inherent content flaws.

F2: Predictive Threshold via LLM-Only Discriminability

The paper proposes $\Delta_{sig}$ (the accuracy gap of an MLP trained only on $F_{LLM}$ vs. random noise) as the predictor for concatenation success.

• Threshold Rule: A bootstrap-derived change-point $\tau = 13.8$ pp separates regimes.
  • If $\Delta_{sig} \le \tau$: Concatenation cost is non-positive (likely to hurt).
  • If $\Delta_{sig} > \tau$: Concatenation cost is positive (likely to help).
• Performance: This rule correctly classifies 7/9 datasets.
• Comparison to Homophily: Homophily ($h$) is a poor predictor ($r^2 = 0.06$), whereas $\Delta_{sig}$ correlates significantly better ($r^2 = 0.38$).

F3: Mechanism Isolation

Through ablations on PubMed, the authors rule out alternative hypotheses:

• Dimensionality: Replacing LLM features with same-dimension zeros causes no degradation ($+0.1$ pp), ruling out simple dimension mismatch.
• Self-Information: Replacing with PCA of original features causes only mild interference ($-2.3$ pp).
• Regularization: Halving weight decay does not change the drop, ruling out weight decay artifacts.
• Conclusion: The $-17$ pp drop is specific to the informational content of the LLM features interacting with the finite-sample constraint.

F4: Shapley Decomposition Context

While direct concatenation shows negative effects, a 4-factor Shapley decomposition shows that $F_{LLM}$ has a positive (but small) aggregate contribution on heterophilous datasets. This highlights that the aggregate "success" reported in prior benchmarks often relies on joint training or selector modules that mitigate the raw concatenation cost.

F5: Scaling Law

The concatenation cost follows a power-law scaling with respect to the ratio of feature dimension to training labels:
$$|\Delta_{concat}| \propto \left(\sqrt{\frac{d_l}{n}}\right)^{1.31}$$
This profile ($r^2 = 0.97$) explains the $-17$ pp drop at $n=60$ and predicts the attenuation of the cost as $n$ increases (e.g., dropping to $<1$ pp at $n \approx 10^4$).

Remediation Strategies

The paper evaluates "cheap fixes" for the low-$\Delta_{sig}$ regime:

1. Drop $F_{LLM}$: The most effective solution. Removing the LLM channel entirely restores the baseline accuracy ($72.1\%$), strictly dominating all learned fixes.
2. Learnable Scalar Gate: A learnable scalar $g$ applied to $F_{LLM}$ (initialized at 0) recovers 89% of the gap. However, it still underperforms the "drop" baseline by $\sim 1.8$ pp ($p \approx 0.008$).
   • Recommendation: Use the gate only when downstream pipelines structurally require the presence of the LLM channel (e.g., shared multi-task inputs). Otherwise, dropping the channel is superior.

Significance and Claims

The paper does not claim that LLM features are universally harmful. Instead, it characterizes a specific regime where the simplest integration method (pure concatenation) backfires.

• Reconciliation with Prior Work: The findings do not contradict aggregate benchmarks (e.g., Wu et al. [5]) that report gains. Those gains are attributed to joint training, variational EM, or gated-fusion modules (as seen in TAPE, GLEM) which provide the necessary inductive bias to prevent the interference observed in pure concatenation.
• Practical Guidance:
  • Practitioners should A/B test $F_{orig}$-only vs. $F_{orig} \parallel F_{LLM}$ before adoption.
  • $\Delta_{sig}$ serves as an interpretive lens for the regime (helping vs. hurting) rather than a precise pre-A/B filter, as measuring it requires similar compute to the A/B test itself.
  • In low-$\Delta_{sig}$, low-label regimes, the "trivial fix" of dropping LLM features is often the optimal strategy.

The paper concludes that the "sign flip" from helpful to harmful is driven by the specific design choice of pure concatenation in small-sample settings, and that the aggregate success of LLM-enhanced GNNs relies on more sophisticated integration scaffolding that mitigates this specific interference.