Technical Summary: Enhancing Small Language Models Reasoning through Knowledge Graph Grounding

Problem Statement

While Large Language Models (LLMs) have established benchmarks for zero-shot reasoning, their deployment is often cost-prohibitive and environmentally taxing. Small Language Models (SLMs) offer a sustainable alternative but struggle significantly with complex, multi-hop relational reasoning tasks, such as inferring kinship relations from narrative text (e.g., the CLUTRR benchmark). The primary limitations of SLMs in this domain are:

• Sequential Deductive Fragility: As reasoning depth increases, minor errors in early fact extraction act as false premises, compounding through the chain and leading to catastrophic logical failures.
• Symbolic State Maintenance: SLMs often fail to maintain a consistent symbolic state over long contexts, leading to "hallucinated" relations.
• The "Lost-in-the-Middle" Phenomenon: SLMs struggle with logical chaining in zero-shot settings, particularly when dealing with compressed architectures under 4 billion parameters.

Methodology

The authors propose a neuro-symbolic agentic framework that transforms the SLM from a standalone reasoner into a minimalist agent. This approach decouples text understanding from relational logic by utilizing two specialized tool calls:

1. extract_facts (Symbolic Extraction): The SLM parses the narrative story to extract kinship triplets $(u, rel, v)$. To align with the dataset's logical schema, the system adopts a reversed edge direction where the triplet represents the relationship from the perspective of the target node (e.g., an edge from father to daughter is labeled "daughter"). The output is a structured Knowledge Graph (KG).
2. get_hint (Neural Expert Reasoning): A pre-trained Relational Graph Convolutional Network (RGCN) processes the KG generated by the SLM. The RGCN acts as an expert, returning the most probable kinship relation and a confidence score for a given query pair of entities. This "hint" is injected back into the SLM's context to assist the final inference.

Experimental Configurations:
The study evaluates the framework across two scenarios using Gemma 3 (1B, 4B) and Llama 3.2 (3B) models:

• Oracle Scenario: The system is provided with ground-truth triplets and hints to establish the theoretical upper bound of reasoning capacity.
• Realistic Scenario: The SLM must extract its own facts via zero-shot prompting, introducing extraction noise (hallucinated or missing relations).

To mitigate extraction noise, the framework incorporates inverse triplets post-hoc to ensure bidirectional information flow. The RGCN is trained on reasoning chains of 2–4 hops and evaluated on test sets requiring up to 10 hops to test systematic generalization.

Key Contributions

1. Framework for Zero-Shot Relational Learning: The authors introduce a neuro-symbolic framework that enhances SLM performance. In realistic settings where the SLM extracts its own facts, the method yields a 1.5 to 2x performance gain over story-only baselines.
2. Comparative Architecture Analysis: A study across different open-source models (Gemma 3 1B/4B, Llama 3.2 3B) reveals how model size and architecture influence the quality of symbolic extraction and resilience to noise.
3. Identification of Failure Modes: An extensive analysis comparing realistic pipelines against Oracle configurations isolates specific failure modes, including the "extraction bottleneck" and a "distraction effect" where noisy self-generated facts degrade performance despite the presence of expert hints.

Results

• Oracle Performance: When provided with ground-truth hints, SLMs show significant improvement. For instance, Llama 3.2 3B accuracy jumps from 16.13% (Story only) to 62.79% (Story + Oracle Hint), suggesting the primary barrier is not linguistic comprehension but the maintenance of symbolic state.
• Realistic Performance & The Extraction Bottleneck: In the realistic scenario, the RGCN's accuracy drops from 60.69% (on Oracle facts) to 24.88% (on SLM-extracted facts). This confirms that a single error in the initial extraction phase renders the knowledge graph logically unrecoverable for the GNN solver.
• The "Distraction Effect": The study identifies a counter-intuitive phenomenon where providing both noisy SLM-extracted facts and expert hints can degrade performance. For Gemma 4B, the configuration with only the GNN hint (19.75%) outperformed the configuration with both SLM facts and the hint (14.98%). This suggests that noisy triplets act as "logical anchors" that mislead the SLM. Llama 3.2 3B demonstrated greater resilience to this effect.
• Architectural Differences: Llama-based models exhibited better robustness as reasoning depth increased toward 10 hops, whereas Gemma variants showed immediate performance decay beyond the 4-hop threshold, likely due to susceptibility to the "Lost-in-the-Middle" phenomenon.

Significance and Claims

The paper characterizes the challenges of symbolic grounding in low-resource agentic systems. Its primary claim is that while SLMs possess the latent capacity for complex relational reasoning when guided by high-quality symbolic structures, their utility is currently constrained by the quality of the extraction phase.

The authors conclude that the bottleneck is not the reasoning capacity of the GNN expert, but the sequential deductive fragility introduced by the SLM's initial fact extraction. They argue that for neuro-symbolic pipelines to be reliable in open-domain environments, future work must focus on iterative verification and symbolic refinement to prune extraction hallucinations, rather than simply relying on the reasoning capabilities of the expert module.