Technical Summary: LANTERN – LLM-Augmented Neurosymbolic Transfer with Experience-Gated Reasoning Networks

Problem Statement

Reinforcement Learning (RL) often struggles with sample inefficiency, particularly in tasks involving long-horizon, non-Markovian objectives that require structured temporal reasoning. While neurosymbolic approaches—integrating symbolic representations like Deterministic Finite Automata (DFAs) into RL—have improved sample efficiency, existing transfer learning methods face three critical limitations:

1. Manual Specification: Most methods require expert-provided DFAs or temporal logic formulas, which are difficult to obtain. Grammatical inference approaches exist but require structured trajectory data unsuitable for sparse or exploratory RL.
2. Single-Source Reliance: Current transfer methods (e.g., automaton distillation, policy distillation) typically rely on a single source task. This limits effectiveness when source and target tasks have heterogeneous goals or only partially aligned structures.
3. Fixed Integration Mechanisms: Existing frameworks use predetermined weighting schemes (e.g., exponential decay) or static hyperparameters, failing to adapt when source relevance varies across states or over time.

The paper addresses the challenge of multi-source neurosymbolic transfer in settings where multiple source tasks possess distinct, partially related goals. In such regimes, direct reuse of a single source policy or automaton is insufficient; instead, the system requires semantic alignment and the aggregation of structured knowledge across heterogeneous tasks.

Methodology: The LANTERN Framework

LANTERN (LLM-Augmented Neurosymbolic Transfer with Experience-Gated Reasoning Networks) is a unified framework designed to overcome the above limitations through five distinct phases:

1. LLM-Enhanced Automaton Generation

Instead of manual specification, LANTERN utilizes Large Language Models (LLMs) to generate target DFAs ($D_{tgt}$) directly from natural language task descriptions ($T_{desc}$).

• Process: A structured prompt instructs the LLM to extract subgoals, define automaton states representing milestones, specify deterministic transitions, and generate natural language semantic descriptions for each state.
• Output: A target DFA and a mapping of automaton states to semantic descriptions, which serve as the basis for cross-task alignment.

2. Semantic Embedding and Neighborhood Construction

To enable transfer across tasks with different goals, LANTERN constructs a shared semantic embedding space.

• Embedding: Natural language descriptions of automaton states (from both source and target tasks) are mapped to a vector space $\phi(\omega) \in \mathbb{R}^d$ using a fixed text-embedding model (e.g., sentence-BERT).
• Similarity & Neighborhoods: Semantic similarity between a target state $\omega_{tgt}$ and source states $\omega_{src}^k$ is computed via cosine similarity. For each target state, a "semantic neighborhood" ($N_M$) is formed by selecting the top-$M$ most similar source states across all $K$ source tasks.
• Weighting: Aggregation weights are normalized based on similarity scores, ensuring that semantically aligned states contribute more strongly to guidance.

3. Multi-Source Knowledge Aggregation

LANTERN aggregates guidance from the semantic neighborhood in two dimensions:

• Strategic Guidance (Automaton-level): Aggregates distilled Q-values ($Q_{k,AD}$) representing high-level task progression from source tasks. The aggregated value $Q_{agg}^{AD}$ is a weighted sum of source strategic values.
• Tactical Guidance (Policy-level): Aggregates teacher policies ($\pi_{teacher}^k$) from aligned source states to provide action-level guidance.

4. Dual-Volatility Experience Gating

To adaptively balance student learning with teacher guidance, LANTERN introduces a composite trust gate $\tau$ that combines two volatility measures:

• Experience Volatility ($V^{exp}$): Tracks the Temporal-Difference (TD) error volatility of the student's learning. High volatility indicates instability, favoring teacher guidance.
• Semantic Volatility ($V^{sem}$): Measures the misalignment between the target state and its best-matching source states (based on similarity). High semantic volatility indicates poor alignment, reducing trust in the source.
• Mechanism: The final trust coefficient $\tau = \tau^{exp} \cdot \tau^{sem}$ is multiplicative. Teacher influence is strong only when learning is unstable and the source is semantically aligned. This prevents negative transfer in poorly aligned regions.

5. Unified Learning Update

The student learns on the product MDP ($M_{tgt} \times D_{tgt}$) using a modified Q-update rule:
$$\Delta Q = \alpha [ \tau \cdot \delta_t + (1 - \tau) \cdot G_{multi} ]$$
where $\delta_t$ is the standard TD error and $G_{multi}$ is the aggregated multi-source guidance (combining strategic rewards and tactical policy discrepancies). When $\tau \to 1$, the update relies on standard TD learning; when $\tau \to 0$, it relies on aggregated teacher guidance.

Key Contributions

1. Formulation of Multi-Source Neurosymbolic Transfer: The paper formulates a setting where source tasks have heterogeneous goals, necessitating semantic aggregation rather than direct reuse of a single source.
2. LANTERN Architecture: The development of a framework integrating LLM-based automaton generation, semantic multi-source aggregation, and adaptive dual-volatility gating.
3. Empirical Validation: Demonstration of significant improvements in sample efficiency across diverse domains (resource management and navigation) while maintaining robustness to poorly aligned sources.

Experimental Results

The framework was evaluated on two domains: Dungeon Quest (sequential navigation) and Blind Craftsman (resource management with inventory constraints). Source tasks included missions with different goals (e.g., rescue vs. treasure hunt; mining vs. farming).

• Sample Efficiency: LANTERN achieved 40–60% improvements in sample efficiency compared to baselines, including single-source transfer methods (Automaton Distillation, CADENT, LARM) and a "No Transfer" baseline.
• Robustness: The system remained robust even when source tasks had distinct structures and only partial semantic overlap with the target.
• Ablation Studies:
  • Multi-source vs. Single-source: Aggregating partial knowledge from multiple sources yielded a 26% improvement over using only the most similar single source.
  • Dual-volatility vs. Experience-only: Incorporating semantic gating prevented negative transfer in misaligned regions, yielding an 18% improvement.
  • Strategic + Tactical vs. Strategic-only: Combining both guidance types provided a 31% improvement, highlighting the synergy between coarse task decomposition and fine action guidance.

Significance and Claims

The paper claims that LANTERN demonstrates that multi-source, adaptively weighted neurosymbolic transfer can significantly improve scalability and robustness in symbolic RL settings. By moving away from manual specification and single-source reliance, the framework enables agents to leverage complementary knowledge from heterogeneous tasks.

The authors emphasize that the dual-volatility gating mechanism is crucial for preventing negative transfer, allowing the system to "gracefully degrade" when source alignment is poor. The results suggest that aligning automaton states in a shared semantic embedding space provides an effective mechanism for transferring structured knowledge across domains with differing resources, layouts, and task semantics.

Limitations noted by the authors:

• Dependence on LLM quality for DFA generation.
• Scalability constraints due to the use of tabular Q-learning.
• Requirements for action-space alignment.

Future work directions identified include automaton refinement with feedback, deep function approximation for continuous domains, learned action mappings, and continual multi-source transfer.