🤖 AI

Grounding LTL Tasks in Sub-Symbolic RL Environments for Zero-Shot Generalization

This paper proposes a method that jointly trains a multi-task reinforcement learning policy and a symbol grounder using Neural Reward Machines to enable agents to follow Linear Temporal Logic instructions in sub-symbolic environments without prior knowledge of the symbol-to-observation mapping, achieving zero-shot generalization that outperforms existing approaches.

Original authors: Matteo Pannacci, Andrea Fanti, Elena Umili, Roberto Capobianco

Published 2026-08-18

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Matteo Pannacci, Andrea Fanti, Elena Umili, Roberto Capobianco

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot entering a room it has never seen before. It sees a coffee machine, a desk, and a mail slot, but to the robot, these are just shapes and colors on a screen. It does not know that the machine is for coffee or that the slot is for mail. Now, imagine a human gives the robot a complex instruction: "Go to the mail slot, then to the coffee machine, and finally to the desk, but do not step on the red floor tiles along the way." For a human, this is easy because we understand the meaning of the words and the objects. For a robot, this is a massive puzzle. It must first figure out which object is which, and then it must remember the order of events while avoiding the red tiles, all while learning how to move. This is the challenge of teaching artificial intelligence to follow instructions in the real world, where it cannot be handed a pre-made dictionary of what things are called.

For decades, researchers have tried to solve this by giving robots a perfect map of their world, telling them exactly which pixel corresponds to "coffee" or "mail." This works in controlled labs but fails in the messy, unpredictable real world. A new approach, detailed in recent research, attempts to teach the robot to learn these meanings on its own while it learns to move. The scientists focused on a specific type of instruction that involves time and sequence, known as temporal logic. Think of this as a way to write rules that say "do A, then B, but never C," rather than just "go to point X." The goal was to see if a robot could learn to follow these complex, time-based rules in a world where it starts with no idea what the objects are, using only its camera feed and a simple signal when it succeeds or fails.

The researchers built a system where the robot learns two things at the same time. First, it learns a policy, which is simply the strategy for deciding what action to take next. Second, and more importantly, it learns a "grounder." This is a component that acts like a translator, trying to figure out what the raw images from the camera mean in terms of the instruction. If the robot sees a red square and the instruction says "avoid red," the grounder must eventually learn to label that red square as "lava" or "danger." The difficulty is that the robot does not get a teacher to correct it. It only gets a sparse reward: a tiny signal when it finishes the task correctly, a negative signal if it fails, and nothing in between. This is like trying to learn a new language by only being told "yes" or "no" at the very end of a conversation, with no feedback on the individual words spoken in the middle.

To solve this, the team used a clever trick involving the structure of the instructions themselves. They treated the instruction as a machine that changes state as the robot moves. If the robot picks up the mail, the machine moves to a new state that says "mail is done, now find coffee." The researchers trained the robot's translator to predict the rewards that this machine would give. By watching the robot move and seeing when the machine says "success" or "failure," the translator slowly learns to match the images it sees with the correct words in the instruction. It is a process of trial and error, where the robot uses the logic of the task to teach itself what the symbols mean. The researchers tested this in two very different worlds. One was a grid-based environment that looked like a simplified video game, where the robot had to navigate a maze of items. The other was a smooth, continuous space where the robot moved like a car on a flat plane, trying to visit colored zones in a specific order.

The results showed that this method works remarkably well. In the grid world, the robot learned to follow the instructions almost as perfectly as if it had been given the correct dictionary from the start. It learned to identify the items and follow the sequence of events, even when the instructions became longer and more complex. In the continuous world, the robot also learned to identify the zones with high accuracy, though it struggled slightly more with the most complex avoidance tasks. Crucially, the robot learned these meanings while it was learning to move, without any separate training phase. The system proved that an agent can learn to ground symbols—connect raw images to abstract concepts—just by trying to solve the task and using the structure of the rules to guide its learning.

The study also compared this new method against a previous attempt to solve the same problem. The older method, which tried to build the robot's brain piece by piece based on the grammar of the instructions, failed when the instructions got longer. It could not handle the complexity of the new tasks. The new approach, however, scaled up effectively, showing that learning the meaning of objects and the strategy for moving are deeply connected and should be learned together. The researchers found that the robot's ability to identify objects improved quickly, often reaching a high level of accuracy within a few million steps of training. This suggests that the indirect feedback from the task structure is enough to teach the robot what things are, even without a human pointing them out.

While the system is not perfect, particularly in the most difficult continuous environments where avoiding danger for the entire duration of a task proved hard for all methods, the core finding is clear. It is possible to train an artificial agent to understand complex, time-based instructions in a world it does not know, by letting it learn the meaning of the world's features as it tries to complete the job. This moves us closer to creating machines that can enter a new environment, listen to a set of rules, and figure out what to do without needing a pre-programmed map of every object they might encounter. The work demonstrates that by combining the logic of the task with the learning of the environment, robots can begin to bridge the gap between raw data and meaningful action.

Technical Summary: Grounding LTL Tasks in Sub-Symbolic RL Environments for Zero-Shot Generalization

Problem Statement

This work addresses the challenge of training Reinforcement Learning (RL) agents to follow multiple temporally extended instructions expressed in Linear Temporal Logic (LTL) within sub-symbolic environments. In such settings, agents receive raw sensory observations (e.g., images) rather than symbolic state descriptions. A critical bottleneck in previous multi-task RL approaches is the reliance on a known "labeling function" (symbol grounding)—a mapping that translates raw observations into the atomic propositions required by LTL formulae.

The authors argue that assuming access to this true labeling function is unrealistic for real-world domains. Consequently, the problem is defined as learning a multi-task policy and a symbol grounder simultaneously, without prior knowledge of the mapping between observations and symbols. The goal is to achieve zero-shot generalization to unseen LTL instructions in these sub-symbolic environments.

Methodology

The proposed method jointly learns a multi-task policy and a symbol grounder using the same experience, leveraging the structure of LTL formulae to provide indirect supervision. The architecture consists of four neural network modules:

Grounder Module ( $L_\theta$ ): A classifier that maps raw observations to a probability distribution over the set of atomic propositions ( $\Delta(P)$ ).
Environment Feature Module: Encodes raw observations into feature vectors.
LTL Module: Encodes the current state of the task (the progressed LTL formula) into a feature vector. This module utilizes a Relational Graph Convolutional Network (R-GCN) based on the formula's Abstract Syntax Tree (AST), pre-trained via an "LTL-Bootcamp" sequence generation task to stabilize learning.
RL Module: A policy and value function that takes concatenated features from the environment and LTL modules to select actions.

Neural Reward Machines (NRMs) for Semi-Supervised Grounding

The core innovation lies in training the grounder using Neural Reward Machines (NRMs). Since explicit labels are unavailable, the system treats the problem as a Semi-Supervised Symbol Grounding (SSSG) task.

Mechanism: For each training task (LTL formula), a corresponding Moore machine is derived. This machine tracks task progress and outputs rewards (+1 for satisfaction, -1 for violation, 0 otherwise).
Training: The NRMs are implemented as differentiable probabilistic automata. The grounder is trained by backpropagating gradients through the NRM structure. The objective is to minimize the cross-entropy between the rewards predicted by the NRM (based on the grounder's probabilistic symbol predictions) and the ground-truth rewards derived from the task's termination conditions.
Multi-Task Advantage: Unlike previous single-task applications of NRMs, this method employs multiple NRMs sharing a single grounder. This multi-task setup provides a denser learning signal, mitigating the "reasoning shortcuts" problem common in sparse reward settings and allowing the grounder to converge effectively without dense, potential-based rewards.

Policy Learning

The policy is conditioned on the task progression using the LTL2Action framework (Vaezipoor et al., 2021). The LTL formula is progressively updated based on the grounder's predicted symbols, and the resulting progressed formula is encoded to guide the policy. This allows the agent to track task requirements directly from raw observations.

Key Contributions

Neurosymbolic Architecture for Joint Learning: The authors propose a novel architecture that jointly learns a multi-task policy and a symbol grounder. This approach eliminates the need for a known labeling function, relying instead on the symbolic specification of the task (LTL formula) and environment trajectories.
Empirical Validation of Zero-Shot Generalization: The paper provides an empirical evaluation demonstrating that the proposed method achieves performance comparable to the "upper bound" (where the true labeling function is known) and significantly outperforms the only existing baseline for multi-task RL without symbol grounding knowledge (Kuo et al., 2020).

Experimental Results

The method was evaluated on two vision-based environments: a discrete Minecraft-like grid-world and a continuous FlatWorld navigation environment. Tasks included "Partially-Ordered" sequences and "Global Avoidance" constraints.

Performance vs. Upper Bound: In the Minecraft-like environment, the proposed method achieved success rates and discounted returns nearly identical to the upper bound (LTL2Action with true grounding) for both training and zero-shot transfer to complex, unseen formulae.
Performance vs. Baseline: The method significantly outperformed the Kuo et al. (2020) baseline. The baseline struggled with longer sequences and failed to scale to complex tasks, whereas the proposed method maintained high performance.
Grounder Convergence: The symbol grounder consistently converged within 1 million frames. It showed faster convergence on "Global Avoidance" tasks, where negative rewards from task violations provided a strong learning signal.
Continuous Environment: In FlatWorld, the grounder achieved over 95% accuracy despite the difficulty of determining proximity to zones. While all methods struggled with the most complex "Global Avoidance" tasks in this continuous setting, the proposed method still significantly outperformed the baseline on "Partially-Ordered" tasks and maintained performance close to the upper bound.

Significance and Claims

The paper claims that by removing the unrealistic assumption of a known symbol grounding, this approach extends multi-task RL to truly sub-symbolic environments. The authors assert that their method represents a significant step toward agents capable of following diverse, previously unseen instructions directly from raw observations.

The work highlights that while Neural Reward Machines have historically struggled with sparse rewards in single-task settings, training on multiple tasks provides sufficient indirect supervision to correctly infer symbol grounding. The authors modestly note that future work could focus on improving grounder robustness against noisy observations, incorporating reward shaping based on task progression, and utilizing curriculum learning for increasingly complex formulae. They also suggest that future iterations might avoid explicit automata construction by leveraging fuzzy logic conformance checking.