Technical Summary: PlatoLTL

Problem Statement

A central challenge in multi-task reinforcement learning (RL) is training generalist policies capable of performing tasks unseen during training. While Linear Temporal Logic (LTL) has emerged as a powerful formalism for specifying structured, temporally extended tasks, existing LTL-guided RL approaches face a critical limitation: they cannot generalize to unseen vocabularies of propositions (or "symbols"). Current methods require composing all valid LTL specifications from a fixed, discrete vocabulary of atomic propositions known a priori. This restricts their applicability to tasks where propositions are parameterized by continuous variables (e.g., picking up an object at a specific coordinate $(x, y)$ or twisting a valve by an angle $\theta$). Existing solutions either discretize the parameter space, which scales poorly with resolution, or append parameters to observations, which scales poorly in multi-task settings and fails to learn shared structure across related propositions.

Methodology: PlatoLTL

The authors propose PlatoLTL (Parameterized logic architecture for temporal objectives in LTL), a novel approach that enables policies to zero-shot generalize both compositionally across LTL structures and parametrically across propositions.

Core Concept: Parameterized Predicates

Instead of treating atomic propositions as distinct, discrete tokens, PlatoLTL models them as instances of atomic predicates with specific parameter values. A proposition $p$ is defined as $p = f(x_f)$, where $f$ is an atomic predicate (a Boolean-valued function representing a truth statement about the environment) and $x_f$ are fixed parameters. This allows the policy to learn shared structure across related propositions by conditioning on the predicate type and its parameters.

Architecture

The approach utilizes a goal-conditioned RL framework where the policy $\pi(a | s, \phi)$ is conditioned on the LTL specification $\phi$. The architecture consists of three main components:

1. Predicate Instance Embedding:

   • An atomic proposition $p = f(x_f)$ is decomposed into its predicate token $f$ and parameters $x_f$.
   • The predicate $f$ is mapped to a learnable embedding $\phi_{prd}(f)$.
   • The parameters $x_f$ are passed through a predicate-specific parameter embedding network $\rho_f(\cdot)$.
   • These embeddings are concatenated and fused via a predicate-specific network $f_f(\cdot)$ to produce the proposition embedding $\phi(p)$. This acts as a learned transformation of the base predicate embedding modulated by the parameters.

2. Sequence Composition:

   • LTL specifications are converted into Limit-Deterministic Büchi Automata (LDBAs).
   • From the LDBA, a reach-avoid sequence of Boolean formulae is extracted, representing a path to an accepting cycle.
   • Each Boolean formula in the sequence is represented as an Abstract Syntax Tree (AST).
   • A Graph Convolutional Network (GCN) processes these ASTs, using the proposition embeddings for leaf nodes and learnable embeddings for logical operators, to produce formula embeddings.
   • A Gated Recurrent Unit (GRU) processes the sequence of formula embeddings (back-to-front) to generate a single task embedding.

3. Policy Execution:

   • The policy network (Actor-Critic) takes the concatenated state embedding (from environment observations) and the task embedding.
   • To handle LDBA $\epsilon$-transitions (unobserved jumps), the action space is augmented with a special $\epsilon$-action. The policy learns to predict when to trigger these transitions.
   • At evaluation, if the LDBA state changes, a new reach-avoid sequence is extracted, and the policy is re-conditioned on this new sequence.

Training

PlatoLTL is trained end-to-end using Proximal Policy Optimization (PPO) with a curriculum of increasingly challenging reach-avoid sequences. Crucially, the training sequences are sampled from a distribution of arbitrary sequences rather than being tied to specific LTL specifications, enabling the policy to learn the underlying structure of predicate instances and generalize to unseen specifications.

Key Contributions

1. Parametric Modeling: The proposal to model LTL propositions as parameterized instances of atomic predicates, enabling generalization across continuous parameter spaces.
2. Novel Architecture: A neural network architecture that meaningfully maps predicate parameters to embeddings and composes them to represent LTL formulae via ASTs and sequence modeling.
3. Zero-Shot Generalization: The demonstration that PlatoLTL achieves zero-shot generalization both compositionally (across LTL structures) and parametrically (across unseen propositions).
4. New Benchmarks: The introduction of four novel predicate-based benchmark environments (RGBZoneEnv, XYZEnv, XYXYEnv, FalloutWorld) that support complex, parameterized LTL specifications, extending existing literature.
5. Empirical Validation: Comprehensive experiments showing PlatoLTL's superior performance and convergence speed compared to state-of-the-art baselines (DeepLTL, LTL-GNN) and parameterization-based methods (Naïve, SIAMS).

Results

Experiments were conducted on both discrete sets of propositions (training and unseen evaluation sets) and continuous sets of propositions.

• Discrete Generalization: On unseen evaluation propositions, PlatoLTL maintained performance parity with training propositions (e.g., ~99% success rate in RGBZoneEnv), whereas baselines (DeepLTL, LTL-GNN) failed to generalize, showing significantly lower success rates.
• Continuous Generalization: In environments with continuous propositions (XYZEnv, XYXYEnv), baselines achieved near-zero success rates despite having access to parameters. PlatoLTL achieved high success rates (e.g., >95% in XYZEnv) and low episode lengths.
• Convergence: PlatoLTL converged significantly faster than baselines. As the number of training propositions increased, the convergence rate of baselines decreased steadily (requiring impractical training times for large vocabularies), while PlatoLTL remained robust and unaffected.
• Embedding Structure: Principal Component Analysis (PCA) revealed that PlatoLTL learned a low-dimensional, structured embedding space where related propositions (e.g., different RGB colors) formed coherent geometries (e.g., a cube), whereas baselines produced scattered, high-dimensional embeddings resembling noise.

Significance and Claims

The paper claims that PlatoLTL addresses a fundamental limitation in neuro-symbolic RL: the inability to generalize to unseen vocabularies of propositions. By treating propositions as parameterized instances of predicates, the approach learns grounded associations between task parameters and the environment.

The authors position PlatoLTL as a step toward creating agents capable of performing arbitrary complex, long-horizon tasks without requiring a fixed, discrete vocabulary. They note that while the approach relies on predicates being known a priori (a limitation shared with prior work, though strictly more general), it significantly improves scalability and generalization in parameterized settings. The work suggests that combining this backbone with methods for exploration efficiency and robustness could enable scaling to high-dimensional, real-world robotic tasks. The authors modestly state that future work should investigate these combinations and the automatic discovery of grounded predicates.