🤖 AI

Neuro-Symbolic Injection of LTLf Constraints in Autoregressive Reinforcement Learning Policies

This paper proposes a neuro-symbolic framework that integrates Linear Temporal Logic over finite traces (LTLf) constraints into transformer-based autoregressive reinforcement learning policies by compiling specifications into differentiable deterministic finite automata, thereby improving constraint satisfaction while maintaining competitive returns in offline RL tasks.

Original authors: Ashkan Ansarifard (Sapienza University of Rome), Matteo Mancanelli (Sapienza University of Rome), Elena Umili (Sapienza University of Rome), Fabio Patrizi (Sapienza University of Rome)

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ashkan Ansarifard (Sapienza University of Rome), Matteo Mancanelli (Sapienza University of Rome), Elena Umili (Sapienza University of Rome), Fabio Patrizi (Sapienza University of Rome)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching Robots to Follow Rules Without Trial-and-Error

Imagine you are trying to teach a robot to navigate a maze. In the real world, if the robot hits a wall, it learns "ouch, don't do that." But in Offline Reinforcement Learning, the robot isn't allowed to touch the real world. It can only study a giant photo album of past attempts made by someone else.

The problem? The robot in the photo album might have made mistakes. If you just tell the robot to "get the most points," it might learn to copy those mistakes because they led to high scores in the past, even if they were dangerous.

This paper introduces a new way to teach these robots: Neuro-Symbolic Injection. Think of it as giving the robot a "rulebook" written in a strict, logical language (LTLf) and forcing it to study that rulebook while it learns from the photo album.

The Core Problem: The "Reward Trap"

Standard AI models (like the ones used in this paper, called Transformers) are like brilliant students who are obsessed with getting an 'A' (maximizing reward). They look at the photo album and say, "Okay, I see that in this picture, the robot stepped on a bomb, but then it got a huge reward later. I'll do that too!"

But in safety-critical situations (like self-driving cars or medical robots), stepping on a bomb is a "Game Over." You can't just say, "Well, I got a reward later." You need to ensure the robot never steps on the bomb, even if it means taking a longer path.

The Solution: The "Traffic Cop" and the "Logic Engine"

The authors built a system that acts like a Traffic Cop standing next to the student robot while it studies.

The Rulebook (LTLf): Instead of vague instructions like "be safe," the researchers use Linear Temporal Logic. This is like a precise legal contract.
- Bad instruction: "Try not to hit bombs."
- LTLf instruction: "You must always avoid bombs, and you must eventually reach the goal."
- This covers the whole journey, not just the next step.
The Traffic Cop (The DFA): The computer translates that legal contract into a Deterministic Finite Automaton (DFA). Imagine a flowchart or a board game map.
- Every time the robot takes a step in its training, the Traffic Cop checks the flowchart.
- "Did you step on a bomb? No? Good, move to the next square on the map."
- "Did you step on a bomb? Yes? STOP. You are now in the 'Fail' zone."
The "Soft" Nudge (Differentiable Loss): Here is the magic trick. Usually, a flowchart is rigid: you are either safe or you are dead. But the robot learns by making tiny adjustments to its brain (mathematically speaking). You can't adjust a "dead" state.
- The authors made the Traffic Cop differentiable. This means the Cop doesn't just say "Fail"; it says, "You are 90% safe, but you are getting closer to the danger zone. Please adjust your brain slightly to move away from the danger."
- This creates a Logic Loss. It's like a teacher giving a student a grade not just on the final test, but on every single step of their homework, gently correcting them before they make a fatal mistake.

How They Tested It: The "ColourBomb" Game

They tested this in a grid-world game called ColourBomb.

The Goal: Reach a colored exit.
The Danger: Red "Bomb" cells that end the game instantly.
The Challenge: The robot had to learn to reach the exit without ever stepping on a bomb.

They compared two types of AI models:

Decision Transformer (DT): A model that decides the next move based on the history.
Trajectory Transformer (TT): A model that predicts the whole path at once.

The Results: From "Chaos" to "Champion"

Without the Logic Cop (The Baseline):
The robots were disastrous. They kept stepping on bombs because they were just trying to maximize points based on the messy data they were given. They failed to reach the goal safely almost 100% of the time.

With the Logic Cop (The New Method):
By adding the "Logic Loss" (the Traffic Cop's gentle nudges), the results changed dramatically:

Safety: The robots stopped hitting bombs. They achieved 100% safety in the best settings.
Success: They successfully reached the goal.
Performance: They didn't just become safe; they actually got better scores than the unsafe robots because they didn't waste time dying and restarting.

The Trade-Off: Finding the Sweet Spot

The researchers found a "Goldilocks" zone.

If the Logic Cop was too weak (too little "nudge"), the robot ignored the rules and hit bombs.
If the Logic Cop was too strong, the robot became so scared of the rules that it froze and never moved (it was safe, but it never reached the goal).
By tuning the "strength" of the cop (a parameter called $\alpha$ ), they found the perfect balance where the robot was both safe and successful.

Summary

This paper shows that you can teach an AI to follow strict, complex safety rules (like "always avoid X, but eventually do Y") even when you can't let it practice in the real world. By translating those rules into a mathematical "flowchart" and using it to gently correct the AI's learning process, they created robots that are much safer and more reliable than those trained by reward alone.

Key Takeaway: You don't need to rewrite the robot's entire brain or the data it studies. You just need to add a "logic layer" that acts as a constant, gentle guide, ensuring the robot learns the right habits from the start.

Technical Summary: Neuro-Symbolic Injection of LTLf Constraints in Autoregressive Reinforcement Learning Policies

Problem Statement
In offline Reinforcement Learning (RL), agents learn policies exclusively from pre-collected datasets without online interaction. While recent sequence-modeling approaches, such as Trajectory Transformers (TT) and Decision Transformers (DT), have successfully framed RL as a sequence generation problem, they typically optimize solely for cumulative reward. This creates a critical gap in safety-critical domains (e.g., robotics, autonomous navigation) where policies must adhere to high-level, temporally extended constraints. Standard reward-based optimization cannot guarantee that a policy satisfies complex temporal logic requirements, such as "eventually reach the goal while always avoiding hazards," especially when violations cannot be corrected via trial-and-error exploration in an offline setting.

Methodology
The authors propose a neuro-symbolic framework that injects Linear Temporal Logic over finite traces (LTL $_f$ ) constraints directly into the training process of autoregressive transformer-based policies. The core methodology involves three main components:

LTL $_f$ to DFA Compilation: High-level temporal specifications are compiled into Deterministic Finite Automata (DFAs). This translation allows logical constraints to be represented as state-transition systems that can process sequences of tokens.
Differentiable Logic Loss: To integrate these discrete automata into the continuous optimization of neural networks, the authors employ a differentiable representation.
- Soft Satisfaction Signals: Instead of hard sampling, the framework uses Gumbel-Softmax sampling to generate differentiable, near-one-hot token vectors during training.
- DeepDFA Integration: These soft tokens are fed into a "DeepDFA," a neural relaxation of the standard DFA. This module computes a probabilistic satisfaction signal indicating the likelihood that the generated trajectory satisfies the LTL $_f$ formula.
- Regularization: A logic loss function ( $L_\phi$ ) is derived from the negative log-probability of satisfaction. This is combined with the standard supervised sequence modeling loss ( $L_{\mathcal{D}}$ ) via a weighted sum: $L = \alpha L_\phi + (1 - \alpha) L_{\mathcal{D}}$ , where $\alpha$ is a trade-off parameter.
Architecture Agnosticism: The framework is designed to be model-agnostic within the class of autoregressive sequence models. It utilizes a shared "logic adapter" to handle token-to-symbol mappings and DFA rollouts, making it applicable to both TT and DT architectures without altering their underlying structures.

Key Contributions
The paper outlines three primary contributions:

Unified Neuro-Symbolic Framework: A method for injecting LTL $_f$ constraints into transformer-based offline RL policies using automata-based modules that are architecture-agnostic.
Canonical Evaluation Stack: A consistent end-of-episode evaluation pipeline that aligns dataset serialization, token-to-symbol mapping, and logical evaluation, preventing semantic mismatches between training and temporal logic assessment.
Empirical Analysis: A systematic evaluation in navigation domains demonstrating the trade-offs between task performance (return) and constraint satisfaction under varying levels of logical guidance.

Experimental Results
The framework was evaluated on the ColourBomb navigation environment, testing two specification types: Invariant Safety ( $G(\neg \text{bomb})$ ) and Reach-while-Safe ( $G(\neg \text{bomb}) \land F(\text{goal})$ ).

Decision Transformers (DT):
- Vanilla DTs (without logic regularization) failed completely, hitting bomb cells and failing to reach goals.
- Introducing logic regularization with a high trade-off parameter ( $\alpha = 0.8$ ) achieved 100% satisfaction for both safety and reach-while-safe specifications.
- Crucially, this high-constraint setting also improved the average return (from -1.460 to -0.780) by eliminating catastrophic failures (bomb hits), demonstrating that constraint satisfaction can align with performance in offline settings.
Trajectory Transformers (TT):
- Vanilla TTs showed partial safety compliance but low goal achievement.
- Logic regularization improved satisfaction rates and reduced bomb-hit rates. For the invariant safety task, $\alpha = 0.8$ yielded the best results. For the more complex reach-while-safe task, a lower weight ( $\alpha = 0.1$ ) provided the optimal balance, improving both satisfaction and return compared to the baseline.
- The authors note that while improvements were observed, they were more moderate in TTs compared to DTs, a discrepancy left for future investigation.

Significance and Claims
The paper claims that integrating symbolic background knowledge directly into the learning objective—rather than modifying the data representation or relying on reward shaping—is a promising direction for improving policy reliability in safety-critical offline RL.

Key claims regarding the method's significance include:

Direct Constraint Injection: Unlike approaches that rely on shielding (which requires explicit environment models) or reward shaping (which can be difficult to engineer), this method injects temporal logic knowledge directly into the training loss.
Generality: The framework supports arbitrary LTL $_f$ formulas, extending beyond the "safety fragment" to include liveness and complex temporal combinations.
Performance Preservation: The results suggest that enforcing strict logical constraints does not necessarily degrade task performance; in the tested scenarios, it actually improved returns by preventing catastrophic outcomes that vanilla models frequently encountered.

The authors maintain a modest tone, acknowledging that the approach is currently limited to offline settings and that the observed improvements in TTs were less pronounced than in DTs. They identify future work as extending the evaluation to additional benchmarks, model architectures, and conducting extensive end-to-end comparisons with established Safe RL frameworks.