🤖 AI

Explaining Reinforcement Learning Agents via Inductive Logic Programming

This paper advances Explainable Reinforcement Learning by employing Inductive Logic Programming to extract symbolic policy representations and introducing a novel set of objective metrics to systematically quantify and analyze the explainability, learning dynamics, and coordination patterns of RL agents across single and multi-agent domains.

Original authors: Celeste Veronese, Edoardo Zorzi, Daniele Meli, Alessandro Farinelli

Published 2026-07-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Celeste Veronese, Edoardo Zorzi, Daniele Meli, Alessandro Farinelli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've built a super-smart robot that learns to play video games or drive a car just by trying things out, making mistakes, and getting points for good moves. This is called Reinforcement Learning. It's like a puppy learning tricks: it doesn't get a manual; it just figures out that pressing a button makes a treat appear, so it presses the button again. The problem is, these robots are often "black boxes." They make brilliant decisions, but even their creators can't always explain why they chose to turn left instead of right. They just know the robot works, but not the logic behind it. This is dangerous if the robot is driving a real car or helping a doctor, because we need to trust its reasoning.

To fix this, scientists are trying to translate the robot's secret brain into plain language, like a set of simple "If-Then" rules. This is where Explainable AI comes in. Think of it as asking the robot to write a diary of its thoughts. But here's the catch: just because the robot wrote a diary doesn't mean the diary is accurate, complete, or useful. How do we know if the robot's explanation is actually telling the truth? Until now, we mostly asked humans, "Does this sound reasonable?" which is subjective and varies from person to person. We needed a way to measure the quality of these explanations with a ruler, not just a feeling.

This paper introduces a new set of "rulers" to measure how well we can explain a Reinforcement Learning robot's brain. The authors, a team of researchers, used a clever trick called Inductive Logic Programming (think of it as a super-organized detective that turns messy behavior into neat, logical rules). They trained robots in different scenarios—like a self-driving car at a busy intersection or teams of robots working together in a warehouse—and then used their detective to write down the rules the robots were following.

The big discovery is that they created four specific ways to check if these written rules are any good. First, they measure the Activation Rate, which is like checking how often the robot actually follows the rules it wrote down. If the robot says, "I slow down when a car is close," but it speeds up half the time, the rule has a low activation rate and isn't a good explanation. Second, they look at Feature Coverage, which tells us which parts of the environment the robot actually cares about. Maybe the robot claims to watch the traffic lights, but the rules show it's actually only looking at the distance to the car in front.

They also invented two ways to compare rules: Syntactic Distance and Semantic Distance. Imagine two robots learning the same task. Syntactic distance checks if they used the same words to describe the rules (like both saying "slow down"). Semantic distance checks if they actually did the same thing, even if they used different words. This is crucial for teams of robots; it helps us see if they are truly working together or just pretending to.

The researchers tested these rulers on robots playing games and driving in traffic. They found that these new measurements revealed things that the standard "score" of the game missed. For example, a robot might get a high score, but the rulers showed that its logic was shaky and only worked in very specific situations, not generally. In team games, the rulers showed how robots developed special roles, with some becoming experts at one job and others at another, even if they were all trying to win the same way.

In short, this paper doesn't just give us a way to ask robots to explain themselves; it gives us a way to grade those explanations. It suggests that by using these objective measurements, we can spot when a robot is confused, when it's learned a bad habit, or when it's ready to be trusted with more complex jobs. It turns the vague idea of "is this robot explainable?" into a concrete, measurable fact, helping us build safer and more reliable AI for the real world.

Technical Summary: Explaining Reinforcement Learning Agents via Inductive Logic Programming

Problem Statement
Explainable Reinforcement Learning (XRL) aims to make learned policies transparent and interpretable, a necessity for safety-critical and human-centric applications. However, current XRL approaches face two primary limitations:

Subjectivity: Most methods rely on user studies (e.g., fidelity, comprehensibility) which target specific audiences and lack shared, objective evaluation metrics.
Lack of Quantification: While logic-based approaches in Explainable AI (XAI) offer human-readable abstractions, there is no systematic way to quantify the degree of explainability of these logical representations. Existing metrics often focus on feature importance in static supervised learning or global return, failing to capture action-specific dynamics, policy evolution during training, or inter-agent relationships in Multi-Agent Reinforcement Learning (MARL).

Methodology
The authors propose a framework that combines Inductive Logic Programming (ILP) with Answer Set Programming (ASP) to extract symbolic representations of RL policies and define a novel set of objective, user-independent metrics to evaluate them.

Policy Extraction via ILP:
- The system observes execution traces (state-action pairs) from trained RL agents (e.g., DQN, MAPPO).
- These traces are formalized as Context-Dependent Partial Interpretations (CDPIs) within the ASP formalism.
- Using the ILASP system, the framework induces a hypothesis $H$ (a set of logical rules) that best explains the observed behavior, serving as a symbolic approximation of the black-box policy $\pi$ .
Novel Explainability Metrics:
The paper defines four metrics to quantify the properties of the learned symbolic policies:
- Activation Rate ( $\alpha$ ): Measures the frequency with which specific rules or actions in the symbolic policy align with the agent's actual behavior. It identifies which action-specific policies are robustly learned and highlights overfitting or instability.
- Feature Coverage ( $\phi$ ): Quantifies the proportion of the symbolic theory that utilizes specific environmental features. Unlike global feature importance, this provides a fine-grained view of feature relevance per action.
- Syntactic Distance ( $\lambda$ ): Computes the structural dissimilarity (Jaccard index of rule bodies) between two symbolic theories. It assesses whether different agents or training checkpoints rely on the same domain concepts.
- Semantic Distance ( $\delta$ ): Measures the behavioral divergence between two theories under identical conditions by comparing their generated answer sets. It captures whether policies produce consistent decisions even if their syntactic structures differ.

Key Contributions

Objective Metrics for XRL: The work introduces the first evaluation framework specifically focused on quantitatively explaining multiple aspects of RL and MARL policies using objective, user-independent criteria.
Action-Specific Analysis: The metrics move beyond global return curves to analyze the convergence, robustness, and specialization of sub-policies for individual actions.
Multi-Agent Dynamics: The framework provides tools to analyze coordination, specialization, and adaptation patterns in MARL by comparing the symbolic approximations of different agents.
Generalization Assessment: The metrics are applied to evaluate the transferability of learned symbolic rules to out-of-distribution scenarios and varying domain difficulties.

Experimental Results
The framework was evaluated across single-agent (Intersection domain) and multi-agent (RWARE cooperative, Simple Adversary contrastive) settings.

Training Dynamics: The activation rate revealed that symbolic representations often converge before the global return curve stabilizes. It also highlighted that certain actions (e.g., "load" in RWARE) are learned more reliably than others (e.g., movement), which global metrics miss.
Multi-Agent Alignment: Syntactic and semantic distances showed that while agents in cooperative tasks (RWARE) develop aligned behaviors (low semantic distance), agents in competitive tasks (Simple Adversary) maintain high divergence. Notably, even agents trained on the same task in Simple Adversary developed semantically distinct strategies (specialization) despite similar returns.
Feature Importance: Feature coverage demonstrated that standard feature importance methods can be misleading. For instance, in RWARE, the "good_shelf" feature became increasingly critical for the "load" action specifically, a nuance obscured by aggregate analysis.
Generalization: When tested in harder environments, the activation rate of specific rules (e.g., "load") remained high, indicating that the framework can identify which learned skills are robust and transferable.

Significance and Claims
The authors claim that this work advances the state of the art by offering rigorous, objective, and interpretable metrics to evaluate symbolic policy representations. The significance lies in:

Debugging and Refinement: Providing tools to understand why an agent behaves a certain way, allowing for the identification of non-stationarity, instability, or overfitting in specific actions.
Trustworthiness: Moving XRL beyond subjective user studies to principled, quantitative assessments of policy robustness and generalization.
MARL Insights: Uncovering coordination and specialization patterns in multi-agent systems that are invisible to conventional performance metrics.

The paper concludes that while the framework is a significant step forward, future work should explore temporal dependencies, integrate user-centric evaluations with these objective metrics, and apply the approach to more complex, real-world safety-critical systems.

Technical Summary: Explaining Reinforcement Learning Agents via Inductive Logic Programming

More like this