Explaining Reinforcement Learning Decisions in Self-adaptive Systems
This paper introduces EARL, a Python library that generates intuitive counterfactual explanations for reinforcement learning decisions in self-adaptive systems, demonstrated through its application to a CitiBike simulation to enhance transparency and user trust.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a city where the traffic lights, the power grid, or the distribution of rental bikes adjust themselves automatically, reacting to the flow of people and weather without human intervention. These are self-adaptive systems, and they increasingly rely on a type of artificial intelligence called reinforcement learning to make their decisions. In this process, a computer program, or agent, learns by trial and error, interacting with its environment to discover the best way to achieve a goal. Over time, it builds a set of rules, or a policy, that tells it exactly what to do in any given situation. However, when these agents use complex neural networks—digital structures modeled after the human brain—their decision-making process becomes a black box. We see the input and the final action, but the path between them is opaque. For the people who rely on these systems, from city planners to everyday users, this lack of transparency is a problem. If a system makes a strange choice, such as moving a bike to a station that already has plenty, no one can easily understand why. Without that understanding, trust erodes, and it becomes difficult to verify that the system is working correctly or safely.
To solve this, researchers Jasmina Gajcin, Juan C. Rosero, and Ivana Dusparic from Trinity College Dublin have developed a new tool called EARL. This is a software library designed to peel back the curtain on these black-box decisions by generating what are known as counterfactual explanations. Instead of simply stating what the agent did, a counterfactual explanation asks "what if?" It shows the user how the outcome would have changed if the situation had been slightly different. For instance, if a bike-sharing agent decides to move a bike from Station A to Station B, the tool can show that if Station A had been slightly emptier, the agent would have chosen to leave the bike there instead. This approach mirrors how humans naturally reason about cause and effect, making the machine's logic much more intuitive. While similar methods have been used in other fields of artificial intelligence, applying them to reinforcement learning in real-world, adaptive systems has been difficult because existing tools were often limited to simple, toy examples.
The team built EARL to bridge this gap, creating a flexible framework that works with complex, realistic environments. They tested their system on a simulation of the CitiBikes network in New York City, a classic example of a self-adaptive system where bikes must be constantly repositioned to match demand. In their simulation, the agent learned to move bikes between five different stations to prevent shortages and overflows. Once the agent was trained, the researchers used EARL to interrogate its decisions. They selected specific moments from the simulation and asked the system to generate alternative scenarios. The tool successfully produced explanations for four different methods of finding these "what-if" situations. Two of the methods relied on searching through the agent's own history of actions and the environment's rules, while another used a generative model trained on a dataset of past transitions to create new, hypothetical states.
The results of this experiment revealed a clear trade-off between speed and realism. The method that relied on a pre-trained generative model could generate an explanation in less than a millisecond, but it only produced plausible instances in 82% of cases, meaning some generated scenarios violated physical constraints like station capacity. In contrast, the methods that searched through the agent's actual execution path took significantly longer—ranging from about thirty-five to nearly two hundred seconds per explanation—but they always produced realistic, plausible scenarios that respected the rules of the bike-sharing world. The researchers found that all four methods were highly successful at generating an explanation for almost every decision they tested, with a success rate of ninety-nine percent. However, the team noted that the slower, rule-based methods were far better at ensuring the explanations made sense in the real world, whereas the faster, dataset-based method required careful filtering to remove impossible outcomes.
This work demonstrates that it is possible to make complex, self-adaptive systems more transparent without sacrificing their ability to function. The EARL library provides a unified way to apply these explanation techniques to different types of agents and environments, moving the field beyond simple benchmarks and into practical applications. The researchers conclude that while the tools are ready for use, there is still work to be done. Future improvements need to focus on making the fast, dataset-based methods more reliable so they do not generate impossible scenarios, and on finding ways to speed up the more realistic, rule-based methods. Ultimately, the goal is to give users the confidence to trust these autonomous systems by showing them not just what the machine decided, but why it made that choice and what would have happened if the world had been just a little bit different.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.