🤖 machine learning

Exploration-Driven Personalized Federated Reinforcement Learning via Intrinsic Motivation

This paper proposes EDPFRL-IM, a novel personalized federated reinforcement learning framework that enhances exploration in sparse-reward environments by integrating intrinsic motivation (RND) at clients and exchanging only minimal novelty summaries with the server, thereby improving policy personalization and sample efficiency while preserving data privacy.

Original authors: Md Rafid Islam, Rafsan Jany, Zahid Hasan, Ratun Rahman

Published 2026-08-12

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Md Rafid Islam, Rafsan Jany, Zahid Hasan, Ratun Rahman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where learning isn't just about sitting in a classroom and listening to a teacher, but about a massive, decentralized game where millions of players are trying to solve puzzles at the same time. This is the realm of Reinforcement Learning (RL), a type of artificial intelligence where computer agents learn by trial and error, trying to get the highest score possible by interacting with their environment. Think of it like a video game character that learns to jump over pits not because someone told them to, but because they tried it, fell, and realized, "Ouch, don't do that again."

Now, imagine these players can't share their game screens or their save files with each other because of strict privacy rules. They have to learn on their own devices. This is Federated Learning, a method that lets many computers train a shared model without ever swapping their private data. When we add "Personalization" to the mix, we get Personalized Federated Reinforcement Learning (PFRL). This is like having a study group where everyone learns from the group's general wisdom, but each student also tailors the lessons to their own specific needs and quirks. The big challenge in this field is exploration: how do you teach an agent to try new, risky things when the rewards (like points or prizes) are rare, delayed, or hard to find? If the agent is too cautious, it never discovers the best moves.

The Curious Travelers: A New Way to Explore Together

Meet the EDPFRL-IM framework, a clever new system proposed by Md Rafid Islam and his team. Think of this system as a group of curious explorers scattered across a vast, foggy archipelago. Each explorer (or "client") is stuck on their own island, trying to find the treasure hidden in the landscape. The islands are different—some have heavy gravity, some have slippery ground, and the treasure is buried in different spots for each person. The catch? They can't show each other their maps or their raw experiences because of privacy rules. They can only send tiny, encrypted postcards.

In the past, these explorers tried to learn by just following the few clues they found (rewards) or by randomly wandering around. But when the treasure was hard to find (sparse rewards) or took a long time to appear (delayed rewards), they often got stuck or gave up. They were like hikers in a dense forest who only move when they see a signpost; if there are no signs, they just stand still.

The Big Idea: Curiosity as a Compass
The authors realized that to find the treasure, the explorers needed a built-in sense of curiosity. They introduced a concept called Intrinsic Motivation, specifically using a tool called Random Network Distillation (RND). Imagine giving every explorer a "novelty detector." This detector is a pair of magic glasses: one lens is a fixed, random pattern, and the other is a trainable lens that tries to guess the pattern. When the explorer sees something new and strange, the two lenses don't match, creating a "prediction error." This error acts like a spark of excitement—a bonus reward just for seeing something new.

So, even if the real treasure (the extrinsic reward) is far away, the explorers get a little "curiosity bonus" for visiting new, unexplored parts of their island. This keeps them moving and investigating, even when the path is dark.

The Secret Postcard System
Here is where the magic of the paper really shines. If every explorer just followed their own curiosity, they might all end up wandering the same boring patch of forest, or worse, they might miss the best spots entirely because they are too isolated. The authors created a way for them to coordinate without breaking privacy.

Every so often, each explorer sends a tiny, compressed "summary" to a central server. This summary isn't a map or a photo; it's just a list of "cool, new things I saw" (like a hash of a unique state or a count of visits). The server gathers these tiny summaries from all 10 explorers (in their simulation) and creates a Global Novelty Prior. Think of this as a giant, shared "Hotspot Map" that highlights which areas of the archipelago are currently under-explored by the group as a whole.

The server then sends this map back to the explorers. Now, when an explorer is deciding where to go next, they don't just follow their own curiosity; they also check the Global Map. If the map says, "Hey, the northern cliffs are very empty right now," the explorer is more likely to head there. This is coordinated exploration: everyone stays private, but they collectively ensure that no interesting corner of the world is left unvisited.

What They Found
The team tested this idea in two classic video game-like environments: MountainCar-v0 (where a car has to drive up a hill but doesn't have enough power to go straight up) and CartPole-sparse (where you have to balance a pole, but you only get points if you keep it balanced for a long time). These are tricky games where rewards are rare and hard to get.

In their simulations, they set up 10 clients (explorers) with slightly different versions of these games—some had heavier gravity, some had different friction levels. They ran the training for 100 communication rounds, with each client doing 10 local updates per round.

The results were promising. The EDPFRL-IM system consistently outperformed other methods, including standard Federated Learning (where everyone just averages their knowledge) and even methods that used curiosity but didn't coordinate (FedRL+RND).

In the MountainCar game, the new method reached an average return of 0.76, while the next best method (FedRL+RND) only hit 0.48.
In the CartPole game, EDPFRL-IM scored 0.74, beating the 0.52 of the runner-up.

The paper suggests that this system is particularly good at helping "cold-start" clients—new explorers who join the group late. Because they can immediately tap into the "Global Novelty Prior" created by the others, they adapt much faster than if they had to start from scratch.

What It's Not
It's important to note what this paper doesn't claim. The authors explicitly argue against the idea that standard exploration methods (like just picking random moves or using simple entropy rules) are enough for these difficult, sparse-reward environments. They also show that simply adding curiosity (RND) without the coordination step isn't the full solution; the "Global Novelty Prior" is the secret sauce that makes the difference.

Furthermore, these results are based on simulations in controlled environments. The paper does not claim this has been tested on real-world robots or in live medical settings yet, though the authors mention health monitoring and robotics as potential future applications. The numbers provided (like the 0.1 weight for the intrinsic reward and 0.5 for the novelty bias) are specific to their experimental setup.

The Takeaway
In simple terms, this paper shows that if you want a group of AI agents to learn effectively without sharing their private data, you shouldn't just let them wander alone. Instead, give them a sense of curiosity to keep them moving, and let them share tiny, anonymous hints about where the "unknown" places are. By combining individual curiosity with a shared sense of direction, the group can explore more efficiently, learn faster, and find better solutions than if they were working in isolation. It's a bit like a group of friends solving a mystery: if they all share their "clues" without revealing their secrets, they can solve the case much faster than any one of them could alone.

Technical Summary: Exploration-Driven Personalized Federated Reinforcement Learning via Intrinsic Motivation (EDPFRL-IM)

Problem Statement

Personalized Federated Reinforcement Learning (PFRL) aims to train decentralized agents that learn optimal policies for specific client environments while preserving data privacy. However, existing PFRL frameworks often rely heavily on exploiting extrinsic reward signals, leading to suboptimal performance in environments characterized by sparse or delayed rewards. Standard exploration strategies (e.g., $\epsilon$ -greedy, entropy regularization) frequently fail in federated settings due to the lack of coordination among independent clients and strict privacy constraints. Consequently, clients may struggle to discover optimal policies, particularly in "cold-start" scenarios or non-stationary conditions where local data is insufficient for effective exploration.

Methodology: EDPFRL-IM

The authors propose EDPFRL-IM, a framework that integrates intrinsic motivation into personalized federated learning to drive coordinated, privacy-preserving exploration. The methodology consists of three core components:

1. Local Exploration via Intrinsic Motivation

To address sparse rewards, each client employs Random Network Distillation (RND).

Mechanism: Each client maintains a fixed random target network ( $f_{tgt}$ ) and a trainable predictor network ( $f_{pred}$ ).
Intrinsic Reward: The intrinsic reward ( $r^{int}_t$ ) is calculated as the prediction error between the target and predictor networks for the next state ( $s_{t+1}$ ):
$r^{int}_t = \|f_{tgt}(s_{t+1}) - f_{pred}(s_{t+1})\|^2_2$
Total Reward: The client optimizes a combined reward signal:
$r^{total}_t = r^{ext}_t + \alpha_i \cdot r^{int}_t$
where $\alpha_i$ balances exploration and exploitation locally. This encourages clients to visit novel states without requiring external reward shaping.

2. Federated Coordination via Exploration Summaries

To prevent redundant exploration and leverage collective knowledge without compromising privacy, the server and clients exchange only compressed statistics rather than raw data or gradients.

Client Side: Clients generate low-dimensional exploration summaries ( $E_i$ ), such as visitation counts, frequency histograms of state embeddings, or top- $k$ novel state hashes derived from intrinsic rewards.
Server Side: The server aggregates these summaries to form a Global Novelty Prior ( $P_{novel}$ ):
$P_{novel}(s) = \frac{1}{1 + v_{global}[\text{cluster}(s)]}$
where $v_{global}$ represents the aggregated visitation counts across all clients. This prior identifies underexplored states globally.

3. Biased Experience Sampling

Clients incorporate the global novelty prior into their policy updates to guide local exploration toward globally novel states.

Sampling Strategy: Clients adjust their experience sampling probability ( $p_i(s)$ ) from their local buffer ( $D_i$ ) based on the global prior:
$p_i(s) \propto (1 + \beta \cdot P_{novel}(s)) \cdot p^{uniform}_i(s)$
where $\beta$ controls the influence of the global prior. This allows clients to coordinate exploration efforts while keeping raw experiences and local policies private.

Key Contributions

Novel Framework: Introduction of EDPFRL-IM, a decentralized PFRL framework that integrates intrinsic curiosity-driven motivation to facilitate personalized exploration.
Privacy-Preserving Protocol: Development of a communication-efficient mechanism that compresses exploration statistics, enabling clients to align their curiosity-driven behaviors without disclosing raw data or local gradients.
Empirical Validation: Demonstration of significant improvements in policy personalization, sample efficiency, and exploration coverage in sparse-reward benchmark environments compared to existing federated and personalized baselines.

Experimental Results

The framework was evaluated in two standard environments with heterogeneous dynamics and sparse rewards: MountainCar-v0 and CartPole-sparse. The setup involved $N=10$ clients with varying environmental parameters (e.g., gravity, friction, reward shaping) to simulate non-IID conditions.

Performance: EDPFRL-IM consistently outperformed baselines including Local RL, FedRL, and FedRL+RND. In MountainCar-v0, it achieved an average return of 0.76 compared to 0.48 for FedRL+RND. In CartPole-sparse, it reached 0.74 versus 0.52.
Personalization: As shown in Table I, EDPFRL-IM provided higher returns for every individual client compared to the FedRL baseline, demonstrating effective adaptation to local heterogeneity.
Ablation Studies:
- Removing the global novelty prior ( $\beta=0$ ) resulted in lower performance (0.63 in MountainCar), confirming the value of coordinated exploration.
- Removing intrinsic motivation entirely (FedRL) led to the lowest performance (0.40), highlighting the necessity of intrinsic drives in sparse-reward settings.
Coverage and Robustness: EDPFRL-IM achieved broader state-space coverage across clients. Furthermore, it demonstrated superior adaptability for "cold-start" clients introduced mid-training, leveraging the global prior to accelerate learning compared to non-coordinated methods.

Significance and Claims

The paper claims that EDPFRL-IM addresses a critical gap in PFRL by enabling flexible, exploratory learning structures that are both privacy-preserving and communication-efficient. By shifting from purely reward-driven exploitation to curiosity-driven exploration coordinated via global novelty priors, the framework allows clients to learn optimal policies in environments where standard methods fail due to sparse rewards. The authors assert that this approach yields significant gains in sample efficiency and personalization without compromising client privacy, making it suitable for real-world applications such as health monitoring and robotics where data heterogeneity and privacy are paramount.