💻 computer science

A Decentralized Multi-Metric Q-Learning Objective Function (MM-QLOF) for Efficient Routing in IoT Networks

This paper proposes MM-QLOF, a decentralized multi-metric Q-learning objective function that integrates packet success rate, expected transmissions, and congestion levels to enable autonomous, adaptive parent node selection in IoT networks, significantly improving packet delivery rates and reducing latency compared to existing RPL standards.

Original authors: Youness TALBI, Abdelhadi ELOUDRHIRI HASSANI, Adil SALBI, Issam BOUGANSSA

Published 2026-06-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Youness TALBI, Abdelhadi ELOUDRHIRI HASSANI, Adil SALBI, Issam BOUGANSSA

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a busy city where thousands of tiny, battery-powered delivery drones (the IoT devices) need to send packages to a central warehouse (the Internet). To do this efficiently, each drone needs to pick a "parent" drone to hand its package to, which then passes it up the chain until it reaches the warehouse.

The problem is that when the city gets too crowded (high traffic), the usual rules for picking a parent break down. The drones might pick a parent that looks close but is actually stuck in a traffic jam, causing packages to get lost, delayed, or wasted energy on re-sending them.

This paper proposes a new, smarter way for these drones to choose their parents. Instead of following a rigid map, they use a "learning system" called MM-QLOF. Here is how it works, broken down simply:

1. The Old Way vs. The New Way

The Old Way (OF0 and MRHOF): Imagine a delivery driver who only looks at two things: "How many stops away is the warehouse?" (OF0) or "Is the road paved and smooth?" (MRHOF). If a road is smooth but completely clogged with traffic, the driver still picks it, leading to delays.
The New Way (MM-QLOF): This system gives every drone a "smart brain" based on Reinforcement Learning. Think of this like a video game character learning by trial and error.
- The Goal: The drone wants to maximize its "score" (a reward).
- The Lesson: If a drone picks a parent and the package arrives quickly and safely, it gets a reward (points). If the package gets lost or delayed, it gets a penalty (it loses points).

2. The Three "Gauges" the Drones Watch

To calculate its score, the drone checks three specific gauges on every potential parent:

Success Rate (PSR): "How often does this parent actually catch the package I throw?" (Reliability).
Transmission Count (ETX): "How many times do I have to try to get this package to this parent?" (Efficiency).
Congestion Level: "Is this parent's mailbox full?" (Traffic).

The system combines these three. If a parent has a full mailbox (high congestion), the drone gets a heavy penalty, even if the road is smooth. This prevents the drone from picking a "smooth but jammed" route.

3. The "Smart Switch" (Hysteresis)

There is a tricky problem in these networks: sometimes a parent looks slightly better than the current one just because of a tiny, temporary fluctuation. If the drone switches parents every time it sees a tiny improvement, the whole network becomes chaotic, like a dance floor where everyone keeps changing partners every second.

To fix this, the paper introduces a Hysteresis Mechanism. Think of it as a "threshold of change."

The drone will only switch to a new parent if the new parent is significantly better (at least 10% better) than the current one.
If the new parent is only "a little bit" better, the drone stays put to keep the network stable.

4. The Results: A Less Crowded City

The researchers tested this system in a computer simulation (a virtual city) with 51 drones. They compared their new "Smart Brain" system against the standard, older systems.

Here is what they found:

Fewer Lost Packages: The new system delivered about 11% more packages successfully than the standard systems, especially when the city was very crowded.
Faster Delivery: Packages arrived 18% to 23% faster because the drones avoided the traffic jams.
Better Battery Life: Because the drones didn't have to keep re-sending lost packages, they saved a small amount of energy.

The Takeaway

The paper argues that by teaching these tiny devices to "learn" from their environment—specifically by paying attention to traffic jams (congestion) in addition to road quality—they can make much smarter decisions. This keeps the network running smoothly even when everyone is trying to send data at the same time.

The authors also noted that the "weights" of these three gauges can be adjusted. If you care most about getting the package there at all costs, you weight "Success Rate" higher. If you care most about speed, you weight "Congestion" higher. This allows the system to be tuned for different needs.

Technical Summary: A Decentralized Multi-Metric Q-Learning Objective Function (MM-QLOF) for Efficient Routing in IoT Networks

Problem Statement
The paper addresses the degradation of performance in Internet of Things (IoT) networks operating under high-traffic conditions. Specifically, in Low-Power and Lossy Networks (LLNs) utilizing the IPv6 Routing Protocol for Low-Power and Lossy Networks (RPL), high traffic loads lead to significant congestion, link quality degradation, and packet interference. These conditions make the selection of an optimal parent node (next hop) extremely difficult for standard RPL Objective Functions (OFs). Traditional approaches like the Zero Objective Function (OF0), which relies on hop count, and the Minimum Rank Objective Function with Hysteresis (MRHOF), which relies on Expected Transmission Count (ETX), often fail to account for dynamic congestion levels and real-time packet success rates. Consequently, nodes may select parents with saturated MAC queues, leading to increased collisions, retransmissions, end-to-end latency, and packet loss.

Methodology
The authors propose MM-QLOF (Multi-Metric Q-Learning Objective Function), a decentralized approach that integrates Reinforcement Learning (specifically Q-learning) into the RPL parent selection process. The methodology operates as follows:

State and Action Definition: Each IoT node acts as an autonomous agent. The "state" is the node's current need to select a preferred parent, and the "action" is the selection of a specific candidate parent from the RPL parent set.
Multi-Metric Reward Function: The core innovation lies in the reward function, which evaluates candidate parents based on three normalized metrics:
- Packet Success Rate (PSR): Measures the reliability of the immediate link (ACKs received vs. packets transmitted).
- Expected Transmission Count (ETX): Indicates the number of transmissions required for successful delivery; lower values are preferred.
- Congestion Level: A MAC-layer metric reflecting queue occupancy and packet drops relative to transmissions.
  The reward function is formulated to maximize PSR and low congestion (high "Normalized Congestion") while minimizing ETX. The formula is:
  $Reward = \frac{w_{psr} \cdot NormalizedPSR + w_{congestion} \cdot NormalizedCONGESTION}{w_{etx} \cdot ETX}$
Q-Learning Mechanism: Nodes maintain a Q-table mapping candidate parents to Q-values. Upon selecting a parent and observing the outcome, the node updates the Q-value using the Bellman equation:
$Q_{new}(s_t, a_t) = (1 - \alpha)Q_{old}(s_t, a_t) + \alpha [R_t + \gamma \max Q(s_{t+1}, a_{t+1})]$
where $\alpha$ is the learning rate, $\gamma$ is the discount factor, and $R_t$ is the calculated reward.
Stability Mechanisms: To prevent network instability caused by frequent parent switching due to minor metric fluctuations, the authors implement:
- Hysteresis: A parent switch occurs only if the new candidate's Q-value exceeds the current parent's Q-value by a specific threshold (10%).
- $\epsilon$ -greedy Strategy: A dynamic balance between exploration (selecting random parents to discover better paths) and exploitation (selecting the parent with the highest Q-value). The exploration rate $\epsilon$ is reduced as Q-values converge.

Key Contributions

Decentralized Learning: The paper demonstrates that nodes can autonomously learn optimal routing decisions without centralized coordination, adapting to variable network conditions.
Integrated Metric Framework: The work proposes a specific combination of PSR, ETX, and Congestion Level within a Q-learning framework, arguing that this triad provides a more comprehensive view of network health than single-metric approaches.
Adaptive Weighting: The study provides an empirical analysis showing how the weighting of these metrics influences the trade-off between Packet Delivery Ratio (PDR) and latency. It suggests that metric weights should be adjusted based on traffic load (e.g., prioritizing congestion metrics under high load).
Algorithmic Implementation: The authors provide detailed algorithms for the Q-value update process and the parent selection process, including initialization strategies and convergence checks.

Results
The MM-QLOF approach was evaluated using the Cooja emulator (Contiki-NG) in a linear topology of 51 nodes under varying traffic loads (from 1 packet every 3 seconds to 1 packet every 0.5 seconds). The results were compared against standard OF0 and MRHOF:

Packet Delivery Ratio (PDR): MM-QLOF improved PDR by 10.7% compared to OF0 and 11.1% compared to MRHOF on average. Under high traffic (120 packets/min), it outperformed OF0 by 4.65% and MRHOF by 1.52%.
End-to-End Latency: The proposed method reduced latency by 18.7% compared to OF0 and 23.6% compared to MRHOF on average. In high-traffic scenarios, the reduction was even more pronounced (34.11% vs. OF0 and 44.66% vs. MRHOF).
Energy Consumption: Due to reduced retransmissions and delays, MM-QLOF achieved a slight reduction in power consumption compared to the baseline functions.
Metric Sensitivity: Empirical studies confirmed that prioritizing PSR and ETX improves PDR but may increase latency, whereas prioritizing Congestion significantly reduces latency but can slightly lower PDR if not balanced correctly.

Significance and Claims
The paper claims that MM-QLOF offers a significant improvement in routing efficiency for IoT networks, particularly under high-traffic conditions where traditional static objective functions fail. By incorporating congestion awareness directly into the parent selection logic via reinforcement learning, the protocol achieves a better trade-off between reliability (PDR), speed (latency), and energy efficiency. The authors emphasize that the approach allows for adaptive behavior, enabling nodes to dynamically avoid congested paths and saturated queues, thereby mitigating the negative effects of traffic spikes. The work suggests that while the current implementation is effective, future work could extend this model to mobile IoT scenarios and incorporate additional metrics like residual energy.