⚡ electrical engineering

Secure Energy Efficient Wireless Transmission: A Finite v/s Infinite-Horizon RL Solution

This paper proposes a finite-horizon reinforcement learning algorithm for joint transmit and jamming power allocation in energy-harvesting full-duplex wireless networks, demonstrating that it outperforms both greedy and infinite-horizon approaches in maximizing secrecy energy efficiency while reducing computational time.

Original authors: Shalini Tripathi, Ankur Bansal, Holger Claussen, Lester Ho, Chinmoy Kundu

Published 2026-01-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shalini Tripathi, Ankur Bansal, Holger Claussen, Lester Ho, Chinmoy Kundu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a relay race where two runners, Source and Destination, are trying to pass a secret message to each other. However, there's a catch: a sneaky Eavesdropper is lurking nearby, trying to steal the message. To stop the Eavesdropper, the Destination runner has a special trick: they can shout a loud, confusing noise (jamming) at the Eavesdropper while simultaneously listening to the Source.

But here's the real challenge: both runners are powered by solar panels (Energy Harvesting). They don't have a battery that lasts forever; they only have the energy they collect from the sun (or wind) at that exact moment, stored in a small, limited battery. If they use too much energy now, they might run out later. If they use too little, the message might get stolen.

The goal of this paper is to figure out the perfect strategy for how much power the Source should use to talk and how much power the Destination should use to shout (jam) so that they get the most "secret bits" across for every drop of energy they use. This is called Secrecy Energy Efficiency (SEE).

The authors compare three different ways to make these decisions:

1. The "Time-Traveling Planner" (FHJPA)

This is the paper's main solution. Imagine you know exactly how many minutes the race will last (a Finite Horizon). You can look at the entire race schedule from start to finish.

How it works: This algorithm works backward from the very last minute of the race to the first. It asks, "If I use a lot of energy now, will I have enough left for the final, crucial minutes?" It calculates the perfect balance for every single moment, knowing exactly when the race ends.
The Result: This is the smartest approach. It wins because it understands the "deadline." It doesn't waste energy early on if it needs to save it for the end, and it doesn't hold back if the end is far away.

2. The "Greedy Sprinter" (GA)

This is a simple, low-complexity approach. Imagine a runner who only cares about the very next step.

How it works: At every moment, it asks, "What gives me the best result right now?" It doesn't worry about tomorrow or the next minute. It just grabs the biggest immediate reward.
The Result: Usually, this is a bad idea because you might run out of energy before the race ends. However, the paper found a surprising twist: If the runners have plenty of solar energy (their batteries are full), this simple "greedy" strategy works almost as well as the complex planner. When energy is abundant, you don't need to worry about saving it for later, so the simple strategy is nearly perfect.

3. The "Forever Runner" (IHJPA)

This approach assumes the race never ends (an Infinite Horizon).

How it works: It plans for a steady state, assuming the race goes on forever. It creates a strategy that works well on average over an infinite amount of time.
The Result: This is a poor fit for a race with a known, short deadline. It's like trying to plan a budget for a lifetime when you only have a week to live. The paper shows that when the race is short, this method performs poorly because it doesn't account for the "end of the line." However, as the race gets longer (more time slots), this method gets better and starts to look more like the "Time-Traveling Planner."

The Big Takeaways

Knowing the Deadline Matters: The "Time-Traveling Planner" (Finite Horizon) is the best because it knows exactly when the transmission stops. It optimizes the energy usage specifically for that duration.
Abundance Changes the Game: If the energy harvest is high (lots of sun), you don't need a complex brain. A simple "greedy" strategy that just maximizes the current moment works almost as well as the complex one.
The Trade-off: The paper highlights a tricky balance. The "Time-Traveling Planner" is great at efficiency (getting the most bits per energy drop), but sometimes the "Greedy Sprinter" actually sends more total secret bits overall because it isn't afraid to use more power.
Speed vs. Smarts: The complex "Time-Traveling Planner" is actually faster to compute (16.6% faster) than the "Forever Runner" method, making it a practical choice for real-world devices.

In short, the paper teaches us that for short, energy-limited wireless transmissions, you need a strategy that knows the deadline. But if you have plenty of energy, you can get away with a much simpler, "live-in-the-moment" strategy.

Technical Summary: Secure Energy Efficient Wireless Transmission: A Finite v/s Infinite-Horizon RL Solution

Problem Statement
This paper addresses the optimization of wireless network security and energy efficiency in systems equipped with Energy Harvesting (EH) capabilities. Specifically, the authors consider a scenario involving a source node (S), a full-duplex destination node (D), and a passive eavesdropper (E). Both S and D are powered by rechargeable batteries with limited capacity and rely on stochastic energy harvesting processes. The destination utilizes its full-duplex capability to transmit jamming signals, thereby degrading the eavesdropper's reception while simultaneously receiving data from the source.

The core challenge is to jointly allocate the transmit power at the source ( $P_S$ ) and the jamming power at the destination ( $P_D$ ) over a finite time duration (a fixed number of time slots, $K$ ). The objective is to maximize the Average Secrecy Energy Efficiency (SEE), defined as the average number of secure bits transmitted per unit of energy consumed. The system is subject to causality constraints (energy cannot be used before it is harvested), battery capacity limits, and the Markovian nature of channel states and energy arrivals.

Methodology
The problem is formulated as a Finite-Horizon Markov Decision Process (MDP) because the transmission duration $K$ is known a priori and the system dynamics (channel gains and energy harvesting) evolve over time. The authors propose three distinct solution approaches to compare finite-horizon and infinite-horizon modeling strategies:

Finite-Horizon Joint Power Allocation (FHJPA): This is the proposed optimal solution for the finite-horizon problem. It utilizes a Backward Induction (BI) algorithm. The algorithm iterates from the final time slot ( $K$ ) backward to the first ($0$), computing the value function and the optimal policy (a look-up table of power pairs) for every possible state. The state includes channel gains, battery levels, and the current time slot.
Greedy Algorithm (GA): A low-complexity sub-optimal baseline. In each time slot, the GA selects the power allocation that maximizes the immediate reward (instantaneous SEE) without considering future rewards or state transitions.
Infinite-Horizon Joint Power Allocation (IHJPA): An algorithm designed for scenarios where the network lifetime is unknown or effectively infinite. It employs Policy Iteration (PI) to find a stationary policy that maximizes the long-term average reward, assuming a discount factor or a geometric distribution of the time horizon.

Key Contributions

Novel Formulation: The paper presents a novel joint optimization framework for transmit and jamming power that simultaneously incorporates EH capabilities at both the source and destination nodes while maximizing SEE. The authors note that prior works either focused on maximizing total secure bits rather than SEE, or assumed EH only at the destination.
Comparative Analysis of Horizons: This work is the first to investigate and compare both finite-horizon and infinite-horizon Reinforcement Learning (RL) models specifically for maximizing SEE in joint power allocation problems.
Algorithm Development: The authors propose the FHJPA algorithm for finite-horizon optimization and the IHJPA algorithm for infinite-horizon optimization, providing a detailed comparative analysis against a greedy baseline.
Insight on Energy Availability: The study provides insights into how the availability of harvested energy affects the optimality of greedy strategies versus optimal dynamic programming approaches.

Results
The performance of the algorithms was evaluated through simulations varying parameters such as harvested energy levels ( $E_S, E_D$ ), harvesting probabilities ( $p, q$ ), and the total number of time slots ( $K$ ).

Performance of FHJPA: The FHJPA algorithm consistently outperforms both the GA and IHJPA algorithms in terms of Average SEE. This superiority is attributed to the accurate modeling of the problem as a finite-horizon MDP, which accounts for the specific time constraints and battery dynamics.
Behavior of GA: While the GA generally underperforms FHJPA, its performance gap narrows significantly when the source node's battery has sufficient energy (due to high harvesting probability or large harvested energy units). In these high-energy scenarios, the greedy maximization of immediate reward becomes nearly optimal.
Behavior of IHJPA: The IHJPA algorithm performs poorly when the time horizon $K$ is small, as the infinite-horizon assumption does not accurately reflect the finite termination of the transmission. However, as $K$ increases, the accuracy of the infinite-horizon model improves, reducing the performance gap between FHJPA and IHJPA.
Trade-offs: The results indicate a trade-off between Average SEE and the total number of transmitted secure bits. The FHJPA algorithm, by optimizing for efficiency, may reduce transmit power to conserve energy, resulting in fewer total secure bits compared to the GA, which aggressively transmits when energy is abundant.
Computational Complexity: The FHJPA algorithm requires a planning phase with complexity $O((K-1)N_S^2 N_A)$ , whereas the GA has no planning phase. Interestingly, the computational time analysis shows that the FHJPA algorithm takes 16.6% less time than the IHJPA algorithm, suggesting that for finite-horizon problems, the backward induction approach is more computationally efficient than policy iteration for infinite horizons in this specific context.

Significance and Claims
The paper claims that for wireless networks with known finite transmission durations and energy harvesting constraints, finite-horizon modeling is superior to infinite-horizon modeling for maximizing Secrecy Energy Efficiency. The authors emphasize that while infinite-horizon models are common in literature, they may yield suboptimal results when the network lifetime is finite and known.

Furthermore, the paper highlights a practical insight: when energy is abundant, low-complexity greedy strategies can approximate optimal performance, offering a viable alternative for systems where computational resources are limited. Finally, the authors conclude that system optimization should not focus solely on maximizing SEE or total secure bits in isolation but should seek a balance between these metrics based on specific network requirements.

1. The "Time-Traveling Planner" (FHJPA)

2. The "Greedy Sprinter" (GA)

3. The "Forever Runner" (IHJPA)

The Big Takeaways

More like this