Reinforcement Learning for Secrecy Optimization in Underwater Energy Harvesting Relay Network

This paper proposes a model-based reinforcement learning strategy to optimize relay power allocation in an underwater hybrid optical-acoustic energy harvesting network, effectively maximizing long-term secrecy performance against acoustic eavesdropping while adapting to dynamic channel conditions and battery constraints.

The Gist

Imagine a team of underwater explorers trying to send a secret message from a ship on the surface to a submarine deep below. They have a middleman—a relay node—that helps pass the message along. But there's a catch: the ocean is a tricky place, and a sneaky eavesdropper is lurking nearby, trying to steal the message.

Here is the story of how the researchers in this paper solved the problem of keeping the message secret while making sure the middleman doesn't run out of battery.

The Setting: A Two-Legged Journey

Think of the message delivery as a relay race with two very different legs:

1. The First Leg (Surface to Relay): This uses light (like a high-speed flashlight). It's incredibly fast and carries a lot of data, but it's fragile. If a fish swims in front of the beam or the water gets murky (turbulence), the light gets blocked or scattered. It's like trying to shine a laser pointer through a foggy window; sometimes it works, sometimes it doesn't.
2. The Second Leg (Relay to Submarine): This uses sound (like a sonar ping). It's slower and carries less data, but it travels far and is reliable. However, sound waves travel through water like ripples in a pond—they spread out everywhere. This means the sneaky eavesdropper can easily hear the message.

The Problem: The "Smart" Middleman

The relay node (the middleman) is special. It doesn't have a giant battery plugged into a wall. Instead, it's like a solar-powered watch that runs on energy harvested from the ocean (perhaps from waves or currents).

• The Energy Issue: Sometimes it gets a burst of energy; sometimes it gets nothing. It has a small battery that can only hold so much.
• The Security Issue: If the relay shouts too loudly (uses too much power), the eavesdropper hears the message clearly. If it whispers too quietly, the message might not reach the submarine, or the eavesdropper might still hear it better than the submarine does.

The goal? Send as many secret bits as possible before the network breaks down or runs out of time.

The Solution: Three Different Strategies

The researchers tested three different ways for the relay to decide how loudly to shout (how much power to use).

1. The "Crystal Ball" Strategy (Reinforcement Learning / OPA)

This is the Optimal Power Allocation (OPA) method.

• The Analogy: Imagine a chess grandmaster who doesn't just look at the next move, but calculates the entire game. This AI agent learns by playing the game thousands of times in a simulation. It learns that "If I use a little power now, I save energy for later when the water is clearer, and I can send a bigger secret message then."
• How it works: It looks at the current battery level, the weather (turbulence), and the eavesdropper's location. It makes a decision that might seem "wasteful" right now but guarantees the most secret messages over the long run.
• Result: This was the winner. It adapted perfectly to the changing ocean conditions.

2. The "Greedy" Strategy (GA)

• The Analogy: This is like a person who only cares about what they can eat right now. "I'm hungry, so I'll eat the whole cake!"
• How it works: The relay looks at the current moment and asks, "What gives me the most secret data this second?" It doesn't care if it runs out of battery in the next minute.
• Result: It did okay, but it often burned through its energy too fast, leaving it with nothing to send when conditions got tough later.

3. The "Naive" Strategy (NA)

• The Analogy: This is like a kid who dumps their entire piggy bank on the table and spends it all in one go.
• How it works: The relay just uses everything it has in the battery for every single message. No planning, no saving.
• Result: This performed the worst. It ran out of energy almost immediately and couldn't send many secret messages.

The Key Takeaways

The researchers found that:

• Thinking ahead pays off: The "Crystal Ball" strategy (RL) was the best because it balanced the need to send data now with the need to survive later.
• Obstacles matter: If the underwater "fog" (obstacles) gets thicker, the light link fails more often, and everyone sends fewer messages.
• Battery size helps: A bigger battery gives the relay more room to make smart choices.
• The Eavesdropper is a problem: If the eavesdropper gets closer to the relay, the "secret" part of the message shrinks, because the relay has to whisper so quietly that the submarine might not hear it at all.

In a Nutshell

This paper is about teaching an underwater robot to be a smart, long-term planner. Instead of just reacting to the moment (like the Greedy or Naive robots), the smart robot learns to save its energy and time its transmissions perfectly to outsmart the eavesdropper and keep the secret message safe, even when the ocean is chaotic and unpredictable.

Technical Summary

Here is a detailed technical summary of the paper "Reinforcement Learning for Secrecy Optimization in Underwater Energy Harvesting Relay Network."

1. Problem Statement

The paper addresses the challenge of secure communication in a hybrid Underwater Wireless Communication (UWC) network. The system consists of:

• Source (S): A surface vehicle transmitting data via an Underwater Optical (UWO) link.
• Relay (R): An underwater node equipped with Energy Harvesting (EH) capabilities, which receives data optically and forwards it via an Underwater Acoustic (UWA) link.
• Destination (D): The intended receiver.
• Eavesdropper (E): A passive node monitoring the acoustic link.

Key Challenges:

• Hybrid Channel Dynamics: The UWO link (S-to-R) is high-speed but susceptible to turbulence, pointing errors, and physical blockage by obstacles. The UWA link (R-to-D) is reliable over long distances but has low bandwidth and is vulnerable to interception.
• Energy Constraints: The relay operates on limited battery capacity and relies on stochastic energy harvesting (modeled as a Bernoulli process).
• Network Lifetime: The network has a random lifetime due to potential hardware failure or physical damage, modeled as a geometric random variable.
• Objective: The goal is to maximize the long-term expected number of securely transmitted bits before the network fails, by optimally allocating the relay's transmit power while satisfying secrecy constraints (Secrecy Rate $\geq$ Threshold).

2. Methodology

The authors formulate the power allocation problem as an Infinite-Horizon Markov Decision Process (MDP) and propose three distinct strategies to solve it.

A. System Modeling

• State Space ($S$): Defined by the channel gains of the Relay-Destination ($G_{RD}$) and Relay-Eavesdropper ($G_{RE}$) links, and the current battery level ($B_R$).
• Action Space ($A$): The set of discrete transmit power levels available to the relay.
• Reward: The instantaneous secrecy capacity ($C_S$) of the acoustic link, provided it meets a Quality of Service (QoS) threshold ($R_{th}$). If the threshold is not met, the reward is zero.
• Channel Models:
  • UWO: Modeled as a composite Gamma-Gamma turbulence channel with pointing errors and a binary obstruction coefficient.
  • UWA: Modeled with frequency-dependent attenuation and ambient noise.

B. Proposed Solutions

1. Optimal Power Allocation (OPA) - Model-Based RL:

   • Utilizes Policy Iteration (PI) to solve the MDP.
   • Planning Phase: Computes an optimal stationary deterministic policy (a lookup table) by iteratively performing Policy Evaluation (updating the value function via Bellman equations) and Policy Improvement (selecting actions that maximize the value function).
   • Transmission Phase: The relay uses the pre-computed lookup table to select the optimal power level based on the current state (battery and channel conditions) without needing future knowledge.
   • Goal: Maximizes the cumulative discounted reward over the infinite horizon.

2. Greedy Algorithm (GA):

   • A sub-optimal, low-complexity approach.
   • Selects the power level that maximizes the instantaneous reward in the current time slot only, ignoring future consequences (no planning phase).

3. Naive Algorithm (NA):

   • The simplest approach.
   • The relay transmits using all available battery energy in every time slot ($P_R = B_R / T_s$), regardless of channel conditions or future energy needs.

3. Key Contributions

• Hybrid Secrecy Framework: The paper is the first to model a hybrid optical-acoustic underwater relay system with energy harvesting and secrecy constraints simultaneously, addressing the trade-off between optical blockage and acoustic eavesdropping.
• MDP Formulation: The problem is rigorously formulated as an infinite-horizon MDP to handle the stochastic nature of energy harvesting, channel fading, and random network termination.
• Algorithm Development:
  • Proposes a Model-Based RL (Policy Iteration) solution (OPA) that adapts to battery dynamics and channel uncertainty.
  • Introduces two baseline algorithms (GA and NA) to benchmark complexity vs. performance.
• Complexity Analysis: Provides a theoretical analysis showing that while OPA has high computational cost during the planning phase ($O(N_S^2 N_A)$), its transmission phase is efficient ($O(K)$). GA and NA have lower planning costs but sub-optimal performance.

4. Simulation Results

The authors evaluated the schemes using parameters such as Gamma-Gamma turbulence, Bernoulli energy harvesting, and varying obstacle densities.

• Performance Comparison:
  • OPA consistently achieves the highest secure throughput. It effectively balances immediate transmission needs with future energy availability and channel conditions.
  • GA performs moderately well but falls short of OPA because it lacks long-term foresight.
  • NA performs the poorest due to its "short-sighted" strategy of draining the battery immediately, leading to premature network failure.
• Impact of Discount Factor ($\Gamma$): Increasing $\Gamma$ (emphasizing future rewards) improves the performance of OPA significantly, confirming the benefit of long-term planning.
• Obstacle Density: Higher obstacle density in the optical link reduces the overall reward for all schemes, as fewer data packets reach the relay to be forwarded securely.
• Energy Harvesting (EH) Probability ($p$):
  • As $p$ increases, the performance gap between OPA, GA, and NA narrows. When energy is abundant, the advantage of complex long-term planning diminishes.
  • Higher harvested energy amounts ($E_R$) directly improve secrecy performance.
• Battery Capacity: Larger battery capacities allow for better energy storage, extending network lifetime and increasing total secure bits.
• Eavesdropper Distance: A shorter distance between the Relay and Eavesdropper ($l_{RE}$) significantly degrades secrecy capacity, highlighting the critical role of physical distance in acoustic security.

5. Significance and Conclusion

This paper demonstrates that Reinforcement Learning is a vital tool for optimizing secure underwater communications in dynamic, energy-constrained environments.

• Practical Implication: The proposed OPA strategy enables underwater networks to operate more intelligently, adapting to unpredictable energy sources and channel blockages to maximize security.
• Trade-off Management: The study highlights that while simple greedy or naive approaches are computationally cheaper, they fail to maximize the network's lifetime and security in realistic, stochastic underwater environments.
• Future Outlook: The work establishes a foundation for intelligent, adaptive underwater networks that can autonomously manage resources to counter eavesdropping threats, which is crucial for applications like environmental monitoring and offshore surveillance.