🔢 mathematics

Robust Beam Codebooks for mmWave/THz Systems: Toward a Stochastic RL Approach

This paper proposes a robust multi-agent reinforcement learning framework for designing adaptive beamforming codebooks in mmWave/THz systems without prior channel knowledge, demonstrating through simulations that the Soft Actor-Critic (SAC) algorithm outperforms deterministic methods in achieving superior stability and gains under hardware impairments and Non-Line-of-Sight conditions.

Original authors: Anouar Nechi, Rainer Buchty, Mladen Berekovic, Saleh Mulhem

Published 2026-03-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anouar Nechi, Rainer Buchty, Mladen Berekovic, Saleh Mulhem

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to talk to a friend in a crowded, noisy stadium. You both have walkie-talkies, but the signal is weak, and the crowd is shouting. To hear each other, you need to point your antennas (like flashlights) in the exact right direction to cut through the noise.

In the world of 6G and next-gen internet (using mmWave and THz frequencies), this "flashlight" is called beamforming. The challenge is that the stadium (the environment) changes constantly. People move, walls block signals, and the walkie-talkies themselves might be a little bit broken or "out of tune."

This paper is about teaching these antennas to learn how to point themselves without needing a perfect map of the stadium.

Here is the breakdown of their solution using simple analogies:

1. The Problem: The "Perfect Map" Doesn't Exist

Traditionally, engineers tried to pre-program a list of directions (a Codebook) for the antennas to check.

The Analogy: Imagine a security guard checking every single door in a massive building to find a lost child. It takes forever (high overhead), and if the child is hiding behind a wall (Non-Line-of-Sight), the guard might miss them.
The Flaw: This method assumes the building is perfect and the guard's eyesight is perfect. But in reality, the building shifts, and the guard's eyesight is blurry (hardware impairments).

2. The Solution: Let the Antenna "Learn by Doing"

Instead of using a pre-written list, the authors use Reinforcement Learning (RL).

The Analogy: Think of a toddler learning to walk. They don't have a manual. They try taking a step, fall, get a "ouch" (negative feedback), try again, and eventually find a way to stand up.
How it works: The antenna tries a direction. If the signal gets stronger, it gets a "high five" (positive reward). If it gets weaker, it gets a "thumbs down." Over time, it learns the best angles without ever needing to know the physics of the room or the exact location of the user.

3. The Big Question: Which "Learner" is Best?

The researchers tested three different types of "learners" (algorithms) to see which one handles a messy, noisy world best:

DDPG & TD3 (The Deterministic Learners): These are like rigid robots. Once they think they found the best path, they stick to it. If the floor tilts slightly (hardware error), they keep walking in the same direction and might fall off a cliff. They are very confident, but not very flexible.
SAC (The Stochastic Learner): This is like a curious explorer. Instead of picking one single "best" direction, it learns a range of good directions. It keeps a little bit of "wobble" in its decision-making.
- The Metaphor: If you are walking in a foggy forest, a rigid robot might walk straight into a tree because it thinks the path is clear. The "wobbly" explorer (SAC) keeps checking its surroundings and adjusting its path slightly, making it much harder to get lost or hit a tree.

4. The Stress Test: Breaking the System

The researchers didn't just test these learners in a perfect lab. They broke things on purpose to see who would survive:

Broken Hardware: They simulated antennas that were slightly out of tune (like a guitar string that is a tiny bit loose).
Noisy Feedback: They simulated the "high five" or "thumbs down" being garbled by static noise.

The Results:

The Rigid Robots (DDPG/TD3) got confused quickly. When the feedback was noisy, they got stuck in bad patterns or fell off the cliff.
The Curious Explorer (SAC) was a champion. Even when the feedback was 40% corrupted (like trying to walk while someone is shouting nonsense in your ear), it kept finding the best path. Its "wobble" allowed it to ignore the noise and find the true signal.

5. The Multi-Agent Team

Since there are many users (people in the stadium), one antenna can't learn for everyone at once.

The Analogy: Imagine a coach splitting the team into small groups based on where they are sitting. Each group gets its own mini-coach (an AI agent) to learn the best way to point the flashlight for that specific group.
This makes the learning process fast and efficient, ensuring everyone gets a strong signal.

The Bottom Line

This paper proves that for future 6G networks, we shouldn't rely on rigid, pre-programmed rules. Instead, we should use AI agents that are flexible and slightly "wobbly" (Stochastic RL).

Just like a skilled sailor doesn't fight the wind with a rigid mast but adjusts the sails to ride the waves, the SAC algorithm adjusts its beam to ride the noisy, imperfect waves of the real world, ensuring your internet stays fast even when the hardware is imperfect or the environment is chaotic.

1. Problem Statement

Millimeter-wave (mmWave) and Terahertz (THz) massive MIMO systems rely on large antenna arrays to overcome high path loss. However, practical deployment faces significant challenges:

Hardware Limitations: Analog beamforming is cost-effective but restricts channel estimation. Predefined beamforming codebooks (BC) often fail in Non-Line-of-Sight (NLoS) conditions and are suboptimal for hardware-limited transceivers.
Impairments: Real-world systems suffer from phase shifter quantization, oscillator phase noise, and imperfect array calibration. These impairments distort beam patterns and degrade performance, especially when Channel State Information (CSI) is unavailable or outdated.
Feedback Noise: In data-driven approaches, noisy feedback from users can mislead learning algorithms, causing them to converge to suboptimal solutions.
The Gap: While Reinforcement Learning (RL) has been proposed to learn adaptive codebooks without explicit CSI, the robustness of these algorithms against hardware impairments and noisy feedback remains under-explored.

2. Methodology

The authors propose a Robust Multi-Agent Reinforcement Learning (MARL) framework designed to learn beam codebooks directly from environmental feedback.

A. System Model & MDP Formulation

Scenario: A Base Station (BS) with a Uniform Linear Array (ULA) serves single-antenna users via analog beamforming.
Objective: Maximize the average beamforming gain across user clusters without explicit CSI.
Markov Decision Process (MDP):
- State ( $S$ ): The current phase values of all $M$ phase shifters.
- Action ( $A$ ): Continuous phase adjustments output by the actor network, mapped to discrete hardware values via K-Nearest Neighbors (KNN).
- Reward ( $R$ ): A ternary reward mechanism ( $\{-1, 0, +1\}$ ${- 1, 0, + 1}$ ) based on relative gain improvements rather than absolute values. This acts as a high-pass filter to ignore static noise biases.
  - $+1$ : Gain exceeds an adaptive threshold.
  - $0$: Gain improves but is below optimal.
  - $-1$: Gain decreases.

B. Multi-Agent Architecture

To handle the high dimensionality of codebook design:

Channel Clustering: Users are grouped into clusters based on angular characteristics (using sensing beams and power differences) rather than raw power, mitigating the "near-far" effect.
Agent Assignment: A linear sum assignment problem (solved via the Hungarian algorithm) matches each cluster to the RL agent best initialized for it.
Parallel Learning: Agents learn independently for their assigned clusters.

C. Algorithm Selection & Robustness Framework

The paper compares three off-policy algorithms:

Deterministic Policies: Deep Deterministic Policy Gradient (DDPG) and Twin Delayed DDPG (TD3). These rely on fixed exploration noise (Ornstein-Uhlenbeck) and are prone to overestimation or failure when the "true" optimal point shifts due to hardware errors.
Stochastic Policy: Soft Actor-Critic (SAC). SAC optimizes a probability distribution over actions, maximizing both reward and policy entropy.
- Key Insight: Entropy regularization allows SAC to learn a "cloud" of valid configurations rather than a single sharp point, making it naturally robust to phase mismatches and noise.

D. Stress-Testing Conditions

The framework is evaluated under two specific impairment models:

Noisy Feedback: Additive White Gaussian Noise (AWGN) scaling with signal strength.
Hardware Impairments: Unknown, fixed phase mismatches ( $\Delta\theta$ ) modeled as Gaussian deviations in the array response.

3. Key Contributions

Robustness Formulation: Introduced a rigorous methodology to model hardware impairments as phase mismatches and feedback as noisy signals, explicitly testing RL resilience.
Algorithmic Comparison: Demonstrated that stochastic policies (SAC) are significantly more robust than deterministic policies (DDPG/TD3) in the presence of hardware impairments and noisy feedback.
Adaptive Exploration Analysis: Showed that SAC's ability to autonomously adjust its entropy temperature (exploration rate) allows it to filter out noise and converge, whereas deterministic algorithms with fixed noise schedules fail to distinguish between exploration and environmental stochasticity.
Performance Benchmarks: Established that the proposed strategy maintains high beamforming gains even with 40% feedback noise, identifying the operational limits for data-driven beamforming in THz systems.

4. Results

Experiments were conducted using the DeepMIMO dataset across 60 GHz (LoS) and 28 GHz (NLoS) scenarios.

Baseline Performance (Ideal Conditions):
- SAC consistently outperformed DDPG and TD3.
- In NLoS scenarios with 16 beams, SAC achieved 80.0% of the Equal Gain Combining (EGC) upper bound, compared to 78.5% for DDPG and 78.0% for TD3.
Robustness to Hardware Impairments:
- As phase mismatch standard deviation ( $\sigma_p$ ) increased, SAC maintained the highest gain.
- Deterministic policies showed higher volatility, particularly in "critical matching" regimes (e.g., 8-beam setups) where minor phase shifts caused beam switching.
Robustness to Feedback Noise:
- Under 40% feedback noise, SAC retained roughly 78% of its noise-free performance in NLoS scenarios.
- In contrast, DDPG lost nearly 36% of its initial gain.
- SAC's stochastic nature effectively averaged out zero-mean noise, while deterministic policies were misled by transient positive noise spikes (overestimation).
Exploration Dynamics:
- SAC's entropy temperature ( $\alpha$ ) dropped rapidly (by 3 orders of magnitude) as it learned, indicating a successful transition from exploration to exploitation.
- DDPG/TD3 relied on rigid, decaying noise schedules that failed to adapt to the noisy environment, hindering convergence.

5. Significance

This paper establishes Stochastic Reinforcement Learning (specifically SAC) as a critical enabler for next-generation mmWave/THz communications.

Hardware Agnosticism: The approach does not require precise calibration or knowledge of array geometry, making it suitable for low-cost, imperfect hardware.
Resilience: It proves that RL-based beamforming can survive severe real-world conditions (phase noise, feedback corruption) where traditional model-based or deterministic learning methods fail.
Practical Deployment: The findings suggest that future massive MIMO systems should adopt entropy-regularized, stochastic learning strategies to ensure reliable connectivity in dynamic and imperfect environments.