⚡ electrical engineering

Optimisation of Resource Allocation in Heterogeneous Wireless Networks Using Deep Reinforcement Learning

This paper proposes a Deep Reinforcement Learning-based xApp for the Near-RT RIC in O-RAN heterogeneous networks that jointly optimizes transmit power, bandwidth slicing, and user scheduling, demonstrating significant reductions in energy consumption and improvements in user fairness compared to traditional heuristics.

Original authors: Oluwaseyi Giwa, Jonathan Shock, Jaco Du Toit, Tobi Awodumila

Published 2026-04-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Oluwaseyi Giwa, Jonathan Shock, Jaco Du Toit, Tobi Awodumila

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a bustling city where the "internet" is like a massive, chaotic traffic system. In this city, we have two types of roads:

Highways (Macro Cells): Big, powerful roads that cover large areas but get very crowded.
Local Streets (Micro Cells): Smaller, quieter roads that handle local traffic but can get blocked easily if the highways are too loud.

In the past, traffic lights (the network controllers) were set to simple, rigid timers. They didn't know if a street was empty or jammed; they just turned green or red on a schedule. This led to massive traffic jams (interference) and wasted fuel (energy).

This paper proposes a super-smart, AI-powered traffic controller for the future of mobile networks (5G and 6G). Here is how they did it, explained simply:

1. The Problem: The "Traffic Jam" of Signals

The authors looked at a real-world map of Cape Town, South Africa. They saw that when you have too many cell towers close together, they start shouting over each other.

The Issue: If every tower shouts at full volume to make sure you hear them, they drown out their neighbors. This wastes battery power and makes the connection slower for everyone.
The Goal: They wanted to find a way to tell each tower exactly how loud to shout (power), which road to use (bandwidth), and who to talk to first (scheduling) so that everyone gets a clear signal without wasting energy.

2. The Solution: The "AI Traffic Cop" (The xApp)

Instead of using old, rigid rules, they built a Deep Reinforcement Learning (DRL) agent. Think of this agent as a super-intelligent traffic cop sitting in a control tower (called the Near-RT RIC).

How it learns: It doesn't have a map of the rules. Instead, it learns by trial and error, just like a baby learning to walk.
- Step 1: It tries a strategy (e.g., "Turn down the volume on Tower A").
- Step 2: It checks the result. Did the traffic flow better? Did we save fuel?
- Step 3: If it worked, it gets a "gold star" (Reward). If it caused a crash, it gets a "ding" (Penalty).
- Step 4: Over thousands of tries, it learns the perfect strategy to keep traffic moving smoothly.

3. The Two "Student" Algorithms

The researchers tested two different types of "student" AI to see which one learned the best:

Student A (TD3): This student is like a speed runner. It learns very fast at the beginning but sometimes gets confused by complex situations and makes mistakes later on. It's good for quick fixes but not always the most stable.
Student B (PPO): This student is like a careful strategist. It learns a bit slower at first, but it is very steady. It doesn't make wild guesses. It carefully explores different options and eventually finds a solution that is better for everyone in the long run.

4. The Results: The "Goldilocks" Winner

They ran a simulation using real Cape Town tower locations and 50 "drivers" (users) moving around. Here is what happened:

The Old Way (Greedy): The traditional method was like a driver who only cares about getting to the destination as fast as possible, ignoring everyone else. It used maximum fuel (energy) and left people on the edge of the city (cell-edge users) with no signal.
The AI Way (PPO): The PPO AI found the "Goldilocks" zone.
- Energy Savings: It cut energy consumption by 70% in crowded areas. It learned to whisper instead of shout when it didn't need to be loud.
- Fairness: It made sure the people on the "edge" of the network got a fair share of the signal, improving fairness by 30%.
- Speed: It kept the internet fast for everyone, not just the lucky ones in the center.

5. Why This Matters for the Future (6G)

The paper concludes that this AI approach is ready for the future of 6G networks.

It's Green: It saves massive amounts of electricity, which is great for the planet.
It's Fair: It ensures that no one gets left behind with a bad connection.
It's Real: They didn't just use a fake computer map; they used real coordinates from a real city, proving this works in the messy, real world.

In a nutshell: The authors built a smart AI traffic controller that learns to manage a crowded city of cell towers. By teaching the towers to whisper when they can and shout only when necessary, they saved huge amounts of energy and made sure everyone got a fair, fast internet connection. The "careful strategist" AI (PPO) won the race, proving that patience and balance beat speed and greed in network management.

1. Problem Statement

The paper addresses the complex challenge of dynamic resource allocation (RA) in Open Radio Access Network (O-RAN) based Heterogeneous Networks (HetNets).

Context: Modern 5G/6G networks overlay traditional macrocells with dense tiers of small cells (micro, pico, femto) to increase capacity. This density creates severe co-tier and cross-tier interference.
Challenge: Traditional RA strategies (classical optimization or static heuristics) fail to handle the non-convex, combinatorial nature of joint RA problems in real-time. They often rely on simplified models and lack a global view of interference.
Goal: To jointly optimize transmit power, bandwidth slicing, and user scheduling to maximize network throughput and user fairness while minimizing energy consumption, all within the near-real-time constraints of an O-RAN architecture.

2. Methodology

The authors propose a Deep Reinforcement Learning (DRL) framework implemented as an xApp (an application running on the Near-Real-Time RAN Intelligent Controller, or Near-RT RIC).

A. System Model & MDP Formulation

Architecture: A centralized Near-RT RIC controls multiple Base Stations (BSs) via the E2 interface.
Markov Decision Process (MDP):
- State Space ( $S$ ): Aggregates global network observables including previous power allocations, estimated interference (from UE Channel Quality Indicator reports), and fixed topology geometry.
- Action Space ( $A$ ): A hierarchical, continuous action space mapping normalized values to physical quantities:
  1. Transmit Power: Scaled to physical limits ( $P_{min}$ to $P_{max}$ ).
  2. Bandwidth Slicing: Dynamic adjustment of bandwidth per BS.
  3. User Scheduling: A priority weight derived via a softmax function to bias the local scheduler toward specific users (e.g., cell-edge) to enforce fairness.
- Reward Function ( $R$ ): A scalarized utility function balancing three objectives:
  - Total Throughput ( $\alpha$ )
  - User Fairness via Jain's Fairness Index ( $\beta$ )
  - Energy Efficiency (inverse of total power) ( $\kappa$ )

B. Algorithms Compared

The paper benchmarks two state-of-the-art continuous control DRL algorithms:

Twin Delayed Deep Deterministic Policy Gradient (TD3): An off-policy algorithm using clipped double Q-learning, delayed policy updates, and target policy smoothing to mitigate overestimation bias and instability.
Proximal Policy Optimisation (PPO): An on-policy algorithm using a clipped surrogate objective to constrain policy updates, ensuring stable training and preventing destructive large steps.

C. Simulation Environment

Topology: Unlike synthetic Poisson Point Process (PPP) models, the simulation uses real-world geospatial coordinates of 3 Macro BSs and 10 Micro BSs from a telecom operator in Cape Town, South Africa.
Channel Model: Includes path loss, log-normal shadowing, and Rayleigh fading.
Baselines: Compared against three standard heuristics:
- Greedy OFDMA (G-OFDMA): Maximizes SINR per user.
- Interference Pricing (IP-PC): Reduces power based on neighbor feedback.
- Proportional Fair (PF-EQ): Standard fairness baseline.

3. Key Contributions

Realistic MDP Formulation: Developed a Near-RT RIC-compatible MDP where a central agent manages power and scheduling using global channel knowledge, justified by O-RAN E2 feedback loops.
Real-World Topology: Implemented a simulation environment using actual BS coordinates to capture realistic interference geometries, moving beyond idealized theoretical models.
Mathematical Derivation: Provided a derivation of throughput and fairness metrics from continuous RL actions, specifically using a softmax-based mechanism to translate RL outputs into valid scheduling weights.
Comprehensive Benchmarking: Conducted a rigorous comparison between PPO and TD3 against standard heuristics across multiple topological scenarios (Dense Urban, Sparse Suburban, Hotspot, Mixed).

4. Key Results

The experiments were conducted over 1,000 episodes with 1,000 steps each.

Energy Efficiency: The PPO-based xApp achieved a ~70% reduction in network energy consumption in dense urban scenarios compared to the throughput-greedy G-OFDMA baseline, while maintaining high throughput.
User Fairness: PPO significantly improved fairness, achieving a Jain's Fairness Index of 0.65–0.75 across all topologies. In the Mixed Scenario, it improved fairness by >300% compared to G-OFDMA and >100% compared to IP-PC.
Throughput vs. Energy Trade-off:
- G-OFDMA offered high throughput but at maximum energy cost.
- IP-PC minimized energy but starved users of throughput.
- PPO found the optimal Pareto front, reducing energy without sacrificing per-user data rates.
Algorithm Performance:
- TD3: Converged faster initially (high sample efficiency) but suffered from instability and performance degradation in later stages due to value overestimation in complex interference landscapes.
- PPO: Demonstrated stable, monotonic convergence and achieved a significantly higher mean reward, making it the superior candidate for this application.
Computational Feasibility: While DRL inference is slower than heuristics, the latency remains within the 10ms–1s window required for Near-RT RIC xApps. The neural network models are lightweight (<10MB), satisfying memory constraints.

5. Significance and Conclusion

Feasibility of AI in 6G: The study validates the feasibility of deploying centralized, energy-aware AI orchestration within future 6G O-RAN architectures.
Beyond Theory: By using real-world topology data, the results bridge the gap between theoretical DRL research and practical deployment constraints.
Strategic Recommendation: While TD3 is suitable for rapid adaptation, PPO is recommended for performance-critical, energy-constrained environments due to its robustness and ability to learn globally optimal policies that balance throughput, fairness, and energy.
Future Work: The authors suggest extending the framework to distributed multi-agent DRL (e.g., MAPPO) to address scalability and incorporating high-speed user mobility effects.

In summary, this paper demonstrates that a PPO-based DRL agent acting as an O-RAN xApp can effectively solve the joint resource allocation problem in heterogeneous networks, offering a superior trade-off between energy efficiency, user fairness, and network capacity compared to traditional heuristic methods.