⚡ electrical engineering

Beam Scheduling for Cross-Layer ISAC: A Deep Reinforcement Learning Approach

This paper proposes a deep reinforcement learning (DRL)-assisted beam scheduling approach for cross-layer ISAC systems that optimizes the trade-off between communication latency and sensing estimation error by leveraging sensing observations to reduce feedback overhead in dynamic environments.

Original authors: Xiyu Wang, Gilberto Berardinelli, Hei Victor Cheng, Petar Popovski, Ramoni Adeogun

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiyu Wang, Gilberto Berardinelli, Hei Victor Cheng, Petar Popovski, Ramoni Adeogun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a high-tech lighthouse operator in a busy, foggy harbor. Your job is twofold: you must shine a bright light to guide ships (the Communication part) and use a radar to track where those ships and floating debris are moving (the Sensing part).

This paper describes a "smart" way to manage this lighthouse using Artificial Intelligence. Here is the breakdown:

1. The Problem: The "Juggling Act"

In a traditional setup, you might have one light for guiding ships and a separate radar for tracking. But in the new world of ISAC (Integrated Sensing and Communication), we want to use the same light beam to do both jobs at once to save energy and equipment.

This creates a massive headache:

The Communication Need: Ships (users) are carrying urgent cargo. If your light isn't pointed exactly at them, they "drop" their cargo (data loss) or get stuck waiting too long (latency).
The Sensing Need: You need to know exactly where the ships are and how fast they are moving so you don't hit a rock or a piece of debris (clutter).
The Conflict: If you spread your light into a wide, soft glow, it’s easier to "see" the whole harbor (good sensing), but the ships can't see the signal clearly (bad communication). If you use a tiny, laser-focused beam, the ships get great signal, but if they move even an inch, you "lose" them in the dark (bad sensing).

2. The Solution: The "AI Lighthouse Keeper"

Instead of following a rigid, manual rulebook, the researchers proposed using Deep Reinforcement Learning (DRL).

Think of this like training a puppy. You don't give the puppy a manual on "How to Sit"; instead, you give it a treat when it sits and a gentle "no" when it doesn't. The AI "agent" (the lighthouse keeper) tries different beam angles and widths.

If it successfully guides a ship and tracks its speed accurately, it gets a "reward" (a digital treat).
If a ship gets lost or a collision is detected, it gets a "penalty."

Over thousands of practice rounds, the AI learns a "gut instinct" for the harbor. It learns that when the fog gets thick or ships start moving faster, it should switch from a single laser beam to a "multi-beam" approach—spreading the light slightly to create a wider safety net.

3. The "Secret Sauce": Learning from Echoes

Usually, to know where a ship is, the ship has to radio you and say, "Hey, I'm over here!" This is called "feedback," and in a busy harbor, those radio calls can clog up the airwaves.

The clever part of this paper is that the AI doesn't ask for help. It uses the "echoes" of its own light bouncing off the ships to figure out where they are. It’s like being in a dark room and throwing a ball against a wall; by listening to how the ball bounces back, you can tell how big the room is and where the furniture is located without ever turning on a flashlight.

4. The Results: Why it Matters

The researchers tested this "AI Keeper" in a simulated digital harbor filled with moving ships and "clutter" (like floating junk). They found that:

It’s a Great Multi-tasker: The AI managed to keep data moving fast (low latency) while keeping a very close eye on the ships' positions.
It’s Resilient: Even when the environment got messy with "clutter," the AI learned to distinguish between a real ship and a piece of floating trash.
It’s Efficient: By using the "echoes" to guide itself, it didn't need to waste time and energy asking the ships for constant updates.

Summary in a Sentence

Instead of choosing between a "flashlight for talking" and a "radar for seeing," this paper uses an AI that learns to use a single beam of light to do both perfectly, by watching how the light bounces off the world.

Technical Summary: Beam Scheduling for Cross-Layer ISAC: A Deep Reinforcement Learning Approach

1. Problem Statement

The paper addresses the challenge of resource allocation (specifically beam scheduling) in Integrated Sensing and Communication (ISAC) systems within dynamic multi-user environments. In 6G networks, ISAC systems must simultaneously satisfy two often-conflicting objectives:

Communication: Achieving low-latency data transmission while managing stochastic data traffic and finite buffer constraints (cross-layer dynamics).
Sensing: Minimizing estimation errors for target parameters (range, velocity, and Angle-of-Departure (AoD)) to maintain accurate tracking.

Key Challenges:

Analytical Intractability: The coupling between physical-layer beamforming, wireless channel variations (including beam drifting), and upper-layer queue dynamics (buffer overflows and packet deadlines) makes it impossible to derive closed-form optimization expressions.
Feedback Overhead: Traditional methods rely on explicit Channel State Information (CSI) or user feedback, which consumes significant control signaling overhead and may be outdated in high-mobility scenarios.
Environmental Complexity: Clutter (stationary objects) and high user mobility create interference and signal degradation, complicating beam alignment.

2. Methodology

The authors propose a Deep Reinforcement Learning (DRL)-assisted beam allocation framework that treats the beam management problem as a Markov Decision Process (MDP).

DRL Framework: The system utilizes the Proximal Policy Optimization (PPO) algorithm. PPO is chosen for its stability and ability to handle discrete action spaces through a clipped surrogate objective, preventing excessively large policy updates.
State Space (Observations): Instead of requiring explicit CSI, the agent observes:
1. Sensing-derived observations: The power of reflected echoes across all angular bins in the DFT codebook.
2. Beam history: The beam allocation from the previous time step.
3. Buffer status: The current number of packets in each user's buffer and the longest waiting time (head-of-line delay).
Action Space: To avoid the "curse of dimensionality" associated with large antenna arrays, the authors use a shrunk action space. Instead of selecting any beam index, the agent chooses from six relative actions (e.g., stay the same, sweep to a new area, or shift to an adjacent beam) for each user.
Reward Function: A multi-objective reward function is designed to balance the two modules:
$r_t = \sum_{u=1}^{U} (\text{Latency Reward}) - (\text{Buffer Overflow Penalty}) - (\text{Deadline Violation Penalty}) + (\text{Sensing Accuracy Reward})$
The sensing reward uses the logarithm of normalized estimation errors to prevent reward explosion.
Multi-Beam Strategy: The framework allows for allocating up to three beams per user. While spreading power across multiple beams may slightly increase latency, it broadens the Field of View (FoV), which helps mitigate beam drifting and improves sensing accuracy.

3. Key Contributions

Cross-Layer Formulation: Formulates a joint optimization problem that integrates physical-layer beamforming with MAC-layer queue dynamics.
CSI-Free Management: Proposes a method to eliminate explicit CSI feedback by leveraging sensing-derived observations (reflected echoes) to inform beam decisions.
Implicit Modeling: Uses DRL to implicitly capture the complex, non-linear interactions between buffer states, channel fading, and sensing precision without requiring explicit transition probability matrices.
Multi-Beam Optimization: Demonstrates that simultaneous multi-beam allocation is an effective strategy for maintaining connectivity in high-mobility/cluttered environments.

4. Results and Discussion

The proposed method was evaluated in a simulated 28 GHz mmWave environment with moving users and stationary clutter.

Convergence: PPO demonstrated superior stability and performance compared to other RL algorithms like DQN, A2C, and TD7.
Communication Performance: The multi-beam DRL approach (PPO-mb) achieved higher throughput and better latency management than heuristic AoD-based methods and standard TDMA schemes, especially in cluttered environments.
Sensing Performance: The DRL agent achieved range, velocity, and angle estimation accuracies close to the "Genie-aided" benchmark (which assumes perfect AoD knowledge).
Robustness: The system proved highly effective at distinguishing user signals from clutter, a task where traditional AoD-based methods failed due to estimation errors.
Complexity: The computational complexity for online deployment is low, as it only requires a forward pass through a relatively small neural network at each Transmission Time Interval (TTI).

5. Significance

This work advances the state-of-the-art in intelligent resource management for 6G ISAC systems. By turning the sensing module into a tool for communication assistance (rather than just a separate task), the paper provides a blueprint for highly efficient, low-overhead, and adaptive wireless networks capable of operating in complex, real-world dynamic environments.