Multi-Agent Reinforcement Learning Counteracts Delayed CSI in Multi-Satellite Systems

This paper proposes a novel bi-level Multi-Agent Reinforcement Learning algorithm called Dual-Stage Proximal Policy Optimization (DS-PPO) to maximize user sum-rate in multi-satellite downlink systems by effectively mitigating the performance degradation caused by outdated Channel State Information due to high propagation delays.

The Gist

The Big Picture: The "Out-of-Date" Map Problem

Imagine you are trying to drive a high-speed race car through a foggy city. To drive fast, you need a perfect map of the road ahead. But here's the catch: your map is always 5 seconds old.

In the world of satellite internet (like Starlink), this is exactly what happens. Satellites zoom around the Earth at 17,500 mph. They try to send data to your phone or laptop. To send data efficiently, they need to know exactly how the signal is traveling through the air (this is called Channel State Information or CSI).

However, because the satellites are so far away and moving so fast, by the time they calculate the map of the road, the road has already changed. The "map" is outdated. If they drive based on an old map, they crash into interference or send the signal in the wrong direction, slowing down the internet.

The Solution: A Team of Coordinated Drivers (Multi-Agent RL)

The authors propose a solution using Multi-Agent Reinforcement Learning (MARL).

Think of the satellites not as individual drivers, but as a team of race cars working together. Instead of one driver trying to guess the whole track, every satellite is an "agent" that learns by trial and error. They talk to each other to figure out the best way to send data, even if their maps are a few seconds old.

The Secret Sauce: The "Two-Stage" Dance (DS-PPO)

The paper introduces a new algorithm called DS-PPO (Dual-Stage Proximal Policy Optimization). To understand this, imagine a dance routine with two distinct parts:

Stage 1: The Solo Practice
First, every satellite practices dancing on its own. It looks at its own old map and tries to figure out the best moves to send data to its specific users. It learns how to be a good soloist.

• The Trick: Instead of sharing its whole messy dance routine with everyone else (which would take too much time and bandwidth), it just shares a few key numbers (called "singular values"). Think of this as sharing the "rhythm" or the "beat" of the dance, rather than every single step.

Stage 2: The Group Performance
Now, the satellites come together. They listen to the "rhythms" shared by their neighbors. Using this shared beat, they adjust their own moves to dance in perfect harmony with the whole group.

• The Result: Even though they are all looking at slightly different, outdated maps, they coordinate their movements so perfectly that they act like one giant, super-powerful antenna. This creates a "distributed MIMO" system (a fancy way of saying many small antennas acting as one big one).

Why This is Better Than Old Methods

• Old Way (Channel Prediction): Previous methods tried to build a crystal ball to predict what the road will look like in the future. This is hard and often wrong.
• The Paper's Way: This method says, "Forget predicting the future. Just learn to drive well despite the old map." It skips the prediction step entirely and goes straight to finding the best action based on the imperfect information it has.

The Results: Fast, Strong, and Smart

The authors tested this "Two-Stage Dance" in a simulation with hundreds of satellites. Here is what they found:

1. It's Robust: Even with the "outdated map" (delayed data), the system performed almost as well as if they had a perfect, real-time map.
2. It's Fast: They achieved internet speeds of around 350 Mbps, which is very fast for satellite internet.
3. It's Efficient: The algorithm is "lightweight." It doesn't require a supercomputer on every satellite; it's smart enough to run on standard hardware.
4. The Sweet Spot: They found that having 6 satellites working together was the "Goldilocks" zone.
   • Too few (4 satellites)? Not enough power.
   • Too many (8 satellites)? The team got too confused by the complexity, and performance actually dropped. It's like a choir: 6 singers harmonize beautifully; 50 singers might start talking over each other.

Summary Analogy

Imagine a group of musicians trying to play a symphony, but they are all in different rooms with a 5-second delay in their earpieces. They can't hear the conductor perfectly.

• Old Method: They try to guess what the conductor will say next.
• DS-PPO Method: Each musician first practices their own part (Stage 1). Then, they share a simple "tempo" signal with the group (Stage 2). Using that shared tempo, they all adjust their playing in real-time to stay in sync, creating a beautiful song despite the delay.

In short: The paper teaches satellites how to be a coordinated team that doesn't panic when their information is slightly late, resulting in faster, more reliable internet for everyone on Earth.

Technical Summary

1. Problem Statement

The paper addresses the critical challenge of outdated Channel State Information (CSI) in Low-Earth Orbit (LEO) satellite communication networks.

• The Core Issue: In satellite communications, the high propagation delay between terrestrial users and satellites causes a significant time lag between when Channel State Information (CSI) is estimated (via pilot sequences) and when it is used for transmission. By the time the data is transmitted, the channel state has changed, rendering the CSI "outdated" or "delayed."
• The Consequence: This delay leads to severe performance degradation in downlink transmission, particularly in cooperative multi-satellite systems where satellites act as a distributed Multiple-Input Multiple-Output (MIMO) base station.
• Limitations of Existing Solutions:
  • Traditional convex optimization fails due to the high statistical uncertainty and non-stationary nature of rapidly time-varying channels at high frequencies (>1 GHz).
  • Existing Deep Learning (DL) approaches often rely on channel prediction, which adds complexity and may not accurately model the specific error distributions of high-mobility LEO scenarios.
  • Standard Multi-Agent Reinforcement Learning (MARL) algorithms (e.g., MADDPG, QMIX) struggle because they assume centralized training or stationary environments, which do not fit the non-IID (Independent and Non-Identically Distributed) nature of individual satellite channels and the massive continuous action space required for precoding.

2. Methodology: The DS-PPO Algorithm

The authors propose a novel Dual-Stage Proximal Policy Optimization (DS-PPO) algorithm. This is a Multi-Agent Reinforcement Learning (MARL) framework designed to directly map delayed CSI to an optimized Transmit Precoding Matrix (TPM) without explicit channel prediction.

Key Architectural Components:

• Augmented Markov Decision Process (MDP): To handle the constant delay in observations, the state space is augmented to include the delayed CSI and the sequence of actions taken during the delay period. This allows the agent to reconstruct the current state context.
• Bi-Level Optimization (Two Stages):
  1. Stage 1 (Individual Optimization): Each satellite acts as an independent agent. Using PPO, it optimizes its local TPM to maximize its own individual sum-rate based on its specific (delayed) CSI.
     • Output: The singular values of the individual TPMs are extracted.
  2. Stage 2 (Cooperative Optimization): The satellites share the singular values from Stage 1 (a compact representation of transmission characteristics) rather than full CSI. A second PPO agent uses these shared singular values and the delayed CSI to optimize the TPM for the global sum-rate, effectively treating the cluster as a distributed MIMO system.
• Information Exchange: By sharing only singular values instead of full channel matrices, the algorithm reduces inter-satellite communication overhead while enabling agents to track power allocation patterns across the cluster.
• Reward Function:
  • Stage 1: Quantized reward based on sum-rate thresholds, improvement over the previous step, and a penalty for violating power constraints.
  • Stage 2: Logarithmic reward based on the global cluster sum-rate and power constraints to ensure stability.

3. Key Contributions

1. Direct Mapping of Delayed CSI: Unlike previous works that rely on channel prediction, this approach directly maps delayed CSI to the optimal TPM, bypassing the prediction step entirely.
2. Novel DS-PPO Algorithm: The introduction of a bi-level optimization framework specifically tailored for cooperative multi-satellite systems with non-IID environments. It effectively handles large continuous action spaces and distributed learning constraints.
3. Theoretical Analysis: The paper provides a rigorous convergence analysis proving that the Stage 2 policy improves upon the Stage 1 baseline (providing a lower bound on performance improvement) and analyzes the computational complexity, demonstrating that DS-PPO is a lightweight algorithm.
4. Robustness to High Mobility: The solution is designed for high-frequency scenarios (>1 GHz) and high-mobility LEO constellations where statistical modeling of channel errors is infeasible.

4. Numerical Results

The authors evaluated DS-PPO using a simulation based on the Starlink constellation (4236 satellites, 9 antennas per satellite, 2 GHz frequency).

• Robustness to Delay: DS-PPO demonstrated remarkable resilience to CSI delays ($T_d = 1$ and $T_d = 3$ time steps). The performance gap between perfect CSI and delayed CSI scenarios was negligible.
• Sum-Rate Performance:
  • The algorithm achieved a minimum guaranteed sum-rate of 300 Mbps and an average of 350 Mbps in cooperative scenarios.
  • It outperformed a baseline Independent PPO (IPPO) by over 75% in sum-rate.
  • It significantly outperformed traditional channel prediction methods (SatCP + [10] method), achieving roughly 3.5x higher sum-rate (350 Mbps vs. ~100 Mbps).
• Satellite Count Scaling:
  • Increasing satellites from 4 to 6 improved sum-rate due to diversity gains.
  • However, increasing to 8 satellites caused a 25% drop in performance, indicating that the algorithm hits a complexity ceiling where the non-IID environment becomes too difficult for the agents to coordinate effectively without further tuning.
• Complexity: The computational complexity is dominated by neural network training (forward/backward passes), with Singular Value Decomposition (SVD) contributing less than 1% of the total FLOPS.

5. Significance

This paper presents a significant advancement in Non-Terrestrial Networks (NTN) by solving the "outdated CSI" problem in a practical, distributed manner.

• Practicality: It eliminates the need for a high-performance central controller or complex channel prediction models, distributing the processing load across satellites.
• Scalability: The use of singular values for information exchange makes the system scalable for large constellations, reducing bandwidth requirements for inter-satellite links.
• Future-Proofing: The approach is specifically tailored for the high-frequency, high-mobility environments expected in next-generation (6G) satellite networks, offering a robust alternative to traditional optimization methods that fail under high uncertainty.

In conclusion, DS-PPO offers a lightweight, robust, and high-performance solution for maximizing throughput in cooperative LEO satellite systems, effectively mitigating the detrimental effects of propagation delays on channel state information.