Learning-Based Planning for Improving Science Return of Earth Observation Satellites

1. Problem Definition

Earth observation satellites face significant constraints in data collection, including fixed orbital trajectories, limited sensor fields of view, and strict energy budgets. Operating sensors consumes substantial power, necessitating that satellites prioritize the most scientifically valuable measurements rather than collecting data indiscriminately.

The specific problem addressed is Dynamic Targeting: the challenge of intelligently reconfiguring and pointing a primary instrument (e.g., a radar) based on data from a "lookahead" sensor. The goal is to maximize the scientific reward (information gathered) while adhering to energy constraints (battery state of charge).

• Context: The satellite has a primary sensor (narrow field of view) and a lookahead sensor (wider field of view) that reveals the environment ahead on the orbital path.
• Objective: Determine when and where to sample to capture high-value targets (e.g., clear skies or specific storm types) while avoiding low-value targets (e.g., clouds) and managing battery life.

2. Methodology

The authors model the dynamic targeting problem as a Markov Decision Process (MDP) and propose two learning-based approaches that leverage Dynamic Programming (DP) as a foundational concept.

A. Baseline and Benchmark

• Heuristics: The study compares new methods against existing strategies: Random, Greedy Nadir, Greedy Lateral, Greedy Radar, and Greedy Window.
• Oracle (Upper Bound): A standard Dynamic Programming (DP) algorithm is used as an "oracle." It assumes full knowledge of the entire future path and calculates the optimal sequence of actions. While computationally too expensive for real-time onboard use, it serves as the performance ceiling.

B. Approach 1: Reinforcement Learning (Q-Learning)

• Algorithm: A model-free Q-learning algorithm is used.
• State Representation: To manage the state space size, the authors do not use raw images. Instead, they construct a feature vector containing:
  1. State of Charge (SOC) (0–100%).
  2. Six binary variables representing the presence of three target types (e.g., clear, mid-cloud, cloud) in both the primary and lookahead sensor ranges.
  • Total States: $101 \times 2^6 = 6,464$ possible states.
• Action Space: Binary decision: Sample (1) or Do Not Sample (0). If sampling, the algorithm targets the highest-value object closest to nadir.
• Training Innovation: Standard Q-learning struggled to visit all states with limited data. The authors introduced a backward iteration strategy derived from DP. Instead of simulating forward, the algorithm iterates backward through the training data, varying the SOC from 0% to 100% for every image. This ensures the Q-table "experiences" every possible state-action pair within the training set, significantly improving generalization.

C. Approach 2: Imitation Learning (Behavioral Cloning)

• Concept: The DP algorithm acts as the "expert." The goal is to learn a policy that mimics the expert's decisions.
• Algorithm: Behavioral Cloning using a Multilayer Perceptron (MLP) neural network.
• Input/Output:
  • Input: A normalized feature vector (13 dimensions) containing more granular information than the Q-learning state.
  • Output: A probability (0–1) of triggering an observation (stochastic action).
• Training: The network is trained via supervised learning on state-action pairs generated by the DP oracle. The loss function minimizes the difference between the network's predicted action and the expert's action.

3. Experimental Setup

• Scenarios:
  1. Cloud Avoidance: Using MODIS data. Rewards: Clear (100), Mid-cloud (10), Cloud (1), Off (0).
  2. Storm Hunting: Using GPM data. Rewards: Convective Core (100), Rainy Anvil (10), No Storm (1), Off (0).
• Simulation: A framework simulating a satellite with a 15° primary sensor and a 45° lookahead sensor. Sampling costs 5% battery; recharging provides 1% per step.
• Data: 39 MODIS datasets and 10 GPM datasets. Half used for training, half for testing.

4. Key Results

A. Data Efficiency

Both learning methods proved highly data-efficient.

• Q-Learning: Performance plateaued after training on 3,000 images (Storm Hunting) and 20,000 images (Cloud Avoidance). Using the full dataset (86,400 images) did not significantly improve performance and, in some cases, led to overfitting.
• Imitation Learning: Converged quickly using only 0.34% to 2.41% of the total possible state space generated by the DP expert.

B. Performance Comparison

The learning methods significantly outperformed all heuristic baselines and approached the DP "Oracle" closely.

Method	Cloud Avoidance (% of Optimal Reward)	Storm Hunting (% of Optimal Reward)
Random	25.91%	14.12%
Greedy Window (Best Heuristic)	87.50%	82.57%
Behavioral Cloning	95.84%	91.27%
Q-Learning	98.67%	94.66%
Oracle (DP)	100.00%	100.00%

• Improvement: Imitation learning performed ~10.0% better than the best heuristic, while Q-learning performed ~13.7% better.
• Sampling Behavior: The learning algorithms successfully prioritized high-reward targets (Clear skies/Convective cores) and minimized sampling of low-reward targets, whereas heuristics often sampled indiscriminately.
• Runtime: Both learning methods execute in microseconds, making them suitable for real-time onboard processing (unlike the DP oracle).

5. Key Contributions

1. Novel Application of Learning: Demonstrated that Reinforcement Learning (Q-learning) and Imitation Learning (Behavioral Cloning) are viable, superior alternatives to traditional heuristic planning for satellite dynamic targeting.
2. Backward Iteration Strategy: Introduced a specific training technique for Q-learning that leverages the ability to step backward in simulation time to ensure full state-space coverage, overcoming the exploration limitations of standard Q-learning with limited data.
3. Data Efficiency: Proved that high-performance policies can be trained on very small subsets of data (thousands of images), which is critical for satellite missions where training data generation is costly.
4. Performance Gap Closure: Showed that learning-based methods can achieve >94% of the theoretical optimal reward (DP), effectively bridging the gap between computationally expensive optimal planning and fast, sub-optimal heuristics.

6. Significance and Future Work

This work addresses a critical bottleneck in Earth observation: the trade-off between data volume and scientific value. By enabling satellites to autonomously select high-value targets with minimal computational overhead and training data, these methods can significantly increase the scientific return of future missions (e.g., storm tracking, cloud sensing).

Future Directions:

• Moving from manually engineered state vectors to raw image inputs (using Deep Q-Networks or PPO) to preserve spatial information.
• Incorporating more realistic constraints, such as the time and energy required to physically move the sensor.
• Flight Demonstration: Deploying these algorithms on small satellite platforms with flight processors capable of running deep learning models (e.g., ESA's Φ-Sat or OPS-SAT).