GUIDE: Guided Updates for In-context Decision Evolution in LLM-Driven Spacecraft Operations

1. Problem Statement

Spacecraft operations face unique challenges that limit the application of standard AI deployment strategies:

• High Cost & Risk: Missions cannot easily reset environments, collect massive new datasets, or retrain policies after deployment.
• Adversarial Uncertainty: Scenarios like autonomous Guidance, Navigation, and Control (GNC) and non-cooperative interactions (e.g., orbital interception) require reasoning under uncertainty rather than just low-level control.
• LLM Limitations: While Large Language Models (LLMs) offer promising supervisory reasoning, they face two coupled constraints in real-time closed-loop systems:
  1. Latency: Extended deliberation is too slow for real-time control loops, yet fast models lack deep reasoning capabilities.
  2. Adaptability: Standard LLMs rely on static prompting. They cannot adapt across episodes (cross-episode learning) without weight updates, which is impractical in space missions.

The core problem is how to enable an LLM agent to learn and adapt its decision-making strategy across multiple mission episodes in real-time without updating model weights, specifically in adversarial, dynamic environments.

2. Methodology: The GUIDE Framework

The authors propose GUIDE (Guided Updates for In-context Decision Evolution), a non-parametric policy improvement framework. Instead of optimizing neural network weights, GUIDE evolves a structured, state-conditioned playbook of natural language rules.

Core Architecture

The system employs a Teacher-Student separation:

1. Lightweight Acting Model (Student): Executes real-time control. It remains fixed (frozen weights) and uses a "playbook-conditioned policy" $\pi(a|s, P)$.
2. Frontier Reasoning Model (Teacher): Operates offline between episode batches. It performs reflection on past trajectories to update the playbook.

Key Components

• Playbook Representation ($P$): A dynamic collection of natural language rules (bullets). Each rule $b_i$ consists of text and a state predicate (condition). Rules are injected into the LLM's context only when their conditions (e.g., distance thresholds) are met.
• Online Decision Loop:
  • The acting model receives a fixed baseline prompt + active playbook rules.
  • It performs a two-step call:
    1. Reasoning Agent: Generates short-horizon intent based on the current state and active rules.
    2. Actor Agent: Maps the intent to a structured control output (thrust vector) via a function call, ensuring strict format compliance.
• Offline $\epsilon$-Biased Reflection:
  • After a batch of episodes, a meta-reasoning LLM (Reflector) analyzes trajectories.
  • Selection Strategy: Uses an $\epsilon$-rule to select trajectories for reflection: $1-\epsilon$ probability of selecting failed trajectories (to correct errors) and $\epsilon$ probability of selecting successful ones (to reinforce good strategies).
  • Evolution: A Curator model converts reflections into structured operations: ADD, UPDATE, or REMOVE rules in the playbook.
  • Selection: Multiple playbook versions are maintained. An Upper Confidence Bound (UCB1) algorithm selects the best version for the next batch, balancing exploration of new rules with exploitation of known good rules.

3. Experimental Setup

• Environment: Kerbal Space Program Differential Games (KSPDG).
• Task: Lady-Bandit-Guard (LBG) interception scenario.
  • Agent (Bandit): Must minimize distance to the "Lady" spacecraft.
  • Adversary (Guard): Attempts to intercept/capture the Bandit.
  • Scenarios: Four distinct Guard behaviors (LG4–LG7) ranging from aggressive pursuit to defensive positioning and stochastic switching.
• Baselines:
  • Static LLM (v0): Same model, no playbook evolution.
  • LQR (Linear Quadratic Regulator): Classical control.
  • Prograde-Alignment: Simple heuristic.

4. Key Results

The study evaluated performance using a composite score $S$ (lower is better), combining the minimum distance to the Lady and the minimum distance to the Guard.

• Performance Gains: GUIDE significantly outperformed the static LLM baseline across all scenarios.
  • LG4 (Aggressive Guard): Score reduced from $4.15 \times 10^5$ to $7.22 \times 10^4$ (p=0.043).
  • LG6 & LG7 (Defensive Guard): Massive improvements, with scores dropping from $\sim 10^7$ to $\sim 10^4$ (p < 0.001).
  • LG5 (Stochastic): Showed a positive trend (non-significant due to high variance in the baseline).
• Comparison to Classical Control: GUIDE outperformed LQR in most scenarios, particularly where strategic maneuvering (conditional evasion) was required over simple trajectory tracking.
• Evolution Mechanism:
  • The first playbook update consistently identified a specific failure mode: the Bandit pursued the Lady while ignoring the closing Guard.
  • The evolved playbook introduced guard-avoidance rules (e.g., "suspend pursuit if Guard is within X distance, apply lateral thrust").
  • This resulted in an 82–99% reduction in the composite score within a few evolutions.
  • Metric Analysis: GUIDE achieved much closer proximity to the Lady (e.g., 28m vs 237m in LG4) while maintaining safe separation from the Guard.

5. Key Contributions

1. Non-Parametric Policy Search: Demonstrates that policy improvement in LLM agents can be achieved by evolving structured context (playbooks) rather than updating model weights, making it suitable for deployment-constrained environments.
2. Teacher-Student Architecture: Proposes a practical separation where a heavy model handles offline reflection and a lightweight model handles real-time control, solving the latency vs. reasoning trade-off.
3. Cross-Episode Adaptation: Shows that LLMs can adapt to unknown adversarial dynamics (e.g., switching Guard strategies) purely through in-context learning and rule evolution.
4. Real-World Applicability: Validates the approach in a high-fidelity orbital dynamics environment (KSPDG), proving viability for spacecraft operations where retraining is impossible.

6. Significance

This work bridges the gap between operator-level reasoning and real-time control in space missions. It challenges the notion that LLMs are too slow or static for closed-loop control by showing that context evolution acts as a form of policy search.

The significance lies in its potential to enable autonomous spacecraft that can:

• Learn from experience without internet connectivity or ground intervention.
• Adapt to novel, adversarial threats in real-time.
• Operate safely in high-stakes environments where "trial and error" retraining is not an option.

The paper suggests a paradigm shift where the "policy" is not a set of weights, but a living, evolving set of natural language instructions that the agent refines over its operational lifetime.