Temporal Action Representation Learning for Tactical Resource Control and Subsequent Maneuver Generation

1. Problem Statement

Autonomous robotic systems often operate under strict resource constraints (e.g., limited battery, ammunition, or computational budgets) in dynamic environments. The core challenge addressed is Tactical Decision-Making under Resource Limits within Hybrid Action Spaces.

• Hybrid Action Space: The problem involves a combination of discrete actions (resource usage, e.g., firing a missile, using a boost) and continuous actions (maneuvers, e.g., flight control, steering).
• Key Gaps in Existing Methods:
  1. Causal Dependency: Prior hybrid-action Reinforcement Learning (RL) methods often fail to capture the causal link between a discrete resource decision and its impact on the feasibility and optimality of subsequent continuous maneuvers.
  2. Multi-modality: Tactical scenarios often allow multiple valid future maneuvers for the same discrete decision depending on the context. Standard RL approaches tend to converge to a single dominant policy, ignoring this diversity.
  3. Temporal Coherence: Existing methods often treat actions as single-step events rather than temporally coupled sequences, failing to model long-horizon tactical dependencies.

2. Methodology: TART Framework

The authors propose TART (Temporal Action Representation learning for Tactical resource control and subsequent maneuver generation). TART learns a structured latent representation that unifies resource control and maneuver generation.

A. Core Objective: Mutual Information Maximization

TART aims to learn state-action representations that capture the predictive information linking the state history + current discrete decision to the future continuous maneuver sequence.

• Goal: Maximize Mutual Information (MI) between the context (state history $z_{t-K+1:t}$ and discrete action $u_{t,d}$) and the future continuous trajectory ($u_{t:t+K-1,c}$).
• Formulation: $J_{TART} = I(u_{t:t+K-1,c}; [z_{t-K+1:t}, u_{t,d}])$.
• Theoretical Guarantee: The authors prove that under standard assumptions, maximizing this objective leads to Joint $Q^*$-sufficiency, meaning the learned representations preserve enough information to evaluate optimal action values without loss.

B. Contrastive Learning with InfoNCE

Since direct MI computation is intractable, TART uses a tractable lower bound via InfoNCE loss:

• Context ($h_t$): Encoded from state history and the discrete action.
• Future ($p_t$): Encoded from the subsequent continuous maneuver sequence.
• Loss Function: The model is trained to align the context with the correct future (positive pair) while pushing it away from futures of other trajectories in the batch (negative pairs). This enforces temporal coherence.

C. Vector Quantization (VQ) for Tactical Modes

To handle multi-modality and create interpretable strategies, the continuous latent representations are quantized into a discrete Tactic Codebook ($\mathcal{C} = \{c_1, ..., c_M\}$).

• Process: The context embedding $h_t$ is mapped to the nearest codebook entry $c_{i_t}$ via $\ell_2$-nearest neighbor.
• Result: This creates a discrete "Tactic Mode" ($q_t$) that represents a recurring tactical pattern (e.g., "aggressive pursuit," "defensive evasion").

D. Factorized Hybrid Policy

The learned tactical code $q_t$ conditions the policy network, which is factorized into two actors:

1. Discrete Actor ($\pi_d$): Selects the resource action (e.g., fire missile) based on the state.
2. Continuous Actor ($\pi_c$): Generates the continuous maneuver based on the state, the discrete action, and the quantized tactic code $q_t$.

• Training: The framework uses an on-policy workflow (PPO) where the representation learning (contrastive + VQ) and policy optimization are interleaved.

3. Key Contributions

1. TART Framework: A novel representation learning framework specifically designed for hybrid action spaces that explicitly models the causal dependency between discrete resource usage and continuous maneuver generation.
2. Temporal Alignment & Multi-modality: The combination of contrastive learning (for temporal alignment) and vector quantization (for multi-modal clustering) allows the agent to learn diverse, context-aware tactical behaviors rather than a single deterministic policy.
3. Theoretical Foundation: Extension of $Q^*$-sufficiency notions to hybrid action settings, providing a theoretical justification for using these representations in value-based or actor-critic methods.
4. Empirical Validation: Extensive experiments in two distinct domains demonstrating superior performance over state-of-the-art baselines.

4. Experimental Results

The authors evaluated TART in two domains: Maze Navigation (limited boost/penetration actions) and High-Fidelity Air-to-Air Combat (F-16 with missiles/defense).

• Performance Metrics:

  • Maze Navigation: TART achieved the highest Success Rate (up to 97.2% on Easy, 72.8% on Hard) compared to baselines like HyAR, HPPO, and PDQN. It showed lower Time-to-Goal (TTG) and more efficient path coverage, particularly in "Hard" scenarios with dynamic obstacles where it effectively used the "Penetration" action to resolve deadlocks.
  • Air-to-Air Combat: TART outperformed all baselines across Easy, Medium, and Hard scenarios (Average Success Rate 87.2% vs. 77.4% for the best baseline). Crucially, it achieved these gains without wasting resources, maintaining lower or comparable Shots-per-Elimination (SPE) and Time-to-Elimination (TTE).

• Ablation Studies:

  • Removing the contrastive loss ($L_{NCE}$) degraded performance and increased variance, proving the necessity of temporal alignment.
  • Removing the vector quantization loss ($L_{VQ}$) reduced the ability to handle multi-modal behaviors, confirming the value of discrete tactical modes.

• Qualitative Analysis:

  • Heatmaps showed TART agents taking more direct paths and resolving deadlocks efficiently.
  • In combat, TART demonstrated context-aware coordination, such as firing a missile and immediately executing a specific evasive maneuver (causal dependency), or switching between offensive and defensive modes based on the tactical code.

5. Significance and Conclusion

This paper addresses a critical bottleneck in deploying autonomous robots in real-world tactical scenarios: the inability of current RL methods to effectively coordinate limited resource usage with complex, continuous maneuvers.

• Significance: TART demonstrates that learning temporally grounded, discrete tactical representations allows agents to reason about the consequences of resource consumption. It moves beyond simple action selection to strategic planning, enabling agents to be both resource-efficient and tactically flexible.
• Future Work: The authors plan to extend this to real-world hardware constraints (energy, bandwidth) and multi-agent/lifelong learning settings.

In summary, TART provides a robust solution for hybrid control problems by unifying discrete resource management and continuous control through a learned, multi-modal temporal representation, significantly outperforming existing hybrid-action RL baselines.