A Self-Evolving Default Action for Cooperative Tasks with Continuous Action Space

Technical Summary: SAFE (Self-Evolving Default Action From Experiences)

Problem Statement
Multi-agent reinforcement learning (MARL) has achieved success in discrete action spaces (e.g., StarCraft II) but faces significant hurdles in real-world continuous control tasks, such as autonomous vehicular networking. Existing approaches encounter three primary challenges:

1. Continuous Action Spaces: Many credit assignment schemes are designed for discrete spaces and cannot be directly applied to continuous control.
2. Decentralization Requirements: Real-world applications often require fully decentralized policies for safety and latency, yet many scalable methods rely on centralized inputs that do not scale well with agent numbers.
3. Credit Assignment in Continuous Domains: While Counterfactual Multi-Agent Policy Gradients (COMA) effectively assign credit in discrete spaces by marginalizing out an agent's action, extending this to continuous spaces via Monte Carlo sampling introduces bias. The sampled actions in continuous spaces are often undertrained, meaning the centralized critic has not sufficiently learned their values. Consequently, the baseline expectation is non-zero under the joint policy, leading to biased policy gradients and a lack of convergence guarantees to local optima.

Methodology
The paper proposes SAFE (Self-Evolving Default Action From Experiences), a novel MARL framework designed to address these limitations within the Centralized Training with Decentralized Execution (CTDE) paradigm.

• Core Mechanism: SAFE introduces a counterfactual baseline conditioned on a self-evolving default action. Unlike prior methods that rely on user-defined default actions or additional simulations, SAFE samples this default action ($\bar{a}_i$) directly from each agent's experience replay buffer ($D_t$).
• Baseline Construction: For a given agent $i$, the baseline $b_i$ is calculated by replacing the agent's actual action $a_i$ with a sampled default action $\bar{a}_i(t)$ in the centralized critic's input, while keeping other agents' actions ($a_{-i}$) constant:
  $$b_i = Q(s, a_{-i}, \bar{a}_i(t)), \quad \bar{a}_i(t) \sim D_t$$
• Self-Evolution: As training progresses, the sampled default action naturally evolves to reflect the agent's average performance. The distribution of sampled actions in the buffer shifts toward the actions most frequently executed by the agent, ensuring the baseline is derived from sufficiently trained values.
• Theoretical Guarantee: The authors prove mathematically (Lemma 0.1) that this baseline introduces zero bias into the deterministic policy gradient. By treating the deterministic policy as a limit case of a stochastic policy, they demonstrate that the expectation of the baseline term vanishes, ensuring convergence to local optima.
• Architecture: The framework utilizes a centralized critic conditioned on the global state and joint action-observation histories, while decentralized actors condition only on their local histories. Parameter sharing is employed among actors for cooperative roles.

Key Contributions

1. Extension to Continuous Spaces: SAFE naturally extends counterfactual credit assignment to continuous action spaces without requiring additional simulations, reward models, or environment-specific priors.
2. Bias-Free Convergence: The paper provides a theoretical proof that the self-evolving default action baseline does not introduce bias into deterministic policy gradients, guaranteeing convergence to local optima—a property often lacking in naive Monte Carlo extensions.
3. Dynamic Adaptation: The baseline adapts dynamically as agent behavior evolves, with the sampled default action increasingly reflecting the agent's average performance during training.
4. Empirical Validation: Extensive experiments on cooperative autonomous driving tasks (Highway-Env) demonstrate that SAFE consistently outperforms state-of-the-art models (VDN, QMIX, IQL, COMA, MAPPO) across scenarios with varying numbers of vehicles and obstacles.

Experimental Results

• Performance: In cooperative highway driving scenarios (2V1O to 7V2O), SAFE achieved superior performance compared to benchmarks. Discrete-action models (VDN, QMIX, IQL) failed to avoid collisions in complex scenarios due to the coarse granularity of discrete meta-actions. COMA and MAPPO failed to converge to optimal policies, largely attributed to sample inefficiency in on-policy training.
• Ablation Studies:
  • Credit Assignment vs. Continuity: Comparisons with baselines that adapt to continuous spaces but use different credit assignment schemes (e.g., Centralized_critic, COMA_cont) confirmed that SAFE's performance gains stem specifically from its credit assignment mechanism, not merely its ability to handle continuous actions.
  • Single vs. Multiple Samples: Parameter analysis revealed that using a single sampled default action (SAFE1) yields better results than averaging multiple samples (SAFE30 to SAFE100). Averaging multiple samples introduced additional biases and reduced estimation accuracy.
  • Default Action Source: Replacing the experience-sampled action with a fixed zero-action (SAFE_a=0) or a batch mean (SAFE_batch_mean) resulted in inferior performance, validating the necessity of the self-evolving, experience-based selection.
• Behavioral Analysis: Visualizations of the default action probability density showed that the sampled actions evolved over time to match the agents' learned strategies (e.g., shifting from neutral steering to specific avoidance maneuvers), confirming the baseline's alignment with the agent's actual policy.

Significance
The paper positions SAFE as an effective solution for real-world multi-agent systems requiring continuous control. By resolving the bias and convergence issues associated with counterfactual credit assignment in continuous domains, SAFE enables the deployment of scalable, decentralized cooperative policies in complex environments like autonomous vehicular networking. The work establishes that accurate credit assignment in continuous spaces can be achieved through experience-based self-evolution rather than external modeling or simulation.