Technical Summary: Adaptive Smooth Tchebycheff Attention for Multi-Objective Policy Optimization

Problem Statement

Deep Reinforcement Learning (RL) in robotic domains often requires balancing multiple, conflicting objectives (e.g., speed vs. safety, search coverage vs. stealth). While standard RL relies on scalar rewards, Multi-Objective Reinforcement Learning (MORL) addresses scenarios where the environment returns a vector of rewards. The goal is to find policies on the Pareto front, where no objective can be improved without degrading another.

Existing approaches face a fundamental trade-off between optimization stability and geometric precision:

1. Linear Scalarization (LS): Combines objectives via a weighted sum. While stable, it is theoretically incapable of recovering solutions in the non-convex regions of the Pareto front, leading to suboptimal policies in complex dynamic environments.
2. Tchebycheff (TCH) Scalarization: Can theoretically recover all Pareto-optimal solutions regardless of convexity by minimizing the maximum weighted deviation from an ideal "utopia" point. However, the non-differentiable max operator causes gradient instability and sparse updates in deep RL.
3. Smooth Tchebycheff (STCH): Replaces the max operator with a Log-Sum-Exp approximation controlled by a smoothness parameter $\mu$. While this enables gradient flow, it introduces a critical bottleneck:
   • Small $\mu$: Approaches the true TCH function but causes gradient starvation, where only the dominant objective receives updates, leading to catastrophic forgetting of other objectives.
   • Large $\mu$: Ensures smooth gradients but reverts the method to a linear scalarization, losing the ability to navigate non-convex regions.
   • Static Tuning: A fixed $\mu$ is difficult to tune because the RL "dataset" (agent experience) is non-stationary; the scale of advantages and the proximity to the utopia point shift as the policy evolves.

Methodology: PASTA

The authors propose PASTA (Policy-optimization via Adaptive Smooth Tchebycheff Attention), a framework that dynamically modulates the curvature of the optimization landscape to resolve the tension between stability and precision.

1. Adaptive Smooth Tchebycheff (ASTCH)

At the core of PASTA is the ASTCH scalarization, which dynamically adjusts the smoothness parameter $\mu$ based on real-time gradient interference.

• Conflict-Driven Controller: The system monitors the Conflict Ratio ($\kappa$), defined as the fraction of sampled objective gradient pairs that exhibit negative inner products (indicating destructive interference).
• Elastic Adaptation:
  • Annealing: As training progresses and conflicts are resolved, $\mu$ decays toward a precise, low value to sharpen the scalarization, allowing the agent to converge to non-convex Pareto solutions.
  • Dynamic Braking: If the conflict ratio exceeds a threshold (indicating unstable optimization), the controller temporarily increases $\mu$. This "brakes" the process, reverting to a smoother, more linear-like approximation to stabilize the policy update and prevent gradient starvation.
  • Inertial Smoothing: An Exponential Moving Average (EMA) is applied to $\mu$ to prevent rapid oscillations that could violate the stationary distribution assumptions of on-policy algorithms.

2. Attention-Aligned Branched Critic

To ensure the value function learns the correct priorities, PASTA introduces a Branched Critic architecture with a shared feature extractor and independent value heads for each objective.

• STCH Attention Mechanism: The gradient of the STCH function naturally induces a normalized, positive weighting ($\delta$) over objectives. Objectives with larger deviations from the utopia point receive exponentially higher attention.
• Maintenance Rate ($\rho$): To counteract gradient starvation in low-$\mu$ regimes, a maintenance rate is introduced. The effective attention weights are a blend of the STCH-derived attention and a uniform distribution, guaranteeing that every objective receives a minimum gradient budget.
• Loss Alignment: The critic's loss is weighted by these attention coefficients, ensuring that value estimation capacity is allocated to the objectives currently acting as bottlenecks.

3. Conflict-Aware Gradient Projection

To further mitigate destructive interference, PASTA integrates a variant of Projecting Conflicting Gradients (PCGrad).

• Instead of scalarizing advantages immediately, the algorithm maintains a vector of advantages.
• Policy gradients for individual objectives are computed separately.
• If a conflict is detected between two objective gradients, one is projected onto the normal plane of the other. This allows the agent to respect individual trust-region constraints while navigating the geometric conflicts inherent in non-convex Pareto fronts.

Key Contributions

1. ASTCH Framework: A novel scalarization method that dynamically adjusts the smoothness parameter $\mu$ based on an online estimate of inter-objective gradient conflict, eliminating the need for manual tuning and preventing gradient starvation.
2. PASTA Algorithm: An on-policy policy-gradient method that integrates ASTCH, a conflict-driven controller, and attention-aligned critics. It explicitly accounts for the non-stationarity of utopia points and value estimates in MORL.
3. Empirical Validation: Extensive evaluation on challenging real-world and simulated tasks, demonstrating robust discovery of Pareto-optimal policies in non-convex regions inaccessible to linear baselines and unstable for static non-linear methods.

Experimental Results

The authors evaluated PASTA across three distinct domains:

1. Stealth Visual Search (Real-World & Simulation):

   • Task: A ground robot must search for objects while minimizing exposure (stealth) and maximizing exploration speed.
   • Results: PASTA achieved a Hypervolume (HV) of 0.291, a 45.5% improvement over the strongest static baseline (STCH with $\mu=10.0$). It achieved a 100% Win Rate across diverse preference vectors, significantly outperforming Linear, Tchebycheff, and fixed-parameter STCH baselines.
   • Real-World Deployment: Validated on a Clearpath Jackal UGV in outdoor environments with uneven terrain and obstacles, successfully balancing search, stealth, and safety.

2. Multi-Objective MuJoCo Benchmarks:

   • Tasks: Continuous control tasks (HalfCheetah, Humanoid, Ant, Walker2D) with conflicting objectives.
   • Results: PASTA consistently achieved higher Hypervolume and better Pareto front coverage compared to baselines across all environments, demonstrating superior sample efficiency and robustness.

3. Quadrotor Navigation (Crazyflie):

   • Tasks: "Frogger" (navigating dynamic obstacles) and "Formation" (maintaining a triangle formation while avoiding obstacles).
   • Results: PASTA outperformed baselines in both HV and task success rates. In real-world flights, the agent achieved a 91.2% task success rate (52/57 objectives) without crashing, demonstrating the method's applicability to aerial platforms with stochastic dynamic obstacles.

Significance and Claims

The paper claims that PASTA provides a robust solution to the long-standing tension between optimization stability and geometric precision in model-free MORL. By treating gradient interference as a feedback signal, the method creates an "elastic" optimization landscape:

• It anneals toward precise, non-convex scalarization when objectives align, enabling the discovery of solutions in concave regions of the Pareto front.
• It elastically brakes into stable, smooth approximations when destructive gradient conflicts emerge, preventing catastrophic forgetting and training collapse.

The authors emphasize that this approach allows robots to autonomously resolve complex, conflicting requirements in unstructured real-world scenarios without the extensive hyperparameter tuning required by static scalarization methods. The work is presented as a step toward reliable, preference-specific multi-objective learning for robotics, though the authors note limitations regarding the interdependence of hyperparameters (conflict ratio vs. maintenance rate) and the focus on preference-specific rather than preference-conditioned policies.