Technical Summary: MAMO – A Multi-Agent System for Multi-Objective Constrained Optimization

1. Problem Statement

Many decision-making problems in computing and networking systems, particularly within the Computing Continuum (CC), can be formulated as cost-minimization problems subject to performance constraints (e.g., latency, throughput, or resource budgets). In dynamic environments characterized by fluctuating workloads and resource availability, Reinforcement Learning (RL) is frequently employed to solve these problems at runtime.

The standard approach to handling constraints in RL involves Lagrangian-inspired formulations, where constraint violations are aggregated with the primary cost into a single scalar reward using manually selected weighted penalty terms. This paper identifies a critical limitation in this paradigm: the behavior of the learned policy is heavily dependent on these fixed weights.

• The Trade-off Dilemma: Selecting appropriate weights is difficult and problem-specific. Incorrect weights lead to policies that are either overly conservative (prioritizing constraints at high cost) or overly aggressive (minimizing costs while frequently violating Quality-of-Service (QoS) requirements).
• Non-Stationarity: In dynamic environments, the relative importance of objectives may change over time, rendering static, hand-tuned weights suboptimal.

The paper proposes a solution to decouple task execution from the design of the objective function, moving away from manual parameter tuning toward an autonomous, adaptive mechanism.

2. Methodology: The MAMO Framework

The authors propose MAMO (Multi-Agent system for Multi-Objective constrained optimization), a hierarchical multi-agent framework designed to learn the trade-off between conflicting objectives. MAMO treats the selection of reward-weighting coefficients as a learning problem rather than a fixed design choice.

Architecture

MAMO consists of two agents operating at different time scales and abstraction levels:

1. Task-Execution (TE) Agent:

   • Role: Interacts directly with the environment to learn a control policy.
   • Mechanism: Uses a standard weighted reward function that aggregates the primary cost and constraint-violation penalties.
   • Behavior: For a fixed set of weights, the TE agent behaves as a conventional RL agent, optimizing the composite objective.

2. Weight-Adaptation (WA) Agent:

   • Role: Operates at a higher level and a slower time scale. It does not act on the environment directly.
   • Mechanism: Selects the values of the weighting coefficients ($w$) that regulate the trade-off in the TE agent's reward function.
   • Learning Signal: The WA agent observes aggregated performance indicators (e.g., average execution cost and average rejection probability) produced by the TE agent.
     • If the rejection probability exceeds the tolerance threshold, the WA agent receives a reward of zero (penalizing constraint violations).
     • If constraints are satisfied, the reward corresponds to the execution cost (encouraging lower costs).
   • Objective: To learn a policy that maps performance summaries to new weight values, minimizing execution costs while enforcing constraints.

Workflow

The system operates in an iterative two-phase loop:

1. Weight Selection & TE Training: The WA agent selects a weight configuration (initially random) and fixes it for a training horizon. The TE agent interacts with the environment and updates its policy to optimize the weighted reward.
2. Evaluation & Adaptation: At the end of the horizon, the WA agent observes the TE agent's performance indicators. Based on this feedback, it selects a new weight configuration, and a new TE training phase begins.

This structure allows MAMO to progressively refine the balance between cost efficiency and QoS preservation directly from experience, without relying on manual tuning.

3. Reference Application: Edge-FaaS Replica Scaling

To illustrate the framework, the authors apply MAMO to a replica scaling problem in a Function-as-a-Service (FaaS) edge computing environment.

• Context: Edge nodes have limited resources (memory, compute) compared to the cloud. Applications are decomposed into stateless functions instantiated on demand.
• Objective: Minimize the cost of initializing and running function replicas (including cold-start overheads) while ensuring the probability of request rejection remains below a specific tolerance ($tol$).
• Challenge: Workloads are non-stationary (e.g., diurnal patterns), and resource availability fluctuates due to co-located services. Accurate modeling of the joint effect of these factors on rejection probability is difficult, making model-free RL suitable.

4. Experimental Analysis

The authors evaluated MAMO in a simplified instance involving a single function with a sinusoidal workload trace (emulating diurnal patterns) and added uniform noise to simulate non-stationarity.

Experimental Setup

• Baseline: The problem was solved offline using the Gurobi optimizer to establish a lower bound on performance under perfect knowledge.
• Control Cases: The TE agent was trained with fixed weights ($w=0.99$ and $w=0.1$) to demonstrate extreme behaviors (over-provisioning vs. under-provisioning).
• MAMO Configuration:
  • TE Agent: Deep Q-Learning with a three-layer fully connected network.
  • WA Agent: Discretized action space (step of 0.01) for weight selection.
  • Training Cycle: TE trains for 15,000 iterations per weight; WA observes the last 300 steps of performance.
  • Constraint: Rejection probability must remain below 0.05.

Results

• Convergence: As the WA agent trained, the observed rejection probability ($p$) approached the 0.05 tolerance threshold, while the weight $w$ converged to values between 0.8 and 0.9.
• Performance: Although the learned policy resulted in slightly higher execution costs (more replicas) compared to the offline optimal solution, it successfully adapted to the noisy workload.
• Key Finding: MAMO consistently kept the rejection probability below the 0.05 threshold, whereas fixed-weight approaches either violated constraints (low $w$) or incurred unnecessary costs (high $w$).

5. Key Contributions

1. Decoupling Execution from Objective Design: MAMO introduces a hierarchical architecture that separates the control policy (TE) from the reward design (WA), allowing the system to adapt its notion of "optimality" as conditions evolve.
2. Autonomous Weight Adaptation: Unlike traditional approaches where weights are manually tuned or optimized via gradient-based meta-learning, MAMO frames weight selection as a sequential decision problem solved by a separate agent.
3. Interpretability: The approach restricts adaptation to a low-dimensional vector of scalarization weights over pre-defined objectives, offering a more structured and interpretable alternative to free-form intrinsic reward models.
4. Model-Free Compatibility: The outer loop (WA agent) does not require differentiability through the inner learner (TE agent), making the framework compatible with arbitrary inner solvers.

6. Significance and Future Work

The paper claims that MAMO represents a "first step towards more autonomous and robust RL-based solutions" for constrained optimization in dynamic environments. The significance lies in its ability to handle the trade-off between conflicting goals without relying on static, hand-tuned parameters that may become obsolete as system dynamics change.

The authors remain modest regarding the scope of the current work, noting that the experimental analysis is based on a "simple use-case." Future research plans include:

• Evaluating MAMO on problems from different application domains.
• Comparing MAMO against other weight-selection strategies, such as dual-decomposition schemes, Bayesian optimization, and multi-policy algorithms (e.g., Optimistic Linear Support).