🤖 machine learning

Surrogate Ensemble in Expensive Multi-Objective Optimization via Deep Q-Learning

This paper proposes SEEMOO, a reinforcement learning-assisted ensemble framework that utilizes a deep Q-network to dynamically select and schedule surrogate models within a single optimization process, thereby enhancing performance on expensive multi-objective optimization problems by eliminating human bias in model selection.

Original authors: Yuxin Wu, Hongshu Guo, Ting Huang, Yue-Jiao Gong, Zeyuan Ma

Published 2026-02-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuxin Wu, Hongshu Guo, Ting Huang, Yue-Jiao Gong, Zeyuan Ma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find the perfect recipe for a complex dish, but tasting the food is incredibly expensive and time-consuming. You only have enough ingredients for 550 taste tests before you run out. This is the challenge of Expensive Multi-Objective Optimization: finding the best balance between several conflicting goals (like taste, cost, and health) when you can't afford to test every possibility.

Usually, scientists use a "surrogate" (a cheap, fast guess) to predict how a recipe will taste before actually cooking it. However, there's a problem: different surrogates are good at guessing different things. One might be great at predicting sweet flavors but terrible at spicy ones. Traditionally, a human expert has to pick one surrogate to use for the whole process. If they pick the wrong one, the whole search fails.

Enter SEEMOO: Think of SEEMOO as a smart, learning sous-chef who doesn't just pick one tool; they manage a whole toolbox.

Here is how SEEMOO works, broken down into simple concepts:

1. The Toolbox (The Model Pool)

Instead of relying on a single guesser, SEEMOO keeps a "pool" of five different surrogates (like a Gaussian Process, a few types of Neural Networks, and others). Each one has a different style of guessing. Some are good at smooth, simple trends; others are good at complex, jagged patterns.

2. The Smart Manager (The Deep Q-Learning Agent)

This is the brain of the operation. Imagine a manager who watches the cooking process in real-time.

The State (What they see): The manager looks at a dashboard showing how the search is going. Are we getting closer to the goal? Is the population of recipes getting more similar? How accurate were the last few guesses?
The Action (What they do): Based on what they see, the manager picks the best tool from the toolbox for the next step. If the search is getting stuck in a complex area, they might switch to a tool good at handling complexity. If things are smooth, they might switch to a faster, simpler tool.
The Learning (The Reward): After the manager makes a choice, the system tests a few real recipes. If the new guesses lead to better results, the manager gets a "positive reward" (like a gold star). If the guesses were bad, they get a negative reward. Over time, the manager learns exactly which tool to use in which situation to get the best results with the fewest taste tests.

3. The Process (The Workflow)

The paper describes a two-level dance:

The Low Level (The Cook): The main algorithm (NSGA-II) tries to evolve better solutions using the predicted values from the chosen surrogate. It doesn't waste expensive real tests yet.
The High Level (The Manager): The AI manager watches the cook, picks the best surrogate for the next round, and learns from the results.

Why is this better?

The paper claims that human experts often make biased choices, picking a tool that works well for their specific problem but fails on others. SEEMOO removes this human bias. It learns a "universal" strategy.

The Analogy: Imagine a human chef who always uses a hammer to fix everything. If they encounter a screw, they fail. SEEMOO is like a chef who learns to look at the problem, realize it's a screw, and instantly grab a screwdriver. If it's a nail, they grab the hammer. They don't just stick to one tool; they adapt dynamically.

The Results

The researchers tested this on 24 different complex "recipes" (mathematical problems).

Performance: SEEMOO consistently found better solutions than using any single tool alone.
Generalization: Even when the AI was trained on a small set of problems and then tested on completely new, unseen problems, it still performed well. It didn't just memorize the answers; it learned how to choose tools.
Ablation Studies: When the researchers removed parts of the system (like taking away the "smart manager" or removing one of the tools from the toolbox), performance dropped. This proved that both the variety of tools and the AI's ability to switch between them are essential.

In short: SEEMOO is a system that uses Artificial Intelligence to act as a dynamic manager, constantly switching between different prediction models to solve complex, expensive problems more efficiently than any single model could on its own.

Technical Summary: Surrogate Ensemble in Expensive Multi-Objective Optimization via Deep Q-Learning (SEEMOO)

Problem Statement

Expensive Multi-Objective Optimization (EMOO) problems, prevalent in complex engineering and design scenarios, involve objective functions that require costly simulations or physical experiments. Traditional Evolutionary Multi-Objective Algorithms (MOEAs), such as NSGA-II and MOEA/D, struggle in these contexts due to the prohibitive cost of function evaluations. While Surrogate-Assisted Evolutionary Algorithms (SAEAs) mitigate this by using lightweight models (e.g., Gaussian Processes, Radial Basis Functions) to approximate objectives, their effectiveness is heavily dependent on the surrogate management strategy.

Existing SAEAs predominantly rely on static, human-defined surrogate models or fixed ensemble schemes. This introduces strong designer bias, limiting the algorithm's generalizability and robustness when applied to out-of-scope tasks or problems with varying landscape complexities. Furthermore, the integration of Meta-Black-Box Optimization (MetaBBO) techniques, specifically Reinforcement Learning (RL), into the surrogate selection process for multi-objective problems remains largely underexplored compared to single-objective domains.

Methodology: The SEEMOO Framework

To address these limitations, the authors propose SEEMOO, a Reinforcement Learning-assisted surrogate ensemble framework. SEEMOO operates within a bi-level structure, treating the optimization process as a sequential decision-making problem where a meta-level RL agent dynamically schedules surrogate models for a low-level evolutionary optimizer.

1. Low-Level Optimizer

The base optimizer is a surrogate-assisted NSGA-II. The process involves:

Initialization: Generating an initial population via Latin Hypercube Sampling (LHS) and evaluating them with true objective functions.
Evolution: At each step, a selected surrogate model is trained on the current database to predict objective values. The population evolves using these predictions (via tournament selection, crossover, and mutation) without incurring true evaluation costs.
Infilling: A batch of promising candidates is selected from the evolved population for true evaluation. These new samples update the training database and the external archive of non-dominated solutions.

2. Meta-Level RL Design

The core innovation lies in the dynamic selection of the surrogate model at each optimization step via a Deep Q-Network (DQN).

State Space ( $s_t$ ): To handle problems with varying numbers of objectives, an attention-based state extractor is employed. For each candidate model in the pool, an 8-dimensional feature vector is constructed, capturing:
- Usage history (cumulative usage, recent frequency).
- Performance metrics (IGD-based improvement, global step-wise improvement).
- Convergence dynamics (decision space and objective space density changes).
- Model-specific accuracy (surrogate error variation).
- Search progress (step ratio).
  These features are processed through a shared linear layer and a Multi-Head Attention mechanism to capture inter-surrogate relationships, followed by Layer Normalization.
Action Space: The agent selects one action from a pool of five distinct surrogate models:
- Gaussian Processes (GP)
- Multi-Layer Perceptrons (MLP1: low budget, MLP2: high budget)
- K-Nearest Neighbors (KNN)
- Radial Basis Function Networks (RBFN)
Reward Function: The reward ( $r_t$ ) is derived from the normalized improvement of the Inverted Generational Distance (IGD) of the external archive between consecutive steps. This provides a dense, continuous signal encouraging actions that improve the approximation of the Pareto front.
Training: The DQN is trained over a distribution of benchmark problems. The agent learns a policy to map the current optimization state to the most suitable surrogate model, maximizing long-term optimization performance.

Key Contributions

SEEMOO Framework: The proposal of a RL-assisted framework that enables dynamic surrogate model scheduling under limited evaluation budgets for expensive multi-objective optimization.
Adaptive Architecture: The integration of a diverse surrogate model pool, an attention-based state extractor for universal problem representation (supporting varying objective counts), and a DQN-based selector that adaptively chooses the best model at each step.
Empirical Validation: Comprehensive comparative studies and ablation analyses within the NSGA-II framework demonstrating that SEEMOO outperforms various individual static surrogate models and validating the necessity of its core design components.

Experimental Results

The framework was evaluated on a benchmark suite of 24 multi-objective problems (ZDT, DTLZ, WFG, OmniTest, Kursawe) with decision variables ranging from 2 to 80 and up to 5 objectives.

Performance: SEEMOO demonstrated generally superior performance across the ZDT, DTLZ, and WFG families compared to five static baseline models (GP, MLP1, MLP2, KNN, RBFN). It achieved notable IGD reductions, particularly on complex multimodal instances.
Generalization: Experiments conducted under two data-partition settings (3:1 and 1:3 training-to-testing ratios) showed that SEEMOO maintains stable performance even when trained on a limited subset of problems and tested on unseen domains. This suggests the learned meta-policy captures invariant optimization-state features rather than memorizing specific instances.
Ablation Studies:
- Surrogate Pool: Removing any specific surrogate model (e.g., GP, MLPs, KNN, RBFN) consistently degraded performance, confirming that a heterogeneous pool provides complementary information essential for the scheduling policy.
- Reward Design: The IGD-based reward outperformed both binary rewards (sparse feedback) and Hypervolume (HV)-based rewards, indicating that dense, smoothly varying feedback is critical for effective meta-learning.

Significance and Claims

The paper claims that SEEMOO effectively bridges the gap between static surrogate management and the dynamic needs of complex optimization landscapes. By automating the surrogate selection process through a learning-driven policy, the framework mitigates human designer bias and enhances adaptability.

The authors position SEEMOO as a promising direction for Meta-Black-Box Optimization, demonstrating that modeling surrogate management as a meta-RL task can overcome the limitations of static ensemble designs. The work validates that dynamic scheduling can boost optimization efficiency in expensive scenarios where evaluation budgets are strictly constrained. Future work is identified as expanding the surrogate pool, integrating with various MOEA architectures, and validating the framework on real-world engineering tasks.