🤖 machine learning

READY: Reward Discovery for Meta-Black-Box Optimization

This paper introduces READY, a framework that leverages Large Language Models with tailored evolution and multi-task architectures to automatically discover effective and efficient reward functions for Meta-Black-Box Optimization, thereby overcoming the design biases and risks associated with human-crafted rewards.

Original authors: Zechuan Huang, Zhiguang Cao, Hongshu Guo, Yue-Jiao Gong, Zeyuan Ma

Published 2026-01-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zechuan Huang, Zhiguang Cao, Hongshu Guo, Yue-Jiao Gong, Zeyuan Ma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to solve a complex puzzle, like finding the lowest point in a vast, foggy mountain range. The robot can't see the whole map; it only knows where it is right now and how high up it is. To learn, the robot needs a "coach" who gives it feedback after every move. This feedback is called a reward.

If the coach says "Good job!" when the robot moves in the wrong direction, the robot gets confused. If the coach is too harsh, the robot gives up. For years, humans have had to sit down and manually write these coaching rules (the reward functions) for different types of puzzles. This is slow, prone to human error, and often results in a coach that isn't very good.

READY is a new system that uses a super-smart AI (a Large Language Model) to act as a "coach designer." Instead of a human writing the rules, READY automatically invents, tests, and improves the coaching rules itself until it finds the perfect set of instructions for the robot.

Here is how READY works, using simple analogies:

1. The Problem: The "Human Coach" Bottleneck

Currently, if you want to build a new robot optimizer, you need a human expert to write the reward rules.

The Analogy: Imagine trying to train a chess player. If you have to write the rulebook for every single new chess variant by hand, it takes forever. Worse, you might accidentally write a rule that lets the player cheat (reward hacking) or a rule that is just too vague to be helpful.
The Issue: Humans are biased and slow. We can't easily test thousands of different rulebooks to see which one is truly the best.

2. The Solution: READY (The "AI Coach Designer")

READY uses an AI to automatically discover these reward rules. It treats the creation of a reward rule like a biological evolution process.

The "Niche" Concept (Specialized Teams):
READY doesn't just try to solve one puzzle at a time. It sets up several "teams" (called niches), each dedicated to a different type of optimization problem.
- Analogy: Imagine a gym with three different training groups: one for runners, one for swimmers, and one for weightlifters. Instead of having one coach try to train everyone at once, each group has its own coach. But, these coaches talk to each other to share tips.
The Evolution Process (Trial and Error):
Inside each team, the AI generates many different versions of reward rules. It tests them, sees which ones work best, and then "breeds" the best ones to create new, better versions.
- The "Mutation" (Debugging): If a rule fails on a specific tricky mountain, the AI analyzes why it failed (like a detective looking at a crime scene) and tweaks the rule to fix that specific weakness.
- The "Crossover" (Sharing Ideas): The AI takes a great idea from the "Swimming" team and tries to mix it into the "Running" team's rulebook. This helps all teams learn faster because they share their best tricks.

3. The Secret Sauce: How READY is Different

The paper highlights three special tricks READY uses to be better than previous methods:

Teamwork Across Tasks (Knowledge Transfer):
Most AI systems work in isolation. READY allows the different "teams" (solving different problems) to share their best discoveries. If the "Swimming" coach figures out a great way to handle slippery surfaces, the "Running" coach might use that same logic for muddy tracks. This speeds up the learning for everyone.
Deep Reflection (The "Think Before You Act" Step):
Before the AI changes a rule, it pauses to think. It looks at the failures and asks, "Why did this fail?" and "What worked in the past?" It doesn't just randomly change code; it uses logic to guide the changes, much like a human engineer debugging a complex machine.
Parallel Processing:
Because it runs multiple teams at once, READY finds good solutions much faster than systems that try to solve one problem after another.

4. The Results: What Happened?

The researchers tested READY on three different types of optimization robots (algorithms) using a standard set of difficult math puzzles.

Better Performance: The reward rules invented by READY helped the robots solve the puzzles much better than the rules written by human experts or other AI systems.
Generalization (The "Magic" Part): This is the most surprising finding. The researchers took a reward rule designed by READY for one specific robot and gave it to a completely different robot it had never seen before. Surprisingly, this "foreign" rule still worked incredibly well, often beating the new robot's original human-designed rules.
- Analogy: It's like taking a coaching manual written for a swimmer and giving it to a cyclist, and the cyclist suddenly becomes a world champion. This suggests READY found some "universal truths" about how to optimize things, rather than just memorizing one specific puzzle.

Summary

READY is a system that automates the creation of "coaching rules" for optimization algorithms. Instead of humans guessing what rules work best, READY uses an AI to evolve, test, and share the best rules across different problems. The result is a set of rules that are not only better than human-made ones but are also so smart that they can be transferred to new, unseen problems and still work perfectly.

The paper claims this makes the entire field of "Meta-Black-Box Optimization" (designing better optimizers) faster, more effective, and less dependent on human guesswork.

Technical Summary: READY: Reward Discovery for Meta-Black-Box Optimization

1. Problem Statement

Meta-Black-Box Optimization (MetaBBO) is a paradigm where a meta-level policy (typically a neural network trained via Reinforcement Learning) learns to design or configure a low-level Black-Box Optimization (BBO) algorithm to solve target problems. A critical bottleneck in MetaBBO is the reward function design. Currently, reward functions are manually crafted by human experts using simple heuristics. This manual dependency introduces design bias, risks "reward hacking," and severely limits scalability, as adapting to new algorithms or real-world scenarios requires laborious, case-specific tuning. Furthermore, suboptimal hand-crafted rewards obscure the true performance potential of different MetaBBO architectures, making fair comparisons difficult.

The paper posits that automating the discovery of reward functions is essential to unlock the full potential of MetaBBO. While Large Language Models (LLMs) have shown promise in automated program search (e.g., FunSearch, Eureka) and reward engineering for control tasks, their application to the complex, bi-level structure of MetaBBO remains unexplored.

2. Methodology: The READY Framework

The authors propose READY (Reward Discovery for Meta-Black-Box Optimization), a framework that utilizes LLMs as automated reward discovery tools. READY employs a multitask evolutionary paradigm to search for optimal reward code across diverse MetaBBO tasks simultaneously.

2.1. Core Architecture

READY treats reward discovery as a multitask optimization problem where the goal is to find a reward function $R$ that minimizes the performance cost $F$ for a set of $K$ MetaBBO tasks. The framework operates via an iterative evolutionary loop (Algorithm 1) involving:

Niche Initialization: Separate populations (niches) are initialized for each MetaBBO task.
Intra-niche Evolution: Evolutionary operators generate new reward candidates within each niche.
Inter-niche Knowledge Transfer: A mechanism to share successful heuristics across different tasks to accelerate convergence.

2.2. Key Components

A. Metadata Modeling & Initialization
To ground the LLM, READY constructs specific metadata ( $M_k$ ) for each task, comprising:

Algorithm Concept ( $C_{alg}$ ): A summary of the MetaBBO method's motivations and architecture derived from its research paper.
Programming Interface ( $C_{code}$ ): A structured dictionary of available variables, functions, and classes from the implementation source code to prevent hallucinations.
Greedy Sampling: The initial population includes a human-designed baseline and LLM-generated candidates that are strictly filtered to ensure they outperform the baseline and differ significantly in logic.

B. Diversified Evolutionary Operators
READY introduces five reflection-based code-level operators to ensure search diversity and continuous improvement:

M1 (Local-Reflection Mutation): Analyzes the top-K failure cases of a reward, generates a textual reflection on why it failed, and refines the code to address specific weaknesses.
M2 (History-Reflection Mutation): Acts as a trend analyst by reviewing the evolutionary trace of the past $L$ generations to extrapolate the next optimization step based on historical momentum.
M3 (Global-Reflection Mutation): Distills universal design principles from a global archive of eliminated rewards across all niches to guide the generation of new individuals.
C1 (Exploitative Crossover): Synthesizes the current individual with the local and global best solutions to integrate proven high-level logic while respecting local constraints.
C2 (Exploratory Crossover): Pairs the current individual with an elite from a different niche to transplant unique algorithmic traits, fostering cross-task diversity.

C. Knowledge Transfer (KT)
Recognizing that LLMs can function as multitask solvers, READY explicitly identifies similar task pairs. The LLM analyzes the source task's successful reward logic and generates a strategy to adapt it to the target task's constraints, replacing the worst individual in the target niche.

3. Key Contributions

The paper claims three primary contributions:

First LLM-Based Reward Design for MetaBBO: READY is the first framework to automate reward design specifically for the MetaBBO domain, moving beyond human-heuristic dependency.
Multitask Evolutionary Architecture: The framework introduces a niche-based multitask paradigm that facilitates knowledge sharing and search effectiveness, coupled with fine-grained reflection operators to ensure diversity.
Generalizability and Performance: The discovered rewards not only outperform existing baselines but also demonstrate the ability to generalize to unseen MetaBBO methods without fine-tuning.

4. Experimental Results

The authors evaluated READY on three distinct MetaBBO environments: RLDAS (Algorithm Selection), RLEPSO (Parameter Control), and DEDQN (Operator Selection), using the BBOB test suite.

Performance Superiority: READY consistently outperformed hand-crafted rewards and state-of-the-art baselines (Eureka, EoH, ReEvo). It achieved the lowest mean cost in 24 out of 48 test scenarios. Notably, READY trained a single set of rewards in one unified run to generalize across all three heterogeneous tasks, whereas baselines required independent training for each.
Zero-Shot Transfer: A significant finding was the generalizability of the discovered rewards. A reward evolved for DEDQN was directly applied to an unseen MetaBBO method (RLDEAFL), achieving up to 99.99% cost reduction on several functions, suggesting the discovery of universal, algorithm-agnostic heuristics.
Efficiency: By leveraging parallel evaluation and cross-niche knowledge sharing, READY achieved a 2x to 4x speedup in wall-clock time compared to sequential baseline execution.
Interpretability: The evolved rewards were found to be interpretable, synthesizing phase-adaptive mechanisms (e.g., shifting weights between exploration and exploitation based on search state) rather than relying on static heuristics.

5. Significance and Claims

The paper positions READY as a significant step toward fully autonomous optimization pipelines. Its significance lies in two main areas:

Advancing MetaBBO Research: By automating reward design, READY removes a major bottleneck, allowing researchers to focus on algorithmic architecture rather than manual reward tuning. This enables fairer performance comparisons and the unlocking of latent potential in existing MetaBBO methods.
Real-World Applicability: The framework's ability to discover robust, transferable reward functions suggests it can boost MetaBBO performance in complex, real-world applications where gradient information is unavailable and manual tuning is infeasible.

The authors conclude that READY demonstrates that LLMs can effectively capture universal search heuristics, paving the way for next-generation self-organized learning agents in the field of black-box optimization.