⚡ electrical engineering

Language Models as Efficient Reward Function Searchers for Custom-Environment Multi-Objective Reinforcement

This paper proposes ERFSL, an efficient framework that leverages Large Language Models as white-box searchers to automatically generate and iteratively optimize multi-objective reward functions in complex custom environments through semantic understanding, a reward critic for code correction, and genetic-like weight adjustment strategies, achieving rapid convergence to Pareto-optimal solutions with minimal human feedback.

Original authors: Guanwen Xie, Jingzehua Xu, Yiyuan Yang, Yimian Ding, Shuai Zhang

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Guanwen Xie, Jingzehua Xu, Yiyuan Yang, Yimian Ding, Shuai Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a team of underwater robots (AUVs) to collect data efficiently. You have a list of rules for them: "Don't crash into each other," "Don't run out of battery," and "Don't let data pile up." In the world of Reinforcement Learning (RL), you have to write a "scorecard" (a reward function) that tells the robots how well they are doing based on these rules.

The problem is, writing this scorecard by hand is incredibly hard. If you get the math wrong, the robots crash. If you make the "don't crash" rule too strong, they stop moving entirely. If you make the "save battery" rule too strong, they move too slowly. Traditionally, humans have to guess and check, tweaking numbers over and over again until it works.

This paper introduces ERFSL, a new system that uses Large Language Models (LLMs)—the same kind of AI that writes essays and code—to act as a super-smart, efficient "scorecard designer."

Here is how it works, broken down into simple steps:

1. The "Architect" (Code Generator)

Instead of asking the AI to write the whole complex scorecard at once, the system breaks the job down. It asks the AI to write a small piece of code for each specific rule (e.g., just the code for "avoiding collisions," then just the code for "saving energy").

The Analogy: Imagine building a house. Instead of asking a contractor to build the whole thing in one go, you ask them to build just the kitchen, then just the bathroom, then just the bedroom.

2. The "Inspector" (Reward Critic)

Once the AI writes a piece of code, a second AI (the "Critic") acts like a strict building inspector. It looks at the code and the rules to see if it makes sense.

The Magic: If the AI wrote code that accidentally rewards the robots for crashing (a common mistake), the Critic spots it immediately. It says, "No, this is wrong," and fixes just that one piece of code.
The Result: The paper claims this "Inspector" is so good that it usually fixes any code errors in just one try, preventing the whole project from failing.

3. The "Tuner" (Weight Searcher)

Now that the code for each rule is correct, the system needs to decide how much each rule matters. This is the "weight." Should "not crashing" be worth 100 points, or 1,000? Should "saving energy" be worth 1 point or 10?

The Problem: Usually, finding the perfect balance takes thousands of guesses.
The Solution: The system uses a "Training Log Analyzer" to summarize the robots' performance into a simple story (e.g., "They crashed a lot, but saved energy"). The AI reads this story and acts like a geneticist or a chef tasting a soup.
- It doesn't just guess randomly. It uses Directional Mutation (like turning a dial up or down based on what happened) and Crossover (mixing the best settings from different attempts).
- The Analogy: Imagine tuning a radio. Instead of spinning the dial randomly until you find a station, the AI listens to the static, realizes "I'm too far left," and turns the dial specifically to the right.

4. The "Head Start" (Weight Initializer)

Before the tuning even begins, the system asks the AI to do a quick mental math calculation to guess a "good starting point" for the weights.

The Benefit: This ensures the AI doesn't start with a terrible guess (like making the "energy" rule 500 times too strong). Because of this head start, even if the starting point is way off, the system only needs about 5 to 6 adjustments to find the perfect balance.

Why is this special?

Zero-Shot Learning: The system works even if you give it no examples of how the robots should behave. It just reads your description and figures it out.
Efficiency: It found the perfect settings in very few tries (average of 5.2 iterations), whereas other methods might take dozens or hundreds.
Flexibility: It works well even with smaller, cheaper AI models (like GPT-4o mini) because the authors broke the big, hard math problem into smaller, easier storytelling problems.

In summary: This paper shows that by using AI to write small pieces of code, check them with a "critic," and then intelligently tune the numbers based on a summary of results, we can design complex robot behaviors much faster and more reliably than before. It turns a chaotic guessing game into a structured, efficient search.

Technical Summary: ERFSL – Efficient Reward Function Searcher for Multi-Objective RL

Problem Statement
Designing effective reward functions for Reinforcement Learning (RL) in complex, custom environments with multiple, often conflicting objectives presents significant challenges. As optimization goals proliferate, the manual adjustment of reward function structures and coefficients becomes labor-intensive and prone to ambiguity. Furthermore, researchers' needs often fluctuate, and requirements can be vague. While Large Language Models (LLMs) have shown promise in code generation and zero-shot tasks, they struggle with "black-box" optimization and balancing numerical weights in multi-objective scenarios. Existing approaches often rely on trial-and-error exploration, which is inefficient when dealing with complex reward functions where code correctness and weight balancing must be addressed simultaneously.

Methodology: The ERFSL Framework
The authors propose ERFSL (Efficient Reward Function Searcher using LLMs), a framework that treats LLMs as efficient white-box searchers by decomposing the reward design process into distinct stages. The framework operates without direct human feedback or reward examples (zero-shot learning) and consists of the following core components:

Task Decomposition and Description Refinement:
- User requirements are decomposed into numerically explicit performance objectives (e.g., "zero collisions," "data overflow reduced to 0").
- A meta-prompt is employed to review the natural language task description, identifying ambiguities (e.g., conflicting objectives) and suggesting clarifications to ensure the LLM receives a clear context.
Reward Code Generation and Critique (Stage 1):
- Instead of generating a monolithic reward function, the LLM generates individual reward components tailored to each specific user requirement.
- A Reward Critic module validates these components. It follows a step-by-step guide: listing potential failure reasons, reviewing the environment code and requirements, and outputting corrected code. This process isolates errors (e.g., incorrect logic or missing dense feedback) and allows for correction with minimal iterations (often just one).
Reward Weight Search (Stage 2):
- Weight Initialization: A Reward Weight Initializer uses the LLM to pre-calculate approximate values for reward components, ensuring they are on the same scale. This prevents initial weight groups from deviating significantly from optimal solutions and helps acquire diverse Pareto solutions without extensive searching.
- Log Analysis: To avoid overwhelming the LLM with lengthy raw training logs, a Training Log Analyzer produces a concise textual summary of the training results (e.g., average data overflow, energy consumption trends).
- Directional Search Strategies: The Reward Weight Searcher processes $K=5$ $K = 5$ input weight groups. Based on the summarized logs, it generates $K=5$ $K = 5$ output groups using strategies inspired by genetic algorithms but adapted for LLMs:
  - Directional Mutation: The searcher specifies starting points for adjusting specific weights (e.g., increasing or decreasing a penalty) based on the performance of specific input groups.
  - Crossover: Adjustments from different mutated groups are integrated to explore the search space efficiently.
- This decomposition separates the generation of adjustment suggestions from the execution of weight updates, reducing the burden on the LLM's numerical and long-context understanding capabilities.

Key Results
The framework was evaluated on a customized data collection task involving multiple autonomous mobile agents (AMAs) with safety, performance, and energy objectives.

Code Correction: The Reward Critic successfully corrected reward code errors (such as reversed penalty signs or missing dense feedback) with only one feedback iteration per requirement. In contrast, baseline methods like EUREKA-S failed to identify logical errors, only modifying weights.
Search Efficiency:
- The Reward Weight Initializer successfully generated solutions within the Pareto solution set, often eliminating the need for further search.
- Even when initialized with weights significantly off-target (e.g., 500 times the optimal value), the system met user requirements in an average of 5.2 iterations using GPT-4o.
- Removing the Training Log Analyzer (TLA) or the weight balancing process resulted in higher iteration counts and less flexible step-size selection.
Model Performance:
- GPT-4o demonstrated superior flexibility in choosing search step sizes and handling complex contexts.
- GPT-4o mini, despite having weaker numerical reasoning and long-context capabilities, performed adequately. The decomposition of the weight search process allowed the smaller model to succeed, though it required slightly more iterations due to less flexible step-size adjustments.

Significance and Claims
The paper claims that ERFSL effectively leverages the semantic understanding capabilities of LLMs to solve multi-objective reward design problems without requiring human reward examples. By separating the design of reward components from the search for reward weights, and by utilizing a white-box approach with clear feedback loops, the framework overcomes the limitations of black-box optimization.

The authors emphasize that ERFSL enables:

Zero-shot adaptability: Designing reward functions for custom environments without prior reward examples.
Error robustness: The ability to self-correct code logic and prevent unrectifiable errors through the Reward Critic.
Efficient Pareto exploration: The ability to generate diverse, balanced reward functions through initialization and directed search strategies, reducing the computational cost of weight tuning.

The work suggests that while LLMs may struggle with pure black-box numerical optimization, they excel when provided with clear task contexts and when complex tasks are decomposed into manageable sub-tasks (code generation, log analysis, and weight adjustment).