Multimodal LLM-assisted Evolutionary Search for Programmatic Control Policies

This paper introduces Multimodal Large Language Model-assisted Evolutionary Search (MLES), a novel framework that combines multimodal LLMs with evolutionary search and visual feedback to automatically generate transparent, verifiable, and human-aligned programmatic control policies that match the performance of deep reinforcement learning methods like PPO.

The Gist

Imagine you are trying to teach a robot to drive a car or land a spaceship.

The Old Way (Deep Reinforcement Learning):
Traditionally, we teach robots by letting them crash thousands of times while we tweak invisible knobs in a giant, complex math brain (a neural network). The robot eventually learns to drive, but it's a "Black Box." You can't ask it why it turned left; it just "feels" like the right move based on billions of numbers. If it crashes, debugging it is like trying to fix a watch by guessing which gear is broken without being able to see inside. It works, but it's scary and hard to trust.

The New Way (MLES):
This paper introduces MLES (Multimodal LLM-assisted Evolutionary Search). Think of this not as training a black box, but as hiring a team of expert engineers and a super-smart AI coach to write the robot's instruction manual together.

Here is how it works, using a simple analogy:

1. The Team: The "Evolutionary" Engineers

Instead of one big brain, MLES uses a population of candidate policies. Imagine a room full of 16 different engineers, each trying to write a Python script (a set of clear, human-readable instructions) for the robot.

• The Goal: Find the best script that makes the robot drive perfectly.
• The Process: It's like survival of the fittest. The engineers write their scripts, the robot tries them out, and the ones that do the worst get fired. The best ones get to "reproduce" and write new, improved versions.

2. The Coach: The Multimodal AI (The "Eye" and "Brain")

This is where the magic happens. In the old days, the coach only looked at the score (e.g., "You crashed, score: 0").
In MLES, the coach is a Multimodal Large Language Model (MLLM). This is an AI that can see and think.

• It watches the video: Instead of just seeing a low score, the AI coach watches a video of the robot crashing.
• It diagnoses the problem: It says, "Ah, I see what happened! The robot turned too sharply at high speed and spun out. The script told it to turn too hard."
• It gives specific advice: It tells the engineer, "Don't just change the numbers randomly. Look at the video. You need to tell the robot to slow down before the turn."

3. The "Thought" Process

Every time an engineer writes a new script, they also write a "Thought" (a short note explaining their logic).

• Example: "I'm slowing down before the curve because the video showed the car spinning out."
  This makes the whole process transparent. You can read the code and the notes and understand exactly why the robot makes a decision.

The Creative Analogy: The Driving School

Imagine a driving school for robots:

• Old Method: You put the robot in a car, let it drive until it crashes, and then you tweak the car's internal wiring (which you can't see) to make it slightly less likely to crash next time. You never know why it crashed, you just know it crashed less often.
• MLES Method: You have a fleet of student drivers (the scripts). A super-smart instructor (the Multimodal AI) watches their driving videos.
  • The instructor sees a student swerving.
  • The instructor says, "You're turning the wheel too fast when you're going 60mph. Here is a new rule: 'If speed > 50, turn wheel slowly.'"
  • The student writes this new rule down in plain English.
  • The next day, the student tries again. They do better.
  • The instructor picks the best students, mixes their best rules, and creates a new, even better student.

Why is this a Big Deal?

1. Transparency: The final result isn't a mysterious black box; it's a readable code with comments. A human can look at it and say, "Yes, that makes sense."
2. Trust: Because we can see the logic, we can verify it's safe before letting it drive a real car or land a real spaceship.
3. Efficiency: By using the AI coach to look at videos of failures (not just scores), the system learns much faster. It doesn't just guess; it diagnoses and fixes specific mistakes.

The Results

The paper tested this on two tasks:

• Lunar Lander: Landing a spaceship.
• Car Racing: Driving a car on a track.

The MLES method performed just as well as the best traditional AI methods (which use the "Black Box" neural networks), but it did it by creating clear, understandable instructions that humans can read and verify.

In short: MLES turns the scary, opaque process of teaching robots into a transparent, collaborative workshop where AI and humans work together to write clear, safe, and effective rules for the future.

Technical Summary

Here is a detailed technical summary of the paper "Multimodal LLM-Assisted Evolutionary Search for Programmatic Control Policies" (MLES), published as a conference paper at ICLR 2026.

1. Problem Definition

The paper addresses two critical limitations in Deep Reinforcement Learning (DRL) for control tasks:

• Opacity: DRL policies are typically represented as opaque neural networks ("black boxes"), making them difficult for humans to understand, verify, or debug. This lack of transparency hinders trust and deployment in safety-critical domains (e.g., autonomous driving, healthcare).
• Inflexibility: The gradient-based optimization of DRL relies on high-dimensional parameter spaces, making it difficult to analyze, intervene in, or reuse learned knowledge.

While existing methods like LLM-assisted Evolutionary Search (LES) have been used to optimize reward functions or heuristics, they have not yet successfully synthesized high-performance programmatic control policies directly. Furthermore, standard LES often relies on scalar metrics (e.g., total reward), which provide limited insight into why a policy fails, leading to inefficient trial-and-error search.

2. Methodology: MLES Framework

The authors propose Multimodal LLM-assisted Evolutionary Search (MLES), a novel framework that integrates Multimodal Large Language Models (MLLMs) with Evolutionary Computation (EC) to discover transparent, executable control policies.

Core Components

1. Policy Representation:

   • Policies are not neural weights but executable Python programs accompanied by a natural language "Thought" (a rationale describing the strategy).
   • This ensures the resulting policies are human-readable, modular, and verifiable.

2. Evolutionary Search Loop:

   • Population: A pool of candidate policies (Individuals).
   • Selection: Uses exponential rank selection to choose parent policies based on performance.
   • Operators: The framework employs four specific operators to guide the MLLM:
     • Exploration (E1, E2): Generates novel strategies by synthesizing fundamentally different logic or generalizing patterns from multiple parents.
     • Multimodal Modification (M1_M, M2_M): The core innovation. These operators use Behavioral Evidence (BE) to refine existing policies.
       • M1_M: Analyzes code and BE to identify behavioral shortcomings and revise control logic.
       • M2_M: Identifies critical parameters and adjusts them based on observed evidence.

3. Visual Feedback-Driven Behavior Analysis (The Key Innovation):

   • Unlike traditional EC that relies solely on scalar rewards, MLES converts raw execution traces into Behavioral Evidence (BE).
   • BE Formats: Includes visual data such as frame-stacked images (for Lunar Lander) or trajectory maps with visual field annotations (for Car Racing).
   • Role of MLLM: The MLLM acts as a reasoning agent that "sees" the failure modes (e.g., erratic movements, reward hacking) in the BE. It provides a diagnostic analysis ("Why did it fail?") and generates targeted code modifications, mimicking how human experts refine policies.

4. Prompt Engineering:

   • The system constructs multimodal few-shot prompts containing the task description, parent code/thoughts, and the BE.
   • Explicit instructions guide the MLLM to analyze the visual evidence before generating code, ensuring the search is grounded in diagnostic reasoning rather than stochastic guessing.

3. Key Contributions

1. Direct Programmatic Policy Synthesis: MLES is the first framework to use LES for the end-to-end discovery of high-performance control policies expressed as code, rather than just optimizing auxiliary components like reward functions.
2. Visual Feedback-Driven Evolution: It introduces a paradigm shift from scalar-metric-driven search to diagnostic refinement. By leveraging MLLMs to interpret visual execution traces, the system identifies specific failure patterns and guides targeted improvements, significantly boosting search efficiency.
3. Transparency and Traceability: The framework produces policies that are fully human-readable. The entire evolutionary process is traceable, showing how specific behavioral evidence led to code changes, bridging the gap between high performance and interpretability.
4. Scalability and Knowledge Transfer: The code-based representation facilitates knowledge reuse and transfer across tasks, overcoming the limitations of domain-specific languages (DSLs) used in previous programmatic RL approaches.

4. Experimental Results

The authors evaluated MLES on two standard OpenAI Gym benchmarks: Lunar Lander (discrete action) and Car Racing (continuous action, image-based).

• Performance:
  • MLES achieved performance comparable to Proximal Policy Optimization (PPO), a strong DRL baseline.
  • On Lunar Lander, MLES achieved a score of 1.090 (vs. PPO's 1.032).
  • On Car Racing, MLES achieved 98.07 (vs. PPO's 99.21), with a best run of 100.00.
• Efficiency:
  • MLES demonstrated superior search efficiency, converging faster than PPO and DQN, particularly on the complex Car Racing task.
  • It required significantly fewer environment resets to reach high-performance thresholds compared to baselines.
• Ablation Studies:
  • Removing Multimodal Modification operators (M1_M, M2_M) caused a catastrophic performance drop (e.g., -42% on Lunar Lander), proving that visual feedback is the primary driver of success.
  • Removing Exploration operators hurt performance on complex tasks but had minimal effect on simpler ones, highlighting the need for a balance between creativity and refinement.
• Cold-Start Capability: MLES could discover high-performing policies from scratch (without initial seeds), though providing initial policies or using more advanced MLLMs (e.g., GPT-5-mini) further accelerated the process.

5. Significance and Impact

• New Paradigm for Trustworthy AI: MLES offers a viable path toward verifiable and transparent AI control systems. By replacing black-box neural networks with executable code, it enables safety-critical applications where policy logic must be auditable.
• Human-in-the-Loop Potential: The modular nature of the generated code allows human experts to easily inspect, edit, and re-insert policies into the evolutionary loop, facilitating seamless human-AI collaboration.
• Cost-Effectiveness: Despite the computational cost of MLLM inference, the total cost to solve complex tasks (e.g., ~$0.42 for Car Racing) is negligible compared to the value of generating transparent, reusable policies.
• Future Direction: The paper suggests that as foundation models improve, MLES will naturally become more efficient and powerful, potentially becoming the standard for developing reliable, interpretable agents in robotics and autonomous systems.

In conclusion, MLES successfully demonstrates that combining the reasoning capabilities of Multimodal LLMs with evolutionary search can produce control policies that are not only high-performing but also transparent, debuggable, and human-aligned.