Synthesizing Interpretable Control Policies through Large Language Model Guided Search

This paper proposes a novel method that leverages Large Language Models to evolve interpretable control policies represented as standard Python programs, offering a transparent and verifiable alternative to black-box neural network controllers for dynamical systems.

The Gist

Imagine you need to teach a robot how to do a tricky task, like swinging a pendulum all the way up to stand on its head, or catching a bouncing ball in a cup.

Usually, when we teach robots today, we use a method called "Deep Learning." Think of this like hiring a genius but mute chef. You give the chef ingredients (data), and they cook a meal (the robot's behavior). The meal tastes amazing, but the chef refuses to tell you the recipe. They just say, "Trust me, it works." If the robot crashes, you have no idea why, and you can't easily fix it because you don't know how the "secret sauce" was mixed. This is what the authors call a "black box."

This paper proposes a different, much more transparent approach. Instead of hiring a mute chef, they use a super-smart, chatty writing assistant (a Large Language Model, or LLM) to write the actual recipe (the code) for the robot.

Here is how their method works, broken down into simple steps:

1. The Goal: A Recipe, Not a Black Box

The authors want the robot's brain to be written in Python, a standard computer language that humans can read.

• Analogy: Instead of a mysterious black box, they want a cookbook. If the robot fails to catch the ball, a human engineer can open the cookbook, read the recipe, say, "Ah, it tried to move the cup too fast," and simply edit the line of text to fix it.

2. The Process: The "Evolutionary Writing Contest"

How do you get a computer to write a perfect recipe from scratch? You don't ask it once and hope for the best. You run a writing contest that evolves over time.

• The Setup: You give the AI a "starter recipe" (a basic, clumsy code) and a set of rules (the task: "swing the pendulum up"). You also give it a Judge (a simulation).
• The Round 1: The AI writes a few versions of the code.
• The Test: The Judge runs these codes in a video game simulation.
  • If the code makes the robot fall over, the Judge gives it a low score.
  • If the code makes the robot swing up, the Judge gives it a high score.
• The Evolution: The AI looks at the two best-scoring recipes from the previous round. It says, "Okay, Recipe A was good at swinging, and Recipe B was good at balancing. Let me mix them together and write a new version that is even better."
• The Loop: This happens thousands of times. The AI keeps generating new "recipes," the Judge tests them, and the best ones are kept to inspire the next generation.

3. The Result: A Human-Readable Solution

After running this contest for a while (which takes about 10 hours on a powerful computer), the AI produces a final piece of code.

• The Pendulum Example: The AI figured out that to swing the pendulum up, it needs to push hard when the pendulum is low (like pumping your legs on a swing), and then switch to a gentle, steady push when it gets near the top.
• The "Ball in Cup" Example: The AI wrote code that tells the cup to move up and down to catch the ball. When the authors looked at the code, they realized the AI was doing something slightly inefficient. Because the code was written in plain English-like Python, a human could easily add one extra sentence: "If the ball is too high, lower the cup slightly."
• The Magic: They added that one line, and the robot got much better at catching the ball. You couldn't do that with a "mute chef" (neural network); you would have to retrain the whole system from scratch.

Why Does This Matter?

• Safety: In critical jobs (like driving a car or flying a drone), you need to know why the machine is doing what it's doing. This method gives you that "why."
• Collaboration: It turns the AI into a partner. The AI does the heavy lifting of finding the solution, but the human can step in, understand the logic, and tweak it based on their own intuition.
• Transparency: There are no hidden secrets. The control policy is just a piece of text that anyone with basic coding knowledge can read and verify.

In summary: The authors built a system where an AI acts as a creative writer that drafts control programs, a simulator acts as a strict editor grading them, and the result is a perfectly written manual for a robot that humans can read, understand, and improve. It bridges the gap between the power of modern AI and the safety requirements of the real world.

Technical Summary

Here is a detailed technical summary of the paper "Synthesizing Interpretable Control Policies through Large Language Model Guided Search" by Carlo Bosio and Mark W. Mueller.

1. Problem Statement

The integration of Artificial Intelligence (AI) and Control Systems faces a critical challenge: interpretability. While modern learning-based control techniques (such as Reinforcement Learning and Neural Networks) achieve high performance, they rely on "black-box" models. These models lack transparency, making it difficult to verify safety, understand failure modes, or manually modify the control logic for specific constraints.

The authors aim to solve the problem of finding high-performance control policies for dynamical systems that are fully interpretable, verifiable, and modifiable by humans without sacrificing the optimization power of Large Language Models (LLMs).

2. Methodology

The proposed framework treats control policy synthesis as a program synthesis problem. Instead of optimizing the weights of a neural network, the system searches the space of executable code (specifically Python) to find a control policy.

Core Components

The algorithm follows an evolutionary loop guided by a pre-trained LLM, consisting of four main stages (illustrated in Fig. 1 of the paper):

1. Specification & Input:

   • The process begins with a specification file containing:
     • A task description and allowed libraries.
     • Starter code: A basic, often random, control function.
     • Evaluation function: A simulation-based scoring mechanism that quantifies the cumulative reward of a policy.

2. Prompt Construction:

   • The system constructs a prompt for the LLM containing:
     • Instructions to improve the policy.
     • The code of the current best-performing programs (or starter code in early iterations).
     • The function signature for the new policy to be generated.

3. Program Generation (The LLM):

   • A pre-trained, frozen LLM (StarCoder2-Instruct, 15B parameters) generates new candidate control programs based on the prompt.
   • Evolutionary Mechanism: The LLM is instructed to "improve" upon previous high-scoring programs. This acts as a crossover operation in evolutionary algorithms, combining logic from different successful iterations.
   • Hyperparameters: The generation uses specific sampling strategies (Temperature $T=1$, Top-$p=0.95$, Repeat-last-$n=15$) to balance creativity and stability.

4. Program Evaluation & Selection:

   • Generated code is parsed and executed in a sandboxed simulation environment (MuJoCo via DeepMind Control Suite).
   • Scoring: The performance is measured by the cumulative reward ($R = \sum r_t$).
   • Filtering: Syntactically incorrect or non-running programs are discarded. High-scoring programs are stored in a Database.
   • Island Model: To avoid local optima, the system runs 10 independent "islands" (parallel populations). Periodically, underperforming islands are re-initialized with the best programs from other islands.

3. Key Contributions

• Interpretable Policy Representation: The primary contribution is representing control policies as standard Python code rather than neural network weights. This allows engineers to read, understand, and manually tweak the logic (e.g., adjusting gains or adding safety constraints) directly.
• LLM-Guided Evolutionary Search: The paper demonstrates how to effectively combine LLMs with evolutionary algorithms for control synthesis. The LLM acts as a generator of functional code, while the simulation acts as the fitness evaluator, shifting the "black box" nature of AI from runtime execution to the design phase.
• Human-in-the-Loop Capability: Because the output is code, human experts can intervene post-synthesis. The authors demonstrate that a human can easily identify logical redundancies or intuitive improvements in the generated code and re-integrate them, leading to further performance gains.

4. Results & Case Studies

The method was validated on two challenging control tasks from the DeepMind Control Suite:

A. Pendulum Swing-Up

• Task: Swing a pendulum from a hanging position to an upright equilibrium with limited torque.
• Outcome: The system synthesized a compact, interpretable policy (approx. 10 lines of code).
• Policy Structure: The resulting code revealed a hybrid control strategy:
  • Bang-bang control: When the angle is far from upright, the policy applies maximum torque in the direction of velocity to build energy.
  • Linear feedback: Once near the upright position, it switches to a linear controller ($u = 5\theta - 0.9\dot{\theta}$) for stabilization.
• Significance: The logic was mathematically derivable and could be verified for local stability using Lyapunov analysis.

B. Ball in Cup

• Task: Catch a ball attached to a cup using a string by moving the cup in 2D.
• Outcome: The LLM generated a complex 8-dimensional policy.
• Human Refinement:
  • The initial generated code contained redundant logic (e.g., checks for values outside the physical bounds of the simulation).
  • A human researcher manually simplified the code and added an intuitive rule: If the ball is higher than the cup, lower the cup slightly to catch it.
  • Performance Gain: This simple human modification significantly reduced the number of episodes where the ball was not caught within the time limit (from ~30% failure to <10% failure), demonstrating the value of human intuition combined with automated search.

5. Significance and Future Directions

• Bridging the Gap: This approach bridges the gap between high-performance learning-based control and the rigorous safety/verifiability requirements of real-world robotics.
• Trade-offs: The authors acknowledge a trade-off: the method is computationally expensive (approx. 10 hours on a single GPU) compared to gradient-based RL because it lacks differentiable gradients to guide the search.
• Future Work:
  • Hybrid Optimization: Combining LLMs for structural generation (skeletons) with gradient-based methods for tuning numerical parameters (gains).
  • Scalability: Using multi-GPU setups and lighter models to handle higher-dimensional systems.
  • User Studies: Formalizing how human feedback can be integrated into the optimization loop.

In conclusion, the paper establishes that code is a superior representation for control policies in safety-critical applications. By leveraging LLMs to search the space of programs, the authors achieve high-performance control that remains transparent, modifiable, and verifiable by human engineers.