🤖 machine learning

SemanticOpt: Towards LLM-Based Semantic Black-Box Optimization

SemanticOpt is a novel framework that enhances Large Language Models for semantic black-box optimization by fine-tuning them on structured Bayesian trajectories enriched with natural language context, enabling them to effectively leverage domain knowledge and outperform both classical and existing LLM-based optimizers in expensive experimental scenarios.

Original authors: Jamison Meindl, Yunsheng Tian, Tony Cui, Veronika Thost, Zhang-Wei Hong, Jie Chen, Wojciech Matusik, Mina Konaković Luković

Published 2026-05-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jamison Meindl, Yunsheng Tian, Tony Cui, Veronika Thost, Zhang-Wei Hong, Jie Chen, Wojciech Matusik, Mina Konaković Luković

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to create the perfect new cookie recipe. You have a limited budget for ingredients and time, so you can only bake a few batches.

The Old Way (Traditional Optimizers):
Usually, a computer program helping you would be like a blind taste-tester. It only sees numbers: "2 cups of flour, 100 degrees heat, 30 minutes." If the first batch burns, it just notes "burned" and tries a slightly different number next time. It doesn't know why it burned. It doesn't know that "too much heat" usually ruins cookies, or that "baking soda" reacts differently than "baking powder." It's purely guessing based on math, ignoring the actual story of cooking.

The New Problem:
Sometimes, the "numbers" aren't enough. Maybe you are designing a new airplane wing, mixing paints, or tuning a chemical reaction. You have a lot of text-based knowledge available: old research papers, expert notes like "high heat speeds up the reaction but creates bad byproducts," or descriptions of similar past experiments. Standard computer programs can't read these notes; they only understand the numbers.

The "Smart" Way (LLMs):
Large Language Models (LLMs) are like super-readers. They can read all those expert notes, papers, and descriptions. They understand that "Phthalo Green" is a strong green paint. However, if you just ask an LLM to "find the best mix," it's like asking a brilliant food critic to cook. They know the theory, but they aren't great at the trial-and-error process of actually baking the cookies step-by-step. They might suggest a great first batch, but then they get stuck or keep making the same mistakes because they aren't trained to be a systematic optimizer.

The Solution: SemanticOpt
The authors created SemanticOpt, which is like hiring a Master Chef who is also a Data Scientist.

Here is how they built it:

The Training: They took a smart LLM (a "super-reader") and taught it how to be a systematic optimizer. They didn't just ask it questions; they showed it thousands of examples of successful experiments. These examples included the numbers (the recipe), the results (did it taste good?), and the context (the expert notes and scientific papers).
The Learning: The model learned to look at the numbers and the text together. It learned that if a paper says "high heat is risky," and the numbers show a high-temperature attempt failed, it should avoid high heat in the future. It learned to balance "exploring" new ideas with "exploiting" what it already knows works.
The Result: When given a new problem, SemanticOpt doesn't just guess numbers. It reads the problem description, looks at the history of what worked or failed, and uses its "common sense" (from the text) to pick the next best experiment.

The Analogy of the Paint Mixer:
The paper gives a great example of mixing paints.

The Goal: Mix 5 available paints to match a target color called "Mint Green."
The Traditional Optimizer: Tries random ratios of paints. It might try 90% Black and 10% White, get a dark gray, and realize it's wrong. It has to try thousands of combinations blindly.
SemanticOpt: You tell it, "The target is Mint Green, which is a 'strong vivid green'." It reads the names of the paints: "Phthalo Green" (a strong green), "Titanium White," etc. It immediately understands that it needs a lot of the green paint and some white. It skips the terrible guesses and finds the perfect mix much faster.

What They Found:

Better Start: SemanticOpt starts with better guesses than standard computers because it reads the instructions.
Better Improvement: Unlike a standard LLM that gets stuck after a few tries, SemanticOpt keeps getting better as it gathers more data, just like a real scientist learning from their mistakes.
The Winner: In tests ranging from designing airplane wings to optimizing chemical reactions, SemanticOpt beat both the traditional number-crunching computers and the raw, untrained super-readers.

The Catch:
The paper notes that this system relies heavily on the quality of the text you give it. If you give it misleading notes or if the problem is purely about random categories with no logical story (like picking a random ID number), it doesn't help as much. But when there is a "story" or "context" to the problem, SemanticOpt uses that story to find the answer much faster.

In short, SemanticOpt teaches a super-smart AI to not just read the manual, but to actually do the experiment, using both the math and the words to solve hard problems efficiently.

Technical Summary: SemanticOpt: Towards LLM-Based Semantic Black-Box Optimization

Problem Definition

Optimizing experimental systems is often hindered by the high cost, time consumption, or difficulty of performing individual evaluations. In these "expensive black-box" scenarios, the relationship between design parameters and outcomes is unknown. Traditional optimizers, such as Bayesian Optimization (BO), rely solely on numerical observations (configurations and scalar outcomes). They lack the ability to incorporate broader domain knowledge, such as expert heuristics, scientific literature, or semantic descriptions of previous experiments. While Large Language Models (LLMs) possess the capacity to interpret such semantic information, state-of-the-art LLMs struggle to perform reliable iterative optimization. They often excel at warm-starting (proposing initial candidates) but fail to maintain an effective exploration-exploitation balance over longer trajectories, leading to performance plateaus.

Methodology

The authors introduce SemanticOpt, a framework that equips LLMs with robust black-box optimization capabilities by fine-tuning them on structured BO trajectories augmented with natural language context.

1. Problem Framing

Optimization is cast as a sequential decision-making problem where an LLM policy $\pi_\theta$ interacts with an unknown objective function $f$ . At each step $t$ , the model receives a history of observations $D_t = \{(x_i, y_i)\}$ and semantic context $C$ (e.g., problem descriptions, historical data, or evaluation feedback). The model generates the next configuration $x_{t+1} \sim \pi_\theta(\cdot | C, D_t)$ .

2. Data Generation and Training

To overcome the scarcity of real-world benchmarks with semantic data, the authors constructed a large-scale training dataset comprising over 100,000 optimization trajectories:

Function Spaces: They generated diverse synthetic objectives using two methods:
- Gaussian Process (GP) Functions: Flexible, scalable generators augmented with nonlinear coordinate warping, discontinuities, constraints, and failure regions to mimic real-world irregularities.
- LLM-Generated Functions: Using Qwen 3.5 35B-A3B, they generated semantic contexts for real-world domains (e.g., chemical reactions, shipping logistics) paired with 100 evaluated points to define the ground truth via surrogate fitting.
Trajectory Generation: The state-of-the-art BO framework HEBO was used to generate high-quality optimization trajectories on these function spaces. The process involved initializing with 5 steps (using Sobol sampling for GP functions and LLM-generated initializations for semantic functions) and running 3 different surrogate models (GP, Random Forest, and SVIDKL). The best-performing surrogate at specific step counts (10, 25, 50, 100) was selected to compile the dataset.
Prompt Structure: The training data is formatted as multi-turn conversations. The system instruction enforces a strict JSON output containing the proposed configuration, predicted mean, standard deviation, and (if applicable) failure probability.
Fine-Tuning: The authors fine-tuned Qwen 3.5 27B (and 9B) models using Low-Rank Adaptation (LoRA). The model is trained to output both candidate configurations and surrogate-style predictions, effectively learning to act as an end-to-end sequential optimizer that integrates numerical feedback with semantic reasoning.

3. Inference Pipeline

During inference, SemanticOpt follows the same multi-turn structure. It begins by proposing 5 initial configurations based on the provided context. Subsequently, it iteratively proposes new configurations. To enhance decision-making, the model samples two candidate configurations per step, and the final selection is made by maximizing the expected improvement weighted by the predicted probability of success, thereby accounting for feasibility.

Key Contributions

Large-Scale Training Dataset: The construction of a dataset with over 100,000 optimization trajectories across diverse GP-generated and LLM-generated function spaces, utilizing BO methods to ensure high-quality optimization data.
Structured Fine-Tuning Approach: A novel method for fine-tuning LLMs on BO data to output both candidate configurations and calibrated surrogate-style predictions, creating the SemanticOpt model.
Comprehensive Evaluation: Evaluation across real-world surrogate benchmarks with varying levels of semantic context, demonstrating that SemanticOpt outperforms both classical BO methods and existing LLM-based approaches when relevant semantic information is provided.

Results

The authors evaluated SemanticOpt against a comprehensive set of baselines, including HEBO (with GP, RF, and SVIDKL surrogates), transformer-based optimizers (PFNs4BO), and frontier LLMs (GPT-5.4, Qwen 3.5 27B).

Performance: Across diverse domains (e.g., Airfoil CFD, MuJoCo Control, Paint Mixing, PD1), SemanticOpt achieved the best average normalized performance. It significantly outperformed base LLMs (Qwen 3.5 27B) in iterative optimization, showing continuous improvement rather than plateauing.
Semantic Context: The inclusion of semantic data (long-form context, historical examples, extra metrics) consistently improved performance. For instance, in a paint-mixing task, SemanticOpt utilized color descriptions to find a target match (score 0.91) far superior to HEBO GP (score 6.5).
Comparison with Frontier Models: While GPT-5.4 showed strong initialization performance on benchmarks derived from public datasets (suggesting potential memorization), SemanticOpt outperformed it on benchmarks generated via simulations with less online data (e.g., Airfoil CFD, MuJoCo Control). When given the same initialization from GPT-5.4, SemanticOpt demonstrated superior iterative optimization capabilities.
Surrogate Quality: SemanticOpt's internal predictions (mean and standard deviation) showed competitive or superior accuracy and uncertainty calibration compared to traditional BO surrogate models.
Ablation Studies:
- Model Size: The 27B model maintained initialization capabilities better than the 9B model, which suffered from "capacity-dependent forgetting" of pretrained priors.
- Initialization vs. Iteration: SemanticOpt's gains stem from both improved warm-starting and superior iterative optimization.

Significance and Claims

The paper claims that SemanticOpt bridges the gap between the semantic reasoning strengths of LLMs and the sample-efficient, uncertainty-aware search of Bayesian Optimization. By fine-tuning LLMs directly on structured BO trajectories augmented with semantic context, the authors demonstrate that LLMs can be trained as effective sequential optimizers.

The significance lies in creating an end-to-end semantic optimizer that does not rely on external BO loops or simple prompting. It enables optimization systems to leverage documentation, prior experiments, and auxiliary measurements when proposing new experiments. The authors note limitations, particularly in purely categorical spaces (where BO methods themselves struggle) and in domains with highly discontinuous objectives where BO calibration may fail. They suggest future work could involve incorporating decision justifications and expanding to more flexible function spaces.