💻 computer science

From Brittle to Robust: Improving LLM Annotations for SE Optimization

This paper introduces SynthCore, a novel prompting strategy that aggregates independent few-shot LLM opinions into an ensemble to effectively annotate high-dimensional multi-objective software engineering optimization tasks, outperforming state-of-the-art alternatives without requiring human-labeled data.

Original authors: Lohith Senthilkumar, Tim Menzies

Published 2026-03-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lohith Senthilkumar, Tim Menzies

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Brittle" Genius

Imagine you have a brilliant, super-fast assistant (a Large Language Model, or LLM) who can read millions of books and write code. You ask this assistant to help you solve a very complex puzzle: How do we configure a software system to be fast, cheap, and bug-free all at the same time?

This is a "multi-objective" problem. It's like trying to buy a car that is the fastest, the most fuel-efficient, and the safest, all while costing the least amount of money.

The problem is that this assistant is brittle.

The Brittle Moment: If you ask the assistant to solve this puzzle just once, it often gives you a confident but wrong answer. It might hallucinate (make things up) or get stuck in a "local trap"—like finding a small hill that looks like a mountain peak, but missing the real mountain range nearby.
The Old Way: Researchers tried to fix this by asking the assistant to "think step-by-step" (Chain of Thought) or by having multiple AI agents debate each other. But these methods are slow, expensive, and sometimes just as confused as the first attempt.

The Solution: SynthCore (The "Swarm" Strategy)

The authors, Lohith and Tim, came up with a new strategy called SynthCore. Instead of asking the assistant to solve the puzzle once and hoping for the best, they treat the assistant like a swarm of bees or a lottery ticket machine.

Here is how SynthCore works, using a simple analogy:

The Analogy: The "Blind Taste Test"

Imagine you are a chef trying to create the perfect soup.

The Old Way (Single Prompt): You ask one chef to make the soup. They taste it, say "It's perfect!" (even if it's too salty), and you serve it. If they are having a bad day, the soup is ruined.
The "Debate" Way: You ask three chefs to argue about the recipe. They talk to each other, get confused, and maybe agree on a bad idea because they are trying to be polite.
The SynthCore Way:
- You ask 20 different chefs to make the soup.
- Crucial Rule: These chefs are in separate rooms. They cannot talk to each other. They don't know what the others are doing.
- You give them slightly different hints (random seeds) so they approach the problem from different angles.
- One chef might focus on salt, another on spice, another on temperature.
- The Magic: Once they are done, you take all 20 bowls of soup, taste them all, and pick the single best bowl.

Because you have 20 different "opinions," the chance that at least one of them stumbled upon the perfect recipe is incredibly high. Even if 19 chefs make a mistake, the one who got it right saves the day.

Why This Matters for Software

In the world of Software Engineering (SE), finding the perfect configuration for a system is like finding that perfect bowl of soup.

The Data: The researchers tested this on 49 different software puzzles (from configuring video encoders to managing project timelines).
The Result: SynthCore (the "Swarm") found better solutions than:
- Traditional math formulas (Gaussian Processes).
- Other AI methods (Tree of Parzen Estimators).
- Even the best human experts in some cases.
The Surprise: They did this without any human help. The "labels" (the answers) were generated entirely by the AI swarm.

The "High-Dimensional" Breakthrough

The paper highlights a specific victory: High-Dimensional Problems.

Low Dimensions: Like a simple puzzle with 3 pieces. A single AI can usually solve this.
High Dimensions: Like a puzzle with 30+ pieces. Previous studies said, "AI gives up here; it's too messy."
The Verdict: SynthCore proved that even in these messy, high-dimensional puzzles, if you use the "Swarm" approach, the AI can find the optimal solution. It turns a "brittle" AI into a "robust" one.

The Takeaway for Everyone

You don't need to be a computer scientist to understand the lesson here:

Don't rely on a single genius. Rely on a diverse crowd.

If you ask one person for advice on a complex problem, they might be wrong. But if you ask 20 people independently, gather their ideas, and pick the best one, you are almost guaranteed to find a brilliant solution.

SynthCore is simply a tool that automates this "crowd wisdom" for software engineers, allowing them to build better, faster, and cheaper software without needing a team of human experts to do the heavy lifting.

Summary in One Sentence

By asking an AI to solve a problem 20 times independently and picking the best answer, the authors turned a flaky, error-prone AI into a super-reliable tool for solving complex software engineering puzzles.

1. Problem Statement

Software Engineering (SE) data annotation is often costly, slow, and prone to human error, particularly for multi-objective optimization (MOO) tasks where multiple competing goals (e.g., minimizing cost while maximizing performance) must be balanced.

The Limitation of LLMs: While Large Language Models (LLMs) have shown promise in automating annotation, recent empirical studies indicate they are "brittle" in high-dimensional, multi-objective SE contexts. Single-prompt LLMs often fail to generalize to complex optimization problems, producing shallow or hallucinated outputs, especially when the dimensionality of independent variables ( $x$ ) exceeds 11.
The Gap: Existing methods like Chain-of-Thought (CoT) or Multi-Agent Debates are computationally expensive or rely on consensus mechanisms that may suppress rare but optimal solutions. There is a need for a robust, scalable method to generate high-quality "warm start" data for active learning in SE optimization without relying on scarce human expertise.

2. Methodology: SynthCore

The authors propose SynthCore, a novel prompting strategy designed to transform brittle single-prompt LLMs into a robust ensemble of few-shot learners.

Core Concept: Instead of relying on a single monolithic answer or a consensus vote (which minimizes variance), SynthCore treats the LLM as an evolutionary search engine. It generates diverse "mutations" (candidate solutions) by running multiple independent prompt sessions.
Mechanism:
1. Initialization: The system selects $L$ random rows from the dataset to serve as initial context.
2. Ensemble Generation: The LLM is prompted $M$ $M$ times (e.g., $M=20$ $M = 20$ or $30$). In each iteration:
  - The context window is reset.
  - The LLM is given a few-shot prompt containing $N$ examples (split into "Best" and "Rest" based on performance).
  - The LLM is tasked with synthesizing a new example that outperforms the provided "Best" examples.
  - Crucially, each trial is independent; the LLM has no knowledge of the other trials' outputs (no crossover).
3. Aggregation: All generated candidates ( $L + M \times N$ ) are collected.
4. Selection: The candidates are sorted using an external objective function (Chebyshev distance to the ideal point), and the single best candidate is selected.
Distinction from Other Methods:
- Vs. Self-Consistency: Self-consistency seeks the "average" truth via voting to minimize noise. SynthCore seeks "outlier" solutions (mutations) to maximize exploration and escape local optima.
- Vs. Chain-of-Thought: SynthCore avoids the overhead of generating reasoning traces, focusing instead on input variation (randomized seeds/shuffled examples) to drive diversity.

3. Experimental Setup

Datasets: The study evaluated SynthCore on 49 multi-objective SE optimization tasks from the MOOT (Multi-Objective Optimization Testing) repository.
- Domains: Software project management (POM3, NASA93), configuration tuning (Makefiles, X264, SQL, Apache), and hyperparameter optimization.
- Dimensionality: Datasets were categorized as Low ( $|x| < 6$ ), Medium ( $6 \le |x| \le 11$ ), and High ( $|x| > 11$ ).
Baselines: SynthCore was compared against:
- Symbolic Methods: Gaussian Process Models (GPM) with Upper Confidence Bound (UCB), Tree-structured Parzen Estimators (TPE).
- Active Learners: Exploitative and Exploratory modes.
- LLM Baselines: Standard few-shot learning (single prompt, no ensemble) and random selection.
Metrics: Performance was measured using Chebyshev Distance to the ideal "heaven" point (minimizing the maximum deviation across all objectives). Statistical significance was determined using Scott-Knott, Bootstrap, and Cliff's Delta.
Model: The primary LLM used was Gemini 1.5 Pro (chosen for its large context window and reasoning capabilities).

4. Key Results

Superiority in High Dimensions: In high-dimensional tasks ( $|x| > 11$ ), standard single-prompt LLMs performed poorly (ranking 2nd or 3rd). However, SynthCore consistently achieved the top rank (Rank 0) in 71% to 86% of cases across all dimensionalities.
Comparison with Symbolic Methods: SynthCore outperformed state-of-the-art symbolic methods (GPM, TPE) and active learners. Notably, in high-dimensional settings, SynthCore was the only method to consistently outperform the UCB-GPM baseline.
Cost Efficiency: The method achieved optimal results with a small labeling budget ( $B=20$ to $30$). Increasing the budget beyond 50 yielded no statistically significant performance gains, suggesting SynthCore is highly sample-efficient.
RQ1 (Ensemble vs. Standard): Yes, ensemble-based few-shot learning significantly outperforms standard single-shot learning in SE active learning.
RQ2 (High-Dimensional Data): Yes, the ensemble approach successfully resolves the failure mode of LLMs in high-dimensional data reported in prior literature.

5. Key Contributions

Novel Prompting Tactic: Introduction of SynthCore, an ensemble strategy that aggregates independent few-shot LLM generations to create a diverse search space for optimization.
Empirical Validation: Comprehensive evaluation across 49 diverse SE datasets, demonstrating that LLMs can be robust contributors to SE automation when orchestrated correctly.
State-of-the-Art Performance: Proof that LLM-based ensembles can outperform traditional Bayesian optimization and active learning methods, particularly in complex, high-dimensional scenarios.
Open Science: Release of all data, scripts, and reproduction packages to support further research.

6. Significance and Implications

Reframing LLMs in SE: The paper shifts the paradigm from viewing LLMs as "oracles" that must be correct to viewing them as generative seeds for optimization. Even if individual LLM outputs are unreliable, their diversity can be harnessed to find optimal solutions.
Practical Application: SynthCore offers a low-cost, human-free alternative for generating "warm starts" in software configuration, project planning, and hyperparameter tuning. It acts as a "subject matter expert amplifier," reducing the cognitive load on human experts by filtering thousands of candidates down to a few high-quality options for validation.
Simplicity and Scalability: Unlike complex multi-agent debates, SynthCore is simple to implement (a wrapper around standard few-shot prompts) and computationally efficient, making it suitable for CI/CD pipelines and real-world deployment.
Future Directions: The authors suggest this approach could enhance other few-shot prompting results and be extended to Next-Release Planning (NACP) and structured domains like patch generation.

In conclusion, the paper demonstrates that by moving from "brittle" single-prompt reliance to "robust" ensemble strategies, LLMs can effectively solve high-dimensional multi-objective optimization problems in Software Engineering, often surpassing traditional symbolic methods without human intervention.