🤖 AI

From Monolithic to Modular: Segment-level Automatic Prompt Optimization

This paper introduces SAPO, a segment-level automatic prompt optimization method that decomposes prompts into distinct components to target specific weaknesses and outperforms existing monolithic approaches across diverse benchmarks.

Original authors: Nikita Kulin, Viktor Zhuravlev, Artur Khairullin, Sergey Muravyov, Ilya Makarov, Daniil Sukhorukov, Ekaterina Averkova

Published 2026-08-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Nikita Kulin, Viktor Zhuravlev, Artur Khairullin, Sergey Muravyov, Ilya Makarov, Daniil Sukhorukov, Ekaterina Averkova

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to write a story, solve a math problem, or summarize a news article. You do this by giving it a set of instructions, which scientists call a "prompt." Think of a prompt like a recipe: if the instructions are vague or messy, the robot might bake a cake that tastes like soap instead of chocolate. For a long time, researchers tried to fix these recipes by rewriting the whole thing from scratch every time the robot made a mistake. It was like throwing away an entire cake because the frosting was slightly off, hoping the next attempt would be perfect. This method, known as "monolithic" optimization, often worked well for one part of the task but accidentally ruined another part, leaving the robot confused and the results inconsistent.

The paper you are about to read tackles this problem with a new approach called SAPO (Segment-level Automatic Prompt Optimization). Instead of tossing the whole recipe, SAPO acts like a master chef who carefully tastes the cake, identifies exactly which ingredient is wrong (maybe too much salt, or not enough sugar), and fixes just that part while leaving the perfect parts alone. The authors tested this method on five different types of tasks, from answering reading comprehension questions to solving complex math problems, using two different robot brains (GPT-3.5-Turbo and GPT-4o-mini). They found that by breaking instructions into smaller, manageable pieces and only fixing the weak spots, they could get much better results than previous methods, improving performance by up to 7.25% on some tasks.

The Recipe for a Better Robot

Let's dive into how this works. Imagine you are giving a robot a very long, complicated instruction manual. In the past, if the robot got the answer wrong, an automatic optimizer would try to rewrite the entire manual in one go. It's like trying to fix a broken car engine by replacing the whole car. Sometimes, this new car works great for driving but terrible for parking. The old method often made the robot better at one thing but worse at another, a problem the authors call "prompt drifting."

The authors of this paper, Nikita Kulin and their team, suggest a smarter way. They realized that a good instruction manual isn't just one big block of text; it's made of different sections, like chapters in a book. They break the prompt down into four specific "segments":

Role: Who is the robot supposed to be? (e.g., "You are a helpful historian.")
Context: What is the background story or situation? (e.g., "You are talking to a student.")
Tasks: What exactly does the robot need to do? (e.g., "Summarize this text in three sentences.")
Output Format: How should the answer look? (e.g., "Give me a bulleted list.")

The SAPO Detective Game

The new method, SAPO, treats these four segments like clues in a detective game. Here is the step-by-step process they use:

First, the system runs the robot through a training session with a bunch of examples. It looks at the results and picks out the top 5 best examples (where the robot got it right) and the bottom 5 worst examples (where the robot messed up).

Next, the system plays detective. It asks a second, smarter robot (acting as a judge) to look at the top 5 and bottom 5 examples and figure out why the robot succeeded or failed. Did the "Role" section confuse the robot? Was the "Task" too vague? The judge identifies which segments were "strong" (working well in the good examples) and which were "weak" (causing trouble in the bad examples).

Then comes the magic part: Constrained Synthesis. Instead of rewriting the whole manual, the system generates new versions of the prompt that keep the strong segments exactly as they are and only change the weak ones. It's like keeping the perfect chocolate cake batter but swapping out the burnt frosting for a fresh batch. This ensures that the parts of the instruction that were already working don't get accidentally broken.

Finally, the system tests these new, partially-revised prompts on a separate set of questions (the validation set). If the new prompt does better, it gets accepted. If not, the system keeps the old one. This happens over and over again until the robot is as good as it can get.

The Results: A Clear Winner

The team tested this method on five very different challenges:

SQuADv2: Answering questions based on a reading passage.
TweetEval: Classifying the mood of tweets.
XSUM: Summarizing news articles.
CommonGen: Creating sentences from a list of words.
GSM8K: Solving grade-school math word problems.

They ran these tests on two different robot models: GPT-3.5-Turbo and GPT-4o-mini. The results were quite impressive. When compared to other popular methods that rewrite the whole prompt at once (like APE, OPRO, and EvoPrompt), SAPO consistently scored higher on average.

On the GPT-3.5-Turbo model, SAPO improved the average score by 5.13% compared to the next best method. On the more advanced GPT-4o-mini, the improvement was even larger, jumping 7.25% higher than the competition. The biggest wins happened on tasks that require very strict answers, like the math problems (GSM8K) and the reading comprehension (SQuAD2), where a tiny mistake in the instructions can lead to a completely wrong answer.

Why This Matters

The authors suggest that this success happens because SAPO stops the "destructive interference" that happens when you change everything at once. By localizing the changes to only the weak parts of the prompt, the system preserves the parts that are already working perfectly. It's a more careful, surgical approach to teaching robots.

However, the paper also notes a few limitations. The current method uses a fixed set of four segments. While this works well for many tasks, it might not be detailed enough for every single type of problem. The authors suggest that future versions could let the robot automatically discover new types of segments if needed, rather than sticking to just these four.

In short, SAPO shows that when you want to teach a super-intelligent robot, sometimes the best way to fix a mistake isn't to start over, but to carefully identify exactly what went wrong and fix just that piece. It's a shift from "rewrite everything" to "fix what's broken," and for now, it seems to be the winning strategy.

Technical Summary: Segment-level Automatic Prompt Optimization (SAPO)

Problem Statement

Automatic Prompt Optimization (APO) aims to iteratively generate and select improved prompts using data and model feedback, offering a deployment-friendly alternative to fine-tuning for Large Language Models (LLMs). However, existing APO pipelines predominantly treat prompts as monolithic strings, optimizing them through global rewrites. This approach suffers from "prompt drifting," where edits intended to fix specific failure cases inadvertently degrade previously correct behaviors. Furthermore, the effectiveness of current optimizers is often sensitive to model capabilities and setup choices, lacking a mechanism to control where and how edits are applied within the prompt structure.

Methodology: SAPO

The paper introduces SAPO (Segment-level Automatic Prompt Optimization), a modular strategy that decomposes prompts into explicit, operationally separable segments rather than optimizing them as a single block.

1. Prompt Decomposition
SAPO structures prompts into four distinct segments:

Role ( $s_{role}$ ): Defines the behavioral stance (e.g., classifier, analyst).
Context ( $s_{context}$ ): Encodes task grounding, input injection, and domain constraints.
Tasks ( $s_{tasks}$ ): Specifies actionable requirements and decision rules.
Output Format ( $s_{output}$ ): Governs response schema and formatting constraints.

2. The Optimization Loop
The method operates via a two-stage generation pipeline within a validation-gated search loop:

Stage A: Contrastive Diagnosis. The current prompt is evaluated on a training set ( $D_{train}$ ). The system extracts top-5 (strong) and bottom-5 (weak) examples based on performance metrics. For discrete metrics, class-aware selection is used to ensure balanced evidence. A single LLM, guided by static meta-prompts, analyzes these examples to diagnose which segments are associated with success (strong) and which with failure (weak), generating specific recommendations for revision.
Stage B: Constrained Candidate Synthesis. The same LLM generates $K$ candidate prompts. Crucially, synthesis is constrained: candidates must preserve segments identified as "strong" while primarily modifying "weak" segments. This reduces cross-segment interference.
Selection and Acceptance. Candidates are evaluated on a validation set ( $D_{val}$ ). A candidate is accepted only if it improves the validation score. In the event of a tie, the candidate with the minimum edit distance to the current prompt is selected to favor conservative updates.

Key Contributions

The paper outlines three primary contributions:

Segment-Explicit Optimization Objective: Unlike global rewrites, SAPO decomposes prompts into role, context, tasks, and output format, enabling targeted edits.
Constrained Candidate Synthesis: The method enforces the preservation of strong segments and the revision of weak ones, utilizing an edit-distance tie-break to ensure updates remain conservative.
Contrastive Diagnostic Stage: Update decisions are grounded in explicit top/bottom evidence, including class-aware handling for discrete per-example metrics, to identify specific failure modes.

Experimental Results

SAPO was evaluated across five diverse datasets (SQuADv2, TweetEval, XSUM, CommonGen, GSM8K) using two model backbones: GPT-3.5-Turbo and GPT-4o-mini. It was compared against Zero-shot prompting and strong APO baselines including APE, OPRO, EvoPrompt, GEPA, and StraGO.

Performance: SAPO achieved the best average score across all datasets and both model backbones.
Gains: Relative to the strongest competing baseline, SAPO yielded an average improvement of +5.13% on GPT-3.5-Turbo and +7.25% on GPT-4o-mini.
Task Sensitivity: The largest relative gains were observed on GSM8K (mathematical reasoning) and XSUM (abstractive summarization), suggesting that segment-level control is particularly beneficial for tasks requiring strict answer extraction or tightly constrained generation.
Robustness: The method demonstrated consistent performance, outperforming all baselines on GPT-4o-mini across all five datasets, indicating that segment-level optimization complements model capability scaling.

Significance and Claims

The authors position SAPO as a bridge between static meta-prompting and structure-aware optimization. The paper claims that the observed improvements stem from the hypothesis that explicit weak/strong segment separation reduces destructive interference between edits. By localizing revisions to weak components while preserving high-performing structures, SAPO addresses the instability and "prompt drifting" common in iterative APO.

The work asserts that SAPO offers a deployment-friendly, black-box compatible pipeline that enforces segment-level controllability without requiring complex implementation overhead. While the authors acknowledge limitations regarding the fixed nature of the segment taxonomy and the potential for missing finer-grained error modes with fixed top/bottom windows, they conclude that SAPO provides an interpretable pathway for future extensions in automatic prompt optimization.