promptolution: A Unified, Modular Framework for Prompt Optimization

1. Problem Statement

While Large Language Models (LLMs) have demonstrated impressive capabilities, their performance on specific tasks is highly sensitive to the quality, formulation, and structure of the input prompts. Manual prompt engineering is tedious, unreliable, and difficult to scale. Although "Automatic Prompt Optimization" (APO) has emerged as a solution, the current landscape suffers from significant fragmentation:

• Siloed Implementations: Most optimizers exist as isolated research repositories with hard-coded experimental setups.
• Maintenance Issues: Many research codes lack active maintenance, software tests, documentation, and robustness.
• Integration Friction: Existing tools often require invasive integration into application frameworks (e.g., rewriting pipelines) or are limited to specific models/tasks.
• Comparison Difficulty: Researchers and practitioners struggle to systematically compare different optimizers due to incompatible codebases and lack of standardized benchmarking tools.

2. Methodology: The promptolution Framework

The authors introduce promptolution, a unified, modular, and open-source Python framework designed to unify the APO landscape. It is built on a low-abstraction, extensible architecture consisting of four core components, all adhering to a unified interface via Base classes:

Core Components

1. LLM Interface: Supports various deployment modes (API, local transformers, vLLM) and handles parallelization and token budget monitoring.
2. Predictor: Defines how to extract predictions from raw LLM outputs. It supports flexible extraction strategies, such as FirstOccurrencePredictor or MarkerBasedPredictor (extracting text between specific HTML-like tags).
3. Task: Encapsulates the dataset, task description, and evaluation logic. It supports:
   • ClassificationTask: Standard metrics (e.g., accuracy).
   • JudgeTask: Uses an LLM-as-a-judge for subjective/creative tasks.
   • RewardTask: Optimizes against custom reward functions (e.g., code execution time).
4. Optimizer: The core engine that iteratively refines prompts. It evaluates the LLM's performance on a development set and updates prompts based on the chosen strategy.
   • Implemented Optimizers: The framework currently integrates OPRO (LLM-as-optimizer), EvoPrompt (Evolutionary Algorithms: GA and DE), and CAPO (Cost-aware, GA-based, jointly optimizing instructions and few-shot examples).
   • Extensibility: New optimizers can be added by inheriting from BaseOptimizer and implementing a _step() method.

Key Design Features

• Non-Invasive Integration: Unlike frameworks like DSPy that require rewriting pipelines into specific abstractions, promptolution returns a plain text string. This optimized prompt can be directly swapped into any existing LLM pipeline without refactoring.
• Systematic Benchmarking: Includes an ExperimentConfig class and helper functions (run_experiment, run_optimization) to facilitate reproducible, large-scale studies across multiple datasets and seeds.
• Token Budget Awareness: Built-in callbacks (e.g., TokenCountCallback) allow users to set strict token limits and terminate optimization early to control costs.
• Auxiliary Modules: Includes tools for post-hoc exemplar selection (adding few-shot examples to fixed instructions) and automatic initial prompt generation.

3. Key Contributions

1. Unified Framework: The first open-source library to integrate multiple contemporary discrete prompt optimizers (OPRO, EvoPrompt, CAPO) into a single, interoperable system.
2. Modularity & Extensibility: Designed for both practitioners (easy single-prompt optimization) and researchers (custom algorithm development and systematic benchmarking).
3. Framework Agnosticism: Returns optimized prompt strings rather than framework-specific objects, ensuring seamless portability across different LLM providers and application architectures.
4. Reproducibility: Provides a structured configuration system and detailed logging/callbacks to track the entire optimization trajectory, addressing the "black box" nature of many research codes.

4. Experimental Results

The authors evaluated promptolution on two standard datasets: GSM8K (math word problems) and SST-5 (sentiment classification), using the gemma-3-27B-it model.

• Baselines: Compared against unoptimized zero-shot prompts and other leading frameworks: AdalFlow (LLM-AutoDiff) and DSPy (GEPA).
• Performance:
  • CAPO (implemented in promptolution) achieved the best results on both datasets: 93.7% accuracy on GSM8K and 56.3% on SST-5.
  • EvoPrompt also performed strongly (91.0% on GSM8K).
  • All optimizers in promptolution significantly outperformed the unoptimized baseline (up to +15% absolute improvement).
• Competitiveness: promptolution matched or exceeded the performance of AdalFlow and DSPy while offering greater flexibility and lower abstraction.
• Efficiency: The framework successfully optimized prompts within a strict token budget (1M tokens, <$0.15 cost), demonstrating cost-efficiency.

5. Significance and Use Cases

The paper positions promptolution as a critical tool for advancing the field of prompt engineering:

• For Practitioners: Enables the optimization of prompts for existing production pipelines without the overhead of migrating to a new, monolithic framework. It supports "plug-and-play" optimization.
• For Researchers: Provides a standardized environment to develop, test, and compare new optimization algorithms. The low abstraction level allows for fine-grained control over optimization dynamics, intermediate results, and budget constraints.
• Future Directions: The authors plan to integrate with higher-level frameworks (like DSPy), add configuration management (Hydra), support multi-agent system prompt optimization, and explore ensemble strategies for optimizers.

Conclusion: promptolution solves the fragmentation in the prompt optimization landscape by providing a robust, modular, and non-invasive framework. It bridges the gap between academic research and practical application, enabling systematic benchmarking and the widespread adoption of automatic prompt optimization.