SPRIG: Improving Large Language Model Performance by System Prompt Optimization

1. Problem Statement

While Large Language Models (LLMs) have demonstrated impressive capabilities, their performance is highly sensitive to prompt engineering. Existing research has predominantly focused on Task Prompt Optimization—crafting specific instructions for individual tasks (e.g., "solve this math problem"). However, this approach is inefficient as the number of tasks grows, requiring new prompts for every new scenario.

Conversely, System Prompts (general instructions preceding task-specific details, such as "think step by step" or "act as an expert") have received less systematic attention. Current system prompting strategies (like Chain-of-Thought or persona adoption) are often isolated, manually crafted, and lack generalizability across diverse tasks, languages, and model families. There is a critical gap in understanding how to systematically construct a single, optimized system prompt that maximizes LLM performance across a wide range of general scenarios.

2. Methodology: SPRIG

The authors propose SPRIG (System Prompt Refinement for Increased Generalization), an edit-based genetic algorithm designed to iteratively optimize system prompts.

A. Prompt Component Corpus

Instead of generating raw text from scratch (which can be incoherent), SPRIG operates on a corpus of pre-defined components.

• Source: A pool of 9,000 components derived from 300 human-crafted prompts and 8,700 synthetic prompts generated by GPT-4o.
• Categories: Components are classified into 9 categories: Good Property, Role, Style, Emotion, Scenario, Jailbreak, Safety, Behavioral, and Chain-of-Thought (CoT).
• Definition: A component is a minimum semantic unit (typically a sentence) that can be combined to form a coherent prompt.

B. Prompt Reward Model

Evaluating every candidate prompt against 42+ benchmarks is computationally prohibitive. To solve this:

• A Reward Model (based on Modern-BERT) is fine-tuned using a max-margin pairwise loss.
• Training Data: 10,000 randomly generated prompts were evaluated on real benchmarks to create 100,000 prompt pairs with scores.
• Performance: The model achieves a Spearman correlation of 0.59 and NDCG@50% of 0.72 in ranking unseen prompts, allowing for efficient estimation of prompt quality without running full benchmarks at every step.

C. The Genetic Algorithm Pipeline

SPRIG iterates through 25 steps to evolve the best system prompt:

1. Initialization: Start with a population of prompts from the component corpus.
2. Evaluation & Elimination: The Reward Model scores the population; the bottom 50% are eliminated.
3. Mutation & Crossover:
   • Top 10% of survivors undergo mutation (Add, Rephrase, Swap, Delete, Merge) or crossover with other survivors.
   • Add/Rephrase/Merge: Uses GPT-4o to generate or combine components.
   • Crossover: Combines components from two parents while maintaining length constraints.
4. Ground Truth Sampling: 100 prompts from the updated population are evaluated on 42 actual benchmarks to obtain ground-truth scores.
5. Reward Model Retraining: The Reward Model is fine-tuned for one epoch using the new ground-truth data to improve its ranking accuracy for the next iteration.

3. Key Contributions

1. System-Level Optimization: Demonstrates that optimizing a single, task-agnostic system prompt yields performance gains comparable to optimizing specific task prompts for each individual task.
2. Complementarity: Proves that system-level and task-level optimizations are complementary. Combining a SPRIG-optimized system prompt with a PROTEGI-optimized task prompt leads to further performance improvements, exceeding the sum of their individual parts.
3. Generalization: Shows that optimized system prompts generalize effectively across:
   • Model Families: (Llama, Mistral, Qwen).
   • Languages: Significant improvements on multilingual benchmarks (MGSM, XCOPA, etc.).
   • Parameter Sizes: While individual optimizations struggle to transfer from small to large models, the combination of system and task optimization generalizes well to larger model sizes.

4. Experimental Results

The study evaluated SPRIG on 47 different tasks across 7 categories (Reasoning, Math, Social Understanding, Commonsense, Faithfulness, Knowledge, Language Understanding) using three medium-sized LLMs (Llama-3.1-8B, Mistral-Nemo, Qwen2.5-7B).

• Performance Gains: SPRIG improved the average score by ~10% over the unoptimized baseline, significantly outperforming the standard "Let's think step by step" (CoT) baseline.
• Comparison with SOTA: While PROTEGI (a state-of-the-art task optimizer) performed slightly better on specific tasks, SPRIG achieved comparable results using a single prompt for all tasks. The System + Task combination outperformed all other methods.
• Task-Specific Insights:
  • Math & Reasoning: Benefited most from system optimization.
  • Knowledge Tasks: Showed the least improvement, suggesting these rely more on pre-trained weights than prompting strategies.
  • Social Understanding & Commonsense: SPRIG outperformed CoT and PROTEGI in these domains.
• Evolutionary Trends:
  • Components: The optimized prompts converged to include 2–3 CoT and Behavioral components (e.g., "decompose first," "rephrase before answering") and one Good Property component.
  • Redundancy: Surprisingly, "Role" components (e.g., "You are an AI assistant") were selected less frequently than chance, suggesting they add little value.
  • Search Dynamics: Early iterations favored exploration (Add/Crossover), while later stages favored refinement (Rephrase/Swap).

5. Analysis: Embedding Space

Using Principal Component Analysis (PCA) on hidden states:

• Task Prompts: Cause minor adjustments within a local vector space (fine-tuning).
• System Prompts: Cause a significant shift in the global distribution of hidden states.
• Hypothesis: System prompt optimization locates a high-performance "global region" in the model's latent space, while task prompts fine-tune within that region.

6. Significance and Conclusion

SPRIG addresses the scalability bottleneck of prompt engineering by shifting focus from task-specific tuning to generalizable system-level instruction optimization.

• Efficiency: It reduces the engineering effort required for new tasks by providing a robust, pre-optimized system prompt.
• Robustness: The method is robust across languages and model families, making it suitable for multilingual and cross-model deployments.
• Future Direction: The findings suggest a new paradigm for LLM interaction: first, optimize the system prompt to establish a high-performance behavioral baseline, and then apply task-specific optimization for fine-grained adjustments.

The paper concludes that system-level instructions are a critical, underutilized lever for maximizing LLM potential, offering a pathway to more efficient and generalizable AI deployment.