Technical Summary: Parameter-Efficient Neuroevolution for Diverse LLM Generation

Problem Statement

Large Language Models (LLMs) frequently exhibit mode collapse, producing homogeneous outputs that fail to explore the full space of valid solutions. In tasks such as code generation or creative writing, models tend to converge on syntactically similar variants or predictable patterns. This limitation hinders applications requiring diverse solutions, such as discovering multiple algorithmic approaches, generating comprehensive test suites, or exploring creative directions. While Quality-Diversity (QD) algorithms have successfully addressed this in robotics and game design by maintaining archives of diverse, high-performing solutions, their application to LLMs has been limited. Previous approaches (e.g., QDAIF, LMX) treat LLMs as fixed black-box generators, evolving only discrete prompts or using LLMs as evaluators without modifying any neural parameters. This prevents integration with the broader neuroevolution paradigm, which historically evolves network weights or architectures to discover diverse, high-performing solutions.

Methodology: The QD-LLM Framework

The paper proposes QD-LLM, a framework for parameter-efficient neuroevolution that bridges QD optimization with LLM generation by evolving prompt embeddings.

1. Evolved Prompt Embeddings

Instead of fine-tuning the entire model or evolving discrete tokens, QD-LLM evolves soft prompt embeddings—learnable continuous vectors ($p \in \mathbb{R}^{n \times d}$) prepended to the input.

• Parameter Efficiency: The framework evolves a compact neural interface of approximately 32,000 parameters (8 virtual tokens $\times$ 4096 dimensions) to steer a frozen LLM with 70B+ parameters.
• Mechanism: These embeddings act as an indirect encoding, creating additional key-value pairs in the attention mechanism that modify information flow throughout the network.
• Implementation:
  • For open-source models (e.g., Llama-3-70B), embeddings are injected directly into the embedding layer.
  • For API models (e.g., GPT-4), a projected discrete approximation is used, where continuous embeddings are mapped to the nearest vocabulary tokens, with bounded projection error.
• Optimization: The system employs gradient-free optimization (a core tenet of neuroevolution). It utilizes adaptive mutation strength (based on CMA-ME principles) and finite-difference gradient estimation for targeted behavioral mutations.

2. Hybrid Behavior Characterization

To organize the QD archive, the framework defines a behavior descriptor $b(t)$ that combines two components:

• Semantic Component: Derived from SBERT embeddings reduced via UMAP to capture high-level semantic diversity.
• Explicit Component: Task-specific features (e.g., cyclomatic complexity, lines of code, and algorithmic paradigm for code; sentiment, formality, and readability for writing).
• Hybrid Fusion: These are combined via weighted concatenation. The paper provides Theorem 5, establishing formal coverage bounds for this hybrid descriptor under the assumption of near-independence between components. Empirical validation confirms low mutual information (NMI $\approx$ 0.08–0.11), supporting the theoretical guarantee of near-multiplicative coverage improvement.

3. Co-Evolutionary Variation Operators

The framework employs operators that jointly mutate prompt embeddings and the resulting text:

• Targeted Behavioral Mutation: Uses finite-difference gradient estimation to steer embeddings toward underexplored regions of the behavior space.
• Exploratory Mutation: Applies Gaussian perturbations for broader exploration.
• Crossover: Interpolates embeddings between two parents and prompts the LLM to combine their text solutions.

4. Archive Management

The system extends CVT-MAP-Elites with uncertainty handling inspired by Extract-QD. It maintains a buffer of evaluations per archive entry, uses median fitness for comparisons to handle LLM stochasticity, and periodically re-evaluates elites.

Key Contributions

1. Evolved Prompt Embeddings for QD: A gradient-free optimization method for soft prompt vectors that enables behavioral steering without model fine-tuning. This approach is applicable to both open-source LLMs (direct access) and API models (projected approximation).
2. Hybrid Behavior Characterization with Formal Guarantees: A principled combination of semantic and explicit features. Theorem 5 provides formal coverage bounds under near-independence, empirically validated with low normalized mutual information (NMI).
3. Co-Evolutionary Variation Operators: Novel operators including targeted behavioral mutation via finite-difference gradient estimation and embedding-space crossover.
4. Comprehensive Validation and Downstream Utility: Extensive evaluation across code and creative writing benchmarks, demonstrating that diverse archives improve downstream tasks like test generation and fine-tuning data quality.

Experimental Results

The framework was evaluated on HumanEval (164 problems), MBPP (974 problems), and creative writing benchmarks (ROCStories, WritingPrompts) using Llama-3-70B, Mistral-Large, and GPT-4-turbo.

• Performance: QD-LLM achieved a 41.4% higher QD-Score and 46.4% higher coverage than the previous state-of-the-art (QDAIF) on HumanEval ($p < 0.001$, Vargha-Delaney $A = 0.94$).
• Ablation Studies: Removing the hybrid behavior characterization reduced performance by 18.6% (semantic only) and 27.0% (explicit only). Disabling prompt evolution dropped the QD-Score by 16.0%.
• Cross-Model Generalization: The method significantly outperformed baselines across Llama-3, Mistral-Large, and GPT-4 (even with projected approximation).
• Downstream Utility:
  • Test Generation: Diverse archives from QD-LLM discovered 34% more edge cases in test inputs compared to QDAIF.
  • Fine-Tuning: Models fine-tuned on QD-LLM archives achieved an 8.3% accuracy gain on held-out HumanEval problems compared to those trained on QDAIF solutions.

Significance and Claims

The paper claims to establish prompt embedding evolution as an effective paradigm for bridging neuroevolution and modern LLMs. By evolving a compact neural interface (indirect encoding) rather than the full network, QD-LLM overcomes the computational prohibitions of full-model evolution while retaining the benefits of neuroevolutionary diversity.

The authors emphasize that this work moves beyond treating LLMs as static black boxes, instead evolving the conditioning signals that directly modify attention mechanisms. The demonstrated downstream utility—specifically in generating diverse test cases and improving fine-tuning data quality—suggests practical value for software engineering and data augmentation. The work validates that parameter-efficient neuroevolution can effectively explore the solution spaces of massive, frozen foundation models.