Technical Summary: PromptPex: Automatic Test Generation for Language Model Prompts

Problem Statement

Large Language Model (LLM) prompts are increasingly integrated into software applications as code-like artifacts that take inputs, generate outputs, and perform specific functions. However, unlike traditional software, prompts face unique engineering challenges:

1. Non-Determinism and Model Dependency: The output of a prompt depends on the specific AI model interpreting it. A prompt that functions correctly on one model may produce invalid or unexpected results on another, complicating model migration and portability.
2. Unpredictable Impact of Changes: While natural language prompts are easy to modify, the consequences of small edits are difficult to predict, making robust editing and regression testing challenging.
3. Lack of Automated Testing Tools: Existing approaches focus on prompt optimization for specific models or manual unit test creation. There is a lack of tools that automatically generate diverse, targeted unit tests to evaluate how a prompt behaves across different models and to identify regressions.

Methodology: PromptPex

The authors developed PromptPex, an LLM-based tool designed to automatically generate and evaluate unit tests for a given prompt (the Prompt Under Test, or PUT). The system operates through a four-stage pipeline:

1. Specification Extraction

PromptPex uses an LLM (specifically gpt-5_2025-08-07 in the experiments) to extract two key artifacts from the PUT:

• Input Specification (IS): A set of constraints defining valid inputs (e.g., "input must be a sentence and a word"). This helps identify under-specification or ambiguities in the prompt's input requirements.
• Output Rules (OR): A set of independent, concrete, and checkable constraints describing the expected output (e.g., "return only the tag," "if unknown, return 'Unknown'"). These rules act as assertions or post-conditions, derived strictly from the prompt text without assuming knowledge of the input.

2. Test Generation

Using the extracted IS and OR, PromptPex generates test cases:

• Rule-Based Generation: Tests are generated to satisfy specific output rules, ensuring coverage of the prompt's constraints.
• Inverse Rule Generation: To create challenging test cases, the system generates "inverse rules" (semantic inversions that violate the original rules, e.g., "output must include the tag and a description"). The LLM then generates inputs designed to trigger these inverse scenarios, effectively stress-testing the prompt's adherence to its constraints.
• Validity Enforcement: The IS is used to ensure generated tests are valid inputs, distinguishing between "valid tests that fail" (indicating a prompt flaw) and "invalid tests" (which are less actionable).

3. Execution

The generated tests are executed against multiple Models Under Test (MUTs). In the evaluation, the authors ran tests on four diverse models: gpt-oss, gemma2:9b, qwen2.5:3b, and llama3.2:1b.

4. Test Evaluation (Compliance vs. Correctness)

Since ground-truth correct outputs are not always available for open-ended prompts, PromptPex employs an LLM-as-a-Judge (o4-mini_2025-04-16) to evaluate compliance rather than correctness. The judge checks if the model's output adheres to the extracted Output Rules (OR) and the original prompt's constraints, returning a pass/fail status and an explanation.

Key Contributions

1. Systematic Automated Test Generation: PromptPex is the first tool to focus specifically on automated test generation for prompts, enabling developers to understand prompt behavior across multiple models.
2. Specification Extraction for Testing: The paper introduces a novel approach to extracting input specifications and output rules from natural language prompts. These artifacts serve as the foundation for test generation and help developers refine their prompts by revealing under-specification or ambiguities.
3. Evaluation of Model Compliance: The authors demonstrate that extracting specifications leads to higher-quality tests. They provide a benchmark suite of 22 prompts and show that specification-driven tests are more effective at identifying model limitations than baseline approaches.

Evaluation Results

The authors evaluated PromptPex against a sophisticated baseline (a zero-shot LLM-based test generator) using 22 benchmark prompts and four different models.

• Higher Non-Compliance Detection: The primary metric for test quality was the percentage of non-compliant outputs (tests that caused the model to violate the prompt's rules). PromptPex consistently generated tests that resulted in higher non-compliance rates across all models compared to the baseline. This indicates PromptPex is better at exposing gaps in prompt constraints.
• Effectiveness of Inverse Rules: Tests generated from inverse rules were found to be particularly effective at causing non-compliance, often outperforming tests generated from standard rules. This confirms that explicitly challenging the rules helps identify edge cases.
• Model Differentiation: The generated tests successfully distinguished the capabilities of different models. More capable models (e.g., gpt-oss) showed lower non-compliance rates, while smaller models showed higher rates. However, the tests also revealed that some smaller models performed well on specific prompts, aiding in model selection.
• Specification Quality: The extracted Output Rules showed high groundedness (89% of rules were present in the original prompt) and high spec agreement (96.8% similarity in behavior between the original prompt and a prompt augmented with the extracted rules), validating the extraction process.
• Input Validity: Almost all tests generated by PromptPex were deemed valid according to the extracted Input Specification, ensuring the tests represent realistic usage scenarios.

Significance and Claims

The paper positions PromptPex as a critical step toward treating prompts as robust software artifacts. Its significance lies in:

• Shift from Optimization to Verification: Moving beyond optimizing prompts for a single model to verifying their robustness and portability across models.
• Actionable Feedback: By extracting specifications, PromptPex provides developers with concrete insights into how a model interprets their prompt, highlighting ambiguities (e.g., whether "return only the tag" allows for reasoning text) that might otherwise go unnoticed.
• Regression Testing: The generated test suites serve as regression tests for future prompt modifications and facilitate safer model migration.

The authors acknowledge limitations, including the current support for single-input prompts only, the reliance on LLM-as-a-Judge (which introduces potential bias), and the lack of support for structured inputs like RAG data. They conclude that while PromptPex does not solve all prompt engineering challenges, it provides a foundational tool for automated testing, specification extraction, and model evaluation.