Technical Summary: BT-APE for Requirements Classification

Problem Statement

Large Language Models (LLMs) are increasingly applied to Requirements Engineering (RE) tasks, such as requirements classification, due to their ability to follow natural language instructions without task-specific training. However, the performance of these models is heavily dependent on the quality of the prompts used to guide them. In the RE domain, prompt design remains a largely manual, ad-hoc process relying on trial-and-error and practitioner intuition. This lack of systematic optimization leads to inconsistent, suboptimal results and limits reproducibility. While Automatic Prompt Engineering (APE) techniques exist in general Natural Language Processing (NLP), their applicability to domain-specific RE tasks and their effectiveness compared to classical prompting strategies remain unexplored. Furthermore, existing APE methods often incur high computational costs, making them unsuitable for resource-constrained environments common in RE, where data confidentiality often necessitates local deployment on smaller servers.

Methodology: BT-APE

The authors propose BT-APE (Backtracking APE), a computationally lightweight framework that frames prompt design as an optimization problem. The method iteratively refines prompts using LLM-generated candidates, a backtracking search strategy, and dynamic example selection.

Core Workflow

The process is divided into three phases using a dataset split into three disjoint pools: a pool for sampling examples ($D_{pool}$), a validation set ($D_{val}$), and a held-out test set ($D_{test}$).

1. Initialization: An initial prompt is scored on the validation set to establish a baseline.
2. Iterative Refinement:
   • Candidate Generation: An LLM proposes a new candidate prompt conditioned on the current prompt and a balanced dynamic example batch. This batch is refreshed at every iteration to include one correctly classified positive, one correctly classified negative, one misclassified positive, and one misclassified negative example from $D_{pool}$.
   • Evaluation: Candidates are scored on $D_{val}$ using 3-run majority voting to mitigate stochasticity.
   • Bounded Backtracking: Unlike methods that immediately revert upon a single failure, BT-APE maintains a ranked list of all evaluated candidates. It continues along a trajectory until a configurable patience threshold ($X=3$) of consecutive non-improving iterations is reached. Only then does it backtrack to the next-best prompt in the history, preventing the search from getting stuck in unproductive regions.
3. Held-out Evaluation: The best prompt found during the search is evaluated exactly once on the held-out test set ($D_{test}$) to provide a final, unbiased performance metric.

Key Design Distinctions

BT-APE is explicitly contrasted with PE2, a state-of-the-art APE baseline. While both are single-trajectory, history-aware methods, BT-APE differs in three key ways to reduce computational footprint:

• Bounded Context: PE2 conditions proposals on the full history of top-$n$ prompts, causing input context to grow linearly. BT-APE conditions only on the current prompt and a compact, balanced batch, keeping context size bounded.
• Balanced Example Selection: BT-APE enforces a strict four-tuple (success/failure $\times$ class) selection, whereas PE2 uses an unconstrained error batch.
• Explicit Patience: BT-APE uses a fixed patience threshold for backtracking, whereas PE2 reacts more immediately to non-improvements.

Experimental Design

The study evaluates BT-APE across:

• Datasets: Three benchmark RE datasets: PROMISE (NFR), PROMISE-Refined, and SecReq.
• Models: Five instruction-tuned LLMs in the 7–8B parameter range (Qwen2, Falcon3, Granite-3.2, Ministral, LLaMA-3).
• Baselines: Four classical prompting strategies (Zero-shot, Few-shot, Chain-of-Thought, CoT+Few-shot) and the PE2 APE baseline.
• Initialization Variants: Two initialization strategies were tested: BT-APE-Uninformed (simple, intuitive definitions) and BT-APE-Informed (definitions curated from RE literature).

Key Results

Performance vs. Classical Baselines

BT-APE significantly outperforms all four classical prompting strategies across nearly all dataset and model combinations.

• Statistical Significance: The mean improvement in weighted F1 ($wF1$) over classical baselines ranges from +0.049 to +0.143, with large effect sizes ($r \ge 0.50$) and statistical significance ($p < 0.05$ after Holm-Bonferroni correction).
• Consistency: BT-APE matches or outperforms the best classical baseline in 11 out of 15 model-dataset configurations.

Performance vs. PE2 (APE Baseline)

• Accuracy: BT-APE achieves performance statistically indistinguishable from PE2 (mean difference $\Delta = +0.001$, negligible effect size). Both methods converge to a similar performance ceiling.
• Efficiency: Despite equivalent accuracy, BT-APE is substantially more efficient. It consumes approximately 72% fewer cumulative input tokens and 66% less wall-clock time than PE2. This makes BT-APE better suited for deployment on small, resource-constrained, or air-gapped servers.

Prompt Feature Analysis (RQ3)

A linear mixed-effects model analysis of prompt evolution revealed specific correlates of high-performing prompts:

• Positive Correlates: Higher verb count (action-oriented language) and higher punctuation marker count (structural clarity) are associated with better performance.
• Negative Correlates: Longer sentence/word counts, higher lexical diversity, and greater syntactic complexity correlate with lower performance.
• Semantic Drift: Iterations involving larger semantic changes (semantic drift) are associated with performance improvements, suggesting that substantive revision is more effective than surface-level paraphrasing.

Initialization Impact (RQ4)

• Informed vs. Uninformed: Providing domain-informed class definitions (BT-APE-Informed) did not yield a statistically significant improvement in final performance over minimal, uninformed definitions (BT-APE-Uninformed).
• Robustness: The iterative optimization process was robust enough to compensate for weaker initial prompts, converging to comparable final performance regardless of the initialization strategy.

Significance and Contributions

The paper claims three primary contributions:

1. A Lightweight APE Framework: BT-APE provides a computationally efficient alternative to existing APE methods, achieving state-of-the-art accuracy with a significantly reduced resource footprint. This enables the application of automated prompt engineering in RE settings where local, private deployment is required.
2. Systematic Empirical Evaluation: This work offers the first systematic comparison of APE against classical prompting for requirements classification across multiple datasets and LLMs. It demonstrates that APE transfers robustly to RE tasks with large effect sizes.
3. Insights into Prompt Evolution: The study identifies specific structural and semantic characteristics of effective prompts (concise, action-oriented, structurally clear) and shows that domain-informed initialization is not strictly necessary for the optimization process to succeed.

The authors position BT-APE as a practical solution that bridges the gap between the high performance of automated prompt optimization and the resource constraints of real-world RE environments. They release an interactive tool and a replication package to facilitate adoption and further research.