Technical Summary: Beyond the Prompt: Assessing Domain Knowledge Strategies for High-Dimensional LLM Optimization in Software Engineering

Problem Statement

Large Language Models (LLMs) have demonstrated efficacy in generating "warm start" samples for software engineering (SE) optimization tasks, significantly reducing the labeling budget required for active learning. However, a critical "dimensional barrier" exists: while LLMs achieve top performance on low-dimensional problems (≤11 features), their effectiveness collapses on high-dimensional problems (>11 features), where traditional Bayesian methods (e.g., Gaussian Processes) remain superior.

This failure is attributed to training data limitations. LLMs excel on common problems well-represented in their training corpora but struggle with specialized, high-dimensional SE tasks (e.g., cloud configuration, aerospace software processes) where public optimal solutions are scarce. Without sufficient training data, LLMs cannot learn the complex feature interactions characterizing high-dimensional spaces. Furthermore, as projections suggest LLMs may exhaust available textual training data by 2028, data-scarce optimization domains are becoming increasingly critical. The core problem is determining how systematic integration of domain knowledge—whether from human experts or automated reasoning—can bridge this divide and enable LLMs to generate effective warm starts for high-dimensional optimization.

Methodology

The study proposes and evaluates four distinct architectures designed to incorporate domain knowledge into LLM warm start generation. These approaches range from human-centric to fully automated strategies, evaluated on the MOOT repository of SE optimization datasets stratified by dimensionality (Low: <6, Medium: 6–11, High: >11 features).

1. Human-in-the-Loop Domain Knowledge Prompting (H-DKP)

This approach leverages human experts to verify and refine constraints iteratively.

• Protocol: An asynchronous 10-day iterative refinement sprint.
• Mechanism: The LLM generates an initial "Belief State" of constraints. Experts critique these via email, focusing on "Most Confusing Failures" (configurations the LLM predicted as optimal but which performed poorly).
• Process: Experts answer a specific question: "What domain rule is the model missing that explains this failure?" Their feedback is parsed and appended to the prompt context for the next day, evolving the LLM's mental model of the domain.
• Constraint: Recruitment is limited to datasets where Subject Matter Experts (SMEs) are accessible via the authors' academic network.

2. Adaptive Multi-Stage Prompting (AMP)

This method decomposes the generation process into a sequential reasoning pipeline to reduce logical inconsistencies common in single-shot prompting.

• Stage 1 (Analysis): The LLM analyzes metadata and few-shot examples to identify influential features, tradeoffs, and improvement directions.
• Stage 2 (Constraint Discovery): The model infers explicit hard constraints (inviolable rules) and soft constraints (heuristics).
• Stage 3 (Constrained Generation): The LLM generates configurations strictly adhering to the discovered constraints and prioritized features.
• Stage 4 (Self-Validation): The LLM acts as a critic, reviewing generated configurations against the constraint set and revising violations.
• Ablation: The study tests 2-stage, 3-stage, and full 4-stage pipelines to isolate the impact of each reasoning step.

3. Dimension-Aware Progressive Refinement (DAPR)

This approach addresses high-dimensional complexity by optimizing in reduced feature spaces and progressively expanding dimensions.

• Feature Ranking: Feature importance is calculated via an ensemble of Spearman coefficient, mutual information, and Random Forest importance, normalized and averaged.
• Progressive Expansion: The algorithm initializes with the top $k$ features. It prompts the LLM to generate configurations using only these features.
• Anchoring: Generated examples are mapped to nearest neighbors in the full dataset for evaluation. The best configuration is tracked, and the feature space is expanded by adding the next $s$ most important features, anchored to values from the current best configuration.
• Goal: To guide the LLM through the search space, preventing it from being overwhelmed by the full dimensionality immediately.

4. Hybrid Knowledge-Model Approach (HKMA)

This strategy synthesizes statistical scouting with Retrieval-Augmented Generation (RAG) to leverage both data-driven patterns and semantic understanding.

• Phase 1 (Statistical Scouting): A Tree-structured Parzen Estimator (TPE) performs a rapid, low-budget exploration ($B_{scout}=10$) to identify high-performing regions.
• Phase 2 (Empirical Priors): The distribution of top-performing configurations is analyzed to extract "Empirical Priors" (e.g., directional trends, boundary conditions).
• Phase 3 (RAG Synthesis): These priors serve as queries to retrieve relevant domain documentation and academic literature from a vector store.
• Generation: The LLM is prompted to synthesize configurations that satisfy both the observed statistical evidence and the retrieved semantic explanations.

Evaluation Metrics and Baselines

• Baselines: Random Sampling, Gaussian Process Model (UCB_GPM), and Standard LLM Warm Start (few-shot prompting with 4 examples).
• Primary Metric: Chebyshev distance to the optimal solution (normalized to [0,1]), measuring proximity to the theoretical optimum.
• Secondary Metrics: Generated example diversity (Average Pairwise Euclidean Distance) and Computational Cost (API tokens).
• Statistical Analysis: Results are ranked using the Scott-Knott Effect Size Difference (ESD) test with Cliff's Delta to ensure differences are both statistically significant and practically meaningful.

Key Contributions

1. Identification of the Dimensional Barrier: The paper systematically documents the performance collapse of LLMs on high-dimensional SE optimization tasks (>11 features), contrasting this with their success on low-dimensional tasks.
2. Taxonomy of Knowledge Integration: It proposes and defines four complementary strategies (H-DKP, AMP, DAPR, HKMA) for injecting domain knowledge into LLMs, moving from human-in-the-loop to fully automated hybrid models.
3. Methodological Frameworks:
   • An asynchronous iterative protocol for extracting tacit expert knowledge (H-DKP).
   • A multi-stage reasoning pipeline to enforce constraint adherence (AMP).
   • A progressive refinement algorithm to navigate high-dimensional search spaces (DAPR).
   • A hybrid framework combining statistical scouting with semantic retrieval (HKMA).
4. Rigorous Evaluation Plan: The study outlines a comprehensive evaluation plan using the MOOT repository, stratified by dimensionality, with robust statistical testing (Scott-Knott/Cliff's Delta) to validate improvements over established baselines.

Results

Note: As this paper is a proposal (indicated by the "Research Questions," "Analysis Plan," and "Execution Plan" sections), it does not contain final experimental results or empirical conclusions regarding which method performed best.

The paper outlines the plan to evaluate these methods against the "dimensional barrier." It hypothesizes that:

• Human-in-the-loop feedback (H-DKP) will provide statistically significant gains over automated methods, particularly in domains with scarce public data.
• Structured reasoning (AMP) and progressive refinement (DAPR) will mitigate the performance collapse seen in high-dimensional tasks compared to standard single-shot prompting.
• The Hybrid approach (HKMA) will effectively bridge the gap between LLM semantic understanding and the quantitative precision of Bayesian methods.

The study explicitly states that human studies will comply with Institutional Review Board policies and that H-DKP analysis will be restricted to datasets where expert recruitment is successful without dropout.

Significance and Claims

The paper claims that no prior work has systematically explored whether injecting domain-specific knowledge can overcome the limitations of LLMs in high-dimensional, data-scarce optimization domains.

The significance of this work lies in its potential to:

• Extend LLM Utility: Enable LLMs to function effectively as heuristic guides for optimization in complex, real-world SE tasks (e.g., software configuration, hyperparameter tuning) where they currently fail.
• Address Data Scarcity: Provide a pathway for optimization in domains where public training data is exhausted or non-existent, a critical concern given projections of LLM training data exhaustion by 2028.
• Reduce Costs: Offer strategies to generate high-quality warm starts with minimal execution costs, addressing the prohibitive expense of obtaining ground truth labels in software engineering.
• Bridge Human and AI: Demonstrate how human tacit knowledge can be systematically integrated with AI to solve problems that neither can solve alone, moving beyond the debate of whether LLMs are "assistive" or "authoritative" to a model where they act as guided optimizers.

The authors maintain a modest stance, framing the work as an assessment of strategies to determine if structured knowledge integration enables LLMs to generate effective warm starts, rather than claiming a definitive solution has already been found.