Technical Summary: Continuous Generation of Test Cases for Large Language Models with Project Memory

1. Problem Statement

In large-scale software systems undergoing agile iterations and continuous delivery, maintaining effective test coverage is challenging due to evolving requirements, interface updates, and recurring defects. Conventional automatic testing scripts and standard Large Language Model (LLM) approaches often fail to effectively reuse historical failure data or cover emerging risks. Existing literature highlights specific limitations:

• Context Deficits: Standard LLMs struggle with unstable assertions and lack of context [1].
• Coverage Stagnation: Pretrained models may improve search-based tests but still face coverage plateaus [2].
• Historical Neglect: Approaches improving coverage often fail to sufficiently reuse project-level historical defects [3].
• Strategy Imbalance: Single-generation strategies struggle to balance coverage with defect-triggering capabilities [4].
• Memory Gaps: Test-driven development (TDD) applications of LLMs often lack mechanisms for long-term version evolution memory [5].

The core problem is defined as a Constrained Risk Coverage Optimization Task in Version Evolution, where the goal is to select an optimal test case set that maximizes weighted coverage of changed business paths and historical defect triggers while minimizing invalid cases and semantic duplication.

2. Methodology

The proposed solution introduces a Project Memory framework, treating memory not as a simple Retrieval-Augmented Generation (RAG) appendix, but as a release-evolving control layer. The methodology consists of three primary stages:

2.1 Construction of the Project Memory Knowledge Base

The system aggregates multi-source test assets at an event granularity, integrating:

• Data Sources: Requirement documents, interface descriptions (OpenAPI), code commits, issue/defect reports, and existing test scripts.
• Data Cleaning: Removal of template text and invalid logs; conversion of sensitive data (e.g., order numbers, SKUs) into irreversible tokens or generic placeholders.
• Semantic Segmentation: Data is tagged by release_id, business_path_id, api_chain, state_pair, risk_tag, and trigger_condition.
• Vectorized Representation: A heterogeneous memory state vector is constructed using an HNSW index. The vector $\mathbf{m}_i$ for a memory unit is a weighted fusion of:
  • Semantic embedding ($\mathbf{e}_{sem}$)
  • Type embedding (requirements, interface, defect, script)
  • Version embedding
  • Risk label embedding
  • Weights: $\alpha=0.45$ (semantic), $\beta=0.15$ (type), $\gamma=0.15$ (version), $\delta=0.25$ (risk).

2.2 Incremental Memory Update Mechanism

Memory evolves based on commits, interface changes, or defect transitions. A Diff Parser extracts changes, and if a threshold (0.35) is exceeded, the memory is versioned; otherwise, weights are updated. The weight update formula incorporates:

• Historical retention coefficient ($\eta = 0.72$)
• Defect recurrence frequency
• Business risk level
• Semantic similarity to current changes
• Version distance decay

2.3 Generative Pipeline

The generation process utilizes Llama3.1-70B-Instruct guided by the following steps:

1. Risk-Related Memory Retrieval: A query vector is built from request changes and API diffs. HNSW retrieves the top-50 memory units, which are then re-ranked by a risk-feedback controller based on defect recurrence, path importance, and version distance.
2. Prompt Construction: The top-12 risk memories, current interface diffs, and historical failure assertions are formatted into JSON fragments. The prompt includes fields for Scenario, API Chain, Preconditions, Input Boundaries, Risk Memory, Expected Assertions, and Negative Cases.
3. Generation & Validation: The model generates candidates with constrained parameters (Temperature=0.2, Top-p=0.85).
4. Deduplication & Consistency Check: Generated cases undergo:
   • Semantic Deduplication: Using Sentence-BERT with a 0.86 cosine similarity threshold.
   • Assertion & State Verification: Checking for new boundary values or exception branches.
   • Schema & State Machine Validation: Ensuring OpenAPI conformance and valid state transitions (e.g., preventing Paid $\to$ Created).

3. Key Contributions

• Versioned Heterogeneous Memory Modeling: Moving beyond static RAG to a dynamic memory system that tracks version evolution and risk.
• Defect-Trigger Feedback: A mechanism that weights memory retrieval based on historical defect recurrence and business risk.
• Risk-Aware Context Selection: A re-ranking strategy that prioritizes defect-dense paths over normal flows.
• Semantic Deduplication with Boundary Preservation: A filtering process that removes redundant cases while retaining critical boundary variations and exception branches.
• State-Consistency Verification: Validation modules that ensure generated test cases adhere to valid state transitions and schema constraints.

4. Experimental Results

The method was evaluated on a real-world E-commerce Order System containing 214 backend APIs, over 9,600 historical scripts, and 3,180 historical defects over a 14-month cycle.

Performance Metrics (vs. No-Memory Prompting):

• Effective Test Case Rate: Achieved 88.7% (±1.1%), compared to 28.1% for non-memory prompting.
• Duplication Reduction: Reduced duplication by 8.9% (±0.7%).
• High-Risk Path Coverage: Reached 91.3%.
• Defect Discovery: Identified 61 defects, a 19.6% improvement over the no-memory baseline (51 defects).

Comparative Analysis:

• The proposed method outperformed baselines including plain RAG, CODAMOSA, CAT-LM, rule templates, and human supplementary testing.
• Ablation Studies: Removing project memory dropped efficiency to 66.1% and high-risk coverage to 73.4%. Removing risk re-ranking lowered coverage to 82.1%. Removing deduplication increased duplication to 21.4%.
• Model Size: While the 70B model showed superior performance in long call-chain reasoning and assertion repair, smaller models (Llama3.1-8B, Qwen2.5-14B) remained competitive (80.4% and 83.6% respectively), suggesting the framework is not entirely dependent on massive parameter counts.

5. Significance and Claims

The paper claims that Project Memory significantly enhances an LLM's ability to understand long-term business rules, cross-interface dependencies, and historic failure patterns. By treating memory as a release-evolving control layer rather than a static appendix, the approach successfully addresses the "stagnation" of test coverage in continuous delivery environments.

The authors note that the method is particularly effective at detecting regression-sensitive failures, such as cross-interface rollbacks, coupon recovery issues, idempotency violations, and renamed field mismatches.

Limitations Acknowledged:
The approach relies on the availability of comprehensive bug records, clear interface patterns, and a specific backend architecture (e-commerce order system). The authors caution that poor documentation or complex architectures could reduce memory recall and portability. The framework is presented as a specific solution for continuous test case generation in evolving software systems, with anonymized data and scripts provided for subsequent validation.