💻 computer science

CASPER-Change-Aware Slice Prioritization for Efficient Regression Testing of LLM-based systems

This paper proposes CASPER, a change-aware slice prioritization framework that improves the efficiency of regression testing for LLM-based systems by identifying semantically consistent test slices and prioritizing them based on behavioral information from execution logs.

Original authors: Biruk Asmare Muse, Lionel Briand, Yiwei Lu, Keheliya Gallaba

Published 2026-08-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Biruk Asmare Muse, Lionel Briand, Yiwei Lu, Keheliya Gallaba

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a massive, high-tech spaceship crewed by a super-smart, but slightly unpredictable, AI navigator. This AI is brilliant at solving problems, like fixing broken engines or plotting new courses, but it's not perfect. Sometimes it makes a tiny mistake, other times it gets it completely right. Now, imagine you want to upgrade the AI's brain or change the instructions you give it. Before you launch the ship, you need to make sure the new version doesn't break anything it used to do. This is called "regression testing."

In the old days of software, testing was like checking a light switch: if it didn't turn on, it was broken. But with AI, it's more like checking a weather forecast. If the AI predicts rain for one city but gets it wrong, does that mean the whole system is broken? Or is it just a fluke? If you check every single city on the planet, it takes forever. But if you only check the big cities, you might miss a storm in a small town. The problem is finding the perfect middle ground: grouping similar cities together so you can check a few representative ones and know what's happening with the whole group. This paper tackles exactly that puzzle for AI systems.

The Problem: The "Needle in a Haystack" of AI Mistakes

When developers tweak an AI system—maybe they change the model's brain, update the instructions (prompts), or swap out a tool—it can cause "regressions." These are moments where the AI starts doing something worse than before. The tricky part is that AI is probabilistic; it might get a task right 90% of the time, but that 10% failure rate can be annoying.

If you test every single possible scenario after a change, it's too slow and expensive. If you just look at the "average" score, you might miss the fact that the AI is now terrible at a specific type of task, even if it's great at others. The authors realized that the solution lies in slicing. Imagine cutting a giant pizza (your test suite) into slices. Each slice should contain pieces of pizza that are very similar to each other (like all the pepperoni slices). If the pepperoni slice tastes bad after the oven upgrade, you know the problem is with the pepperoni, not the cheese.

The Solution: CASPER, the Smart Pizza Cutter

The paper introduces a new framework called CASPER (Change-Aware Slice Prioritization). Think of CASPER as a super-smart robot chef that does two main things:

It cuts the pizza perfectly (Slice Identification):
Instead of just guessing how to group the test cases, CASPER uses a clever evolutionary search (like a digital version of natural selection) to find the best groups. It looks at how the AI behaves during its "thought process" (its conversation logs) to see which tasks are similar. It groups tasks that share the same behavioral patterns and, crucially, tasks where the AI either consistently succeeds or consistently fails. This ensures that if one task in the group fails, the others likely will too.
It picks the most important slices to taste first (Slice Prioritization):
Once the pizza is sliced, CASPER doesn't just taste them in random order. It uses a "failure prediction model" trained on the AI's past behavior. When a change happens, CASPER looks at a few representative tasks from each slice and asks: "Based on how the AI was thinking before, does this new change look like it's going to break this specific slice?" It then ranks the slices, putting the ones most likely to be broken at the very top of the list.

How They Tested It: The Software Fixer Challenge

To see if CASPER actually works, the researchers tested it in the world of software issue resolution. Imagine an AI that is given a bug report and a bunch of code, and its job is to write a fix (a "patch"). They used a famous dataset called SWE-bench Verified, which contains 500 real-world coding problems.

They simulated different kinds of changes:

Model Changes: Swapping the AI's brain for a newer version (e.g., from one Claude model to another, or from a "Devstral" model to a "Kimi" model).
Prompt Changes: Changing the instructions given to the AI (e.g., telling it to be more careful or to use different tools).

They compared CASPER against two other methods:

Random Ranking: Just picking slices to test in a random order (like picking pizza slices with your eyes closed).
Clustering Baselines: Using standard math tools (GMM and HDBSCAN) to group the slices, which are common methods but not specifically designed for this AI regression problem.

The Results: Finding the Broken Slices Faster

The results were quite clear. When it came to cutting the pizza (identifying the slices), CASPER did a much better job than the standard clustering methods.

Consistency: The slices created by CASPER had a 97% to 98% output consistency. This means that within a single slice, the AI either got almost all the tasks right or almost all of them wrong. The standard methods only managed about 74% to 84% consistency.
Coherence: CASPER also kept the slices "coherent," meaning the tasks inside them were truly similar in how the AI approached them.

When it came to picking the slices to test (prioritization), CASPER was a game-changer.

In a scenario where developers only have time to test a small fraction of the slices (a limited budget), CASPER found the broken slices much faster than random guessing.
For some prompt changes, CASPER improved the ability to find regressions by nearly 50% compared to random ranking.
Even for model changes, where the differences were subtler, CASPER still outperformed the random baseline by a significant margin (up to 46% improvement).

What This Means

The paper suggests that by looking at how an AI thinks (its behavioral signals) rather than just what it outputs, we can group its tasks into meaningful buckets. This allows developers to test AI systems much more efficiently. Instead of running thousands of tests, they can run a few smartly chosen ones and be confident they've caught the problems.

The authors note that while CASPER worked very well in this specific domain (fixing software bugs), the method is flexible. It doesn't rely on specific types of data or pre-written descriptions, meaning it could potentially be adapted for other AI tasks, like generating summaries or answering questions. However, they also point out that the method works best when the changes to the AI are significant enough to cause noticeable shifts in behavior; if the changes are tiny, it's harder to distinguish the impact.

In short, CASPER is a tool that helps developers stop guessing which parts of their AI system might break after an update, saving time and ensuring that the AI stays reliable even as it evolves.

Technical Summary: CASPER — Change-Aware Slice Prioritization for Efficient Regression Testing of LLM-based Systems

1. Problem Definition

Regression testing for Large Language Model (LLM)-based systems faces unique challenges compared to traditional software. In conventional systems, a failure in a single test case reliably indicates a fault. However, due to the probabilistic nature of LLMs, a single instance failure does not necessarily signify a meaningful system-level regression, nor does it provide sufficient diagnostic information. Conversely, evaluating performance solely at the aggregate test suite level is too coarse-grained, failing to identify which specific behaviors are affected by a change.

The core problem is to identify "regression slices"—subsets of semantically related regression instances where the system exhibits consistent performance (either predominantly successful or predominantly failing)—and to prioritize these slices based on their likelihood of being impacted by a system change (e.g., model updates, prompt modifications, or hyperparameter tuning). This prioritization is critical for cost-effective regression testing, particularly in CI/CD pipelines where testing budgets are limited.

Existing slice identification techniques (e.g., decision-tree-based, clustering, or embedding-driven methods) are largely designed for predictive models, specific modalities (images, time-series), or classification tasks. They often rely on external corpora of slice descriptions or domain-specific architectures, making them unsuitable for the generative, text-based nature of LLM-based systems.

2. Methodology: The CASPER Framework

The authors propose CASPER (Change-Aware Slice Prioritization for Efficient Regression Testing of LLM-based systems), a two-stage framework designed to identify meaningful regression slices and prioritize them after a system change.

Stage 1: Slice Identification (CASPER-SI)

The first stage partitions the regression testing dataset into slices that satisfy two competing objectives: output consistency (instances within a slice share similar success/failure outcomes) and behavioral coherence (instances share similar behavioral characteristics).

Behavioral Signal Extraction: CASPER extracts quantitative features from agent conversation logs (reasoning steps, tool invocations, execution feedback). For the software issue resolution domain, these signals include:
- File-level recall: Alignment between generated and gold patches.
- Mean Absolute State Movement (MASM): Stability of state transitions.
- Trajectory Entropy Deviation (TED): Deviation of state-transition entropy from successful baselines.
- Testing effort, Test failure rate, Dominant File Ratio, Cross-state file overlap, and Plan compliance metrics (PPC, POC, PPF).
Multi-Objective Search: Slice identification is formulated as a multi-objective optimization problem. CASPER employs NSGA-II (a non-dominated sorting genetic algorithm) to evolve a population of candidate slices. The optimization minimizes:
1. Output Inconsistency (OI): The proportion of instances in a slice that deviate from the dominant success/failure outcome.
2. Intra-Slice Distance: The average pairwise Euclidean distance between behavioral signal vectors within a slice (maximizing coherence).
3. Inter-Slice Distance: The distance between slice centroids (maximizing separation).
Selection: The algorithm uses a knee-point analysis on the resulting Pareto front to select the optimal trade-off solution, ensuring slices are neither too granular nor too coarse.

Stage 2: Regressed Slice Ranking

The second stage prioritizes slices based on the likelihood of regression after a change (from version $V_0$ to $V_1$ ).

Failure Prediction Model: A model (Logistic Regression or SVM) is trained on $V_0$ data to map behavioral signals to a failure impact score (probability of success or distance from the decision boundary).
Representative Sampling: For each identified slice, a small subset of representative instances is selected for execution on the changed system ( $V_1$ ) using either Closest Distance Sampling (CDS) or Furthest Point Sampling (FPS).
Impact Classification: The system computes the change in failure rate ( $\Delta FR$ ) and the sampling ratio. Two independent binary classifiers (Random Forest) predict whether a slice has Regressed (R), Improved (I), or is Not Changed (NC).
Ranking: Slices are ranked first by their impact category (R > I > NC) and then by the aggregated change in failure impact score within their category. This ensures developers see the most likely regressions first.

3. Key Contributions

CASPER Framework: An end-to-end framework enabling cost-effective regression testing for LLM-based systems by identifying meaningful slices and prioritizing them based on change impact.
Evolutionary Slice Identification (CASPER-SI): A model-agnostic approach that formulates slice identification as a multi-objective optimization problem using NSGA-II. It does not require external description corpora or domain-specific model architectures, making it applicable to generative tasks.
Behavioral Signal-Based Ranking: A novel approach that learns failure prediction models from agent execution logs and uses changes in predicted failure impact to rank slices, rather than relying solely on aggregate performance metrics.
Empirical Validation: A comprehensive evaluation in the software issue resolution domain (using SWE-bench Verified) demonstrating significant improvements over clustering baselines and random ranking.

4. Evaluation and Results

The authors evaluated CASPER on the SWE-bench Verified dataset using the Mini-swe-agent system. They tested four scenarios: model changes (different families), same-family model changes, and two distinct prompt changes.

Slice Identification Quality (RQ1):
- CASPER-SI significantly outperformed clustering baselines (GMM and HDBSCAN) in Output Consistency (OC), achieving mean OC > 97% compared to ~74–84% for baselines. This indicates CASPER produces slices where the LLM's performance is highly uniform.
- In terms of Slice Coherence Metric (SCM), CASPER-SI achieved comparable or better results than baselines in most datasets, though GMM performed slightly better in the same-family model change scenario due to distributional alignment.
- Statistical tests (Kruskal-Wallis and Dunn's) confirmed that CASPER-SI's improvements in output consistency are statistically significant.
Slice Ranking Effectiveness (RQ2 & RQ3):
- CASPER consistently outperformed a random ranking baseline across all datasets and testing budgets.
- Recall@K and AUC: CASPER achieved higher Recall@K values, particularly under limited testing budgets (e.g., inspecting only the top 20-40% of slices).
- Relative Gain: The improvement was most pronounced for prompt changes (up to ~49% relative gain in AUC) and model changes (up to ~46%), while same-family model changes showed more modest gains due to subtler behavioral shifts.
- The approach proved robust to variations in failure prediction models and sampling strategies, though SVM with Closest Distance Sampling (CDS) generally performed best for same-family changes.

5. Significance and Claims

The paper claims that CASPER addresses a critical gap in the regression testing of LLM-based systems by moving beyond the binary "pass/fail" of individual instances and the overly general "suite-level" metrics.

Practical Utility: By identifying semantically coherent slices and prioritizing those most likely to regress, CASPER allows developers to focus limited debugging resources on the most relevant changes, significantly reducing the cost of regression testing in CI/CD pipelines.
Generalizability: While evaluated on software issue resolution, the authors claim the framework is model-agnostic and can be adapted to other LLM domains by customizing the behavioral signals and performance metrics.
Modesty: The authors acknowledge that the behavioral signals are engineered proxies and may not capture all aspects of agent behavior. They also note that performance varies depending on the magnitude of the behavioral shift induced by the change (e.g., same-family model updates are harder to distinguish than cross-family updates).

In conclusion, CASPER provides a structured, automated approach to organizing and prioritizing regression tests for LLM systems, offering a practical solution to the challenges of non-determinism and coarse-grained evaluation in modern AI-driven software development.