ReproBreak: A Dataset of Reproducible Web Locator Breaks

Technical Summary: ReproBreak

Problem Statement
Automated GUI testing frameworks like Cypress and Playwright rely heavily on element locators (e.g., IDs, CSS selectors, XPath) to interact with web applications. A "locator break" occurs when structural changes to the Application Under Test (AUT)—such as renaming an element ID or restructuring the DOM—cause a locator to fail, even if the underlying functionality remains correct. While locator fragility is a well-documented challenge in GUI testing, existing research and datasets have primarily focused on Selenium. There is a significant gap in the literature regarding large-scale evaluation of locator fragility specifically for Cypress and Playwright, which utilize different internal mechanisms for querying and interacting with elements compared to Selenium's external driver model. Consequently, there is no existing dataset to empirically evaluate or compare locator repair techniques and robustness strategies for these modern frameworks.

Methodology
The authors constructed ReproBreak, a dataset of reproducible locator breaks, through a three-stage automated pipeline applied to open-source repositories:

1. Data Source and Identification: The study began with the E2EGit dataset, filtering for 374 repositories using Cypress, Playwright, or both. From these, 359 repositories were analyzed. The authors identified test files by filtering for specific extensions (e.g., .spec.ts) and framework-specific keywords (e.g., cy.get for Cypress, page.locator for Playwright). Using git log and git diff, they identified commits that modified test files and extracted code hunks containing locator changes ($L_c$).
2. Docker-Based Environment Setup: To validate whether a locator change constituted a break, the authors needed to execute tests in a consistent environment. They focused on the top 10 projects with the highest number of locator changes. Due to manual effort requirements and dependency issues, they successfully created reproducible Docker environments (containing Dockerfile and execution scripts) for the top 4 projects.
3. Locator Break Validation: For each identified locator change ($L_c$) in the validated projects, the authors performed a two-step verification:
   • Step 3a: Execute the test with the new locator on the updated application. If it passes, the new locator is valid.
   • Step 3b/c: Replace the new locator with the old locator in the same commit and re-execute. If the test fails, the change is classified as a locator break ($L_b$). If it passes, it is classified as a non-breaking locator change ($L_{nb}$), indicating the update was prophylactic.

Key Contributions

• ReproBreak Dataset: A structured relational SQLite database containing 449 confirmed, reproducible locator breaks across four major open-source projects. The schema links locator changes to specific commits, repositories, and reproduction scripts.
• Automated Reproduction Scripts: The authors provide reproduce.py scripts that allow researchers to automatically set up the Docker environment, inject the old locator, and confirm the break. This moves beyond static snapshots to dynamic, executable validation.
• Framework-Specific Analysis: The dataset specifically targets Cypress and Playwright, addressing the lack of resources for evaluating fragility in these frameworks compared to Selenium.

Results

• Scope: The analysis covered 359 repositories (189 Playwright, 154 Cypress, 16 both).
• Locator Changes: The pipeline identified a total of 9,572 locator changes across the repositories.
• Validation: After filtering for projects where a reproducible Docker environment could be established, the authors validated 580 locator changes.
• Breaks Identified: Of the validated changes, 449 were confirmed as locator breaks ($L_b$), while 131 were non-breaking changes ($L_{nb}$).
• Distribution: The dataset includes data from four projects: ghiscoding/angular-slickgrid, nasa/openmct, tryghost/koenig, and microsoft/playwright.

Significance and Claims
The paper positions ReproBreak as a foundational artifact for advancing research in GUI test maintenance for modern frameworks. The authors claim that the dataset enables:

• Evaluation of Robustness Techniques: Allowing researchers to test if Selenium-based resilient locator generation techniques (e.g., Robula+, Sidereal) are applicable to Cypress and Playwright.
• Fragility Score Assessment: Providing a basis to adapt and validate fragility scores originally designed for Selenium to these new frameworks.
• Repair Approach Benchmarking: Enabling the assessment of locator repair tools (e.g., Similo, Vista) and the development of LLM-based agents for automated repair.
• NL-Based Testing Research: Offering a large-scale dataset to evaluate the robustness of Natural Language-based testing approaches against locator breaks.

The authors emphasize that the inclusion of automated reproduction scripts is a critical differentiator, allowing researchers to actively confirm breaks and test repair techniques in a real execution environment rather than relying on static analysis. They note that while the current dataset covers four projects, the methodology is extensible to other frameworks by defining new locator regex patterns.