A Meta-Study on Replication Papers in Usable Security & Privacy
This mixed-method study analyzes 24 replication papers in usable security and privacy to reveal the field's current lack of clear guidelines, the prevalence of modified rather than exact replications, and the significant role of temporal and contextual factors, ultimately offering practical recommendations to strengthen replication practices.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Science relies on a simple but powerful idea: if you repeat an experiment and get the same result, you can trust the finding. This process, called replication, is the bedrock of reliable knowledge. In the field of usable security and privacy, researchers study how real people interact with security tools, from password managers to privacy settings. Because human behavior is messy and changes over time, confirming that a study's results hold true in different situations is vital. Without these checks, the community risks building safety guidelines on shaky ground. However, for a young and fast-growing field, the rules for how to repeat a study have been unclear, leaving researchers unsure of what counts as a valid repetition and journals unsure of what to accept.
A team of researchers from the Karlsruhe Institute of Technology and the University of Southern Denmark set out to map the current state of this practice. They wanted to know if the community was actually repeating studies, how those repetitions were being done, and what motivated the scientists to do the work. To find the answers, they looked at the official guidelines of thirteen major conferences and workshops where security and privacy research is shared. They then conducted a systematic search of papers published between 2016 and 2025, hunting for studies that claimed to repeat previous work. This search yielded a specific set of twenty-four relevant papers. The team then analyzed these papers using a detailed framework to categorize exactly how they differed from the original studies, and finally, they sent a survey to the authors of these papers to ask why they chose to replicate the work in the first place.
The investigation revealed that while the idea of repeating research is widely accepted, the practice is surprisingly rare. Out of thousands of papers published across the thirteen venues over nearly a decade, the researchers found only twenty-four that qualified as replications of user studies. This scarcity suggests that despite the community acknowledging the importance of replication ten years ago, it has not yet become a standard part of the research workflow. The analysis showed that the venues themselves are not always clear on what they want. Only about half of the conferences explicitly encouraged authors to submit replication papers, and even among those, most provided no specific instructions on how to write them or what details to include. One major venue, SOUPS, did offer specific categories for replication, but even there, the guidelines have shifted over time, dropping specific definitions in favor of broader descriptions.
When the researchers examined the twenty-four papers they found, they discovered that strict, exact repetitions simply do not exist in this field. In an exact replication, a scientist would try to copy the original study perfectly, using the same people, the same tools, and the same questions, just to see if the numbers come out the same. None of the papers in this study followed that path. Instead, almost all of them were "conceptual" replications, meaning the researchers changed multiple aspects of the original work. About two-thirds of the studies altered more than one major part of the experiment. They might have tested a different group of people, such as swapping university students for older adults, or they might have changed the method, like moving a study from a computer lab to an online survey. In many cases, the way the data was analyzed was also different. The researchers noted that it was often difficult to tell exactly what had changed just by reading the papers, as authors did not always clearly list their modifications.
The survey sent to the authors provided insight into why they chose to do this work. The most common reason was to see if the original findings still held true as time passed and technology evolved. Security threats and user habits change quickly, so a result from five years ago might not apply today. Another strong motivator was efficiency; researchers often reused existing materials or methods to save time and resources, allowing them to focus on new questions rather than building everything from scratch. Some authors also wanted to confirm that results could be generalized to different cultures or demographics. Interestingly, the researchers found that many studies were motivated by a desire to validate earlier results, yet the papers themselves often lacked a clear statement explaining how they were connected to the original work.
Based on these findings, the authors propose a set of practical steps to improve the situation. They suggest that conference organizers should provide clearer guidelines for authors and reviewers, explicitly stating what constitutes a replication and what details must be included. They recommend that authors of future replication studies use a simple table to list exactly how their work differs from the original, covering the people involved, the methods used, and the analysis performed. This transparency would help reviewers and readers understand the value of the new study. The goal is not to force researchers to copy studies perfectly, but to ensure that when they do repeat research, they do so in a way that is clear, honest, and useful for the entire community. By making these practices standard, the field can move toward a stronger, more reliable foundation for understanding how people interact with security and privacy.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.