💻 computer science

The Impact of Documentation on Test Engagement in Pull Requests in OSS

This study investigates the relationship between testing documentation and contributor behavior in open-source software, finding that more comprehensive documentation—particularly regarding how to run and write tests—is positively correlated with increased testing engagement in pull requests.

Original authors: Teal Amore, Nathan Berman, Siyuan Jiang

Published 2026-04-28

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Teal Amore, Nathan Berman, Siyuan Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The "Instruction Manual" Effect: Why Good Directions Make Better Builders

Imagine you walk into a massive, community-run LEGO workshop. There are thousands of bricks everywhere, and people are constantly building new parts to add to a giant, shared castle.

In this workshop, there is one golden rule: If you build a new tower, you should also build a small "test" model to make sure the bricks don't fall over.

However, most people in the workshop are volunteers. They are busy, they are tired, and they often forget to build those little test models. Usually, the "bosses" (the project maintainers) only notice a mistake after a tower has already collapsed. They then have to run over and tell the builder, "Hey, you should have tested that!" This is reactive—it’s like trying to put out a fire after the building is already burning.

The Big Question:
The researchers in this paper wanted to know: What if we gave people the instructions before they started building? If we provided a clear manual on "How to build a test model" and "How to check if your bricks are sturdy," would people actually do it more often?

The Researchers' "Measuring Tape" (The TER)

It’s hard to measure "effort" in a giant workshop. You can't watch every single person every second. So, the researchers invented a clever shortcut called the Test Engagement Ratio (TER).

Think of the TER like a "Double-Duty Score."

If a builder brings in a new piece of code (a new LEGO brick), the researchers check: Did they also bring in a test (a little stability model)?
If they did both, they get a high score. If they only brought the brick and skipped the test, their score goes down. It’s a way to see how often people are "double-dipping" into testing.

What They Found (The Results)

The researchers looked at 160 different "workshops" (Open Source software projects) and found some very cool things:

1. The "Manual" actually works! 📖
They found that projects with more detailed "How-To" guides had much higher "Double-Duty Scores." In simple terms: The more helpful the instructions, the more likely people were to include tests. It wasn't a massive explosion of testing, but there was a clear, steady connection.

2. Not all instructions are created equal. 🛠️
They discovered that some chapters in the manual are more important than others.

The Winners: Instructions like "How to Run a Test" and "How to Write a Test" were the superstars. These are the practical, "hands-on" guides.
The Losers: General "Best Practices" or "Tips" didn't seem to move the needle much. It turns out, builders don't want vague advice; they want to know exactly which buttons to press.

3. Popularity doesn't matter. ⭐
You might think that a "famous" workshop (one with lots of followers and stars) would have better testing. Surprisingly, no! Whether a project was a superstar or a small hobby project, the documentation was what mattered, not the fame.

The Bottom Line

If you want to build a high-quality digital world, don't just wait for things to break and then complain. Write a clear, practical manual.

If you tell your volunteers exactly how to run a test and how to write one, they are much more likely to do it. Good documentation acts like a proactive roadmap, guiding builders to do the right thing before the first brick is even laid.

Technical Summary: The Impact of Documentation on Test Engagement in Pull Requests in OSS

Problem Statement

In Open-Source Software (OSS) development, ensuring that contributors include tests alongside code changes is critical for maintaining software quality. However, most current interventions to encourage testing—such as code coverage metrics or reviewer feedback—are reactive, meaning they occur only after a pull request (PR) has already been submitted. This reactive nature fails to proactively educate contributors on project-specific testing standards. The authors investigate whether proactive documentation regarding testing can serve as an effective tool to increase testing engagement among contributors.

Methodology

The study analyzes data from 160 OSS repositories (derived from a dataset by Falcucci et al.). The researchers employed several novel and standard techniques:

Metric Development (Test Engagement Ratio - TER): To bypass the technical difficulty of building and running diverse codebases to calculate traditional coverage, the authors proposed TER. This metric measures the proportion of PRs that modify both production code and test files relative to the total number of PRs that modify production code.
- $TER = \frac{\sum I_{both}(r)}{\sum I_{prod}(r)}$
Documentation Quantification: The authors used a Test Documentation Diversity Score, which counts the number of specific testing categories (out of 10 possible types, such as "How to Run Tests" or "Unit Testing") present in a repository's contribution guidelines.
File Classification: A heuristic-based approach was used to classify files as either "production" or "test" code, including support for .spec files common in JavaScript environments.
Statistical Analysis: The researchers used Spearman’s rank correlation coefficient ( $\rho$ ) to analyze the relationships between documentation diversity, repository metadata (stars, forks, watchers), and TER, as the data distributions were non-normal.
Validation: To ensure TER was a valid proxy for testing activity, they correlated it with the established Test Code Ratio (the ratio of non-empty lines of code in tests vs. total code).

Key Contributions

Introduction of the TER Metric: A scalable, non-intrusive proxy for measuring testing engagement that does not require executing the repository's test suite.
Empirical Evidence of Proactive Documentation: The first study to provide empirical evidence linking the comprehensiveness of testing documentation to actual contributor testing behavior.
Identification of High-Impact Documentation Types: Pinpointing which specific types of documentation (e.g., "How to Run Tests") are most strongly associated with increased testing.

Results

RQ1 (Documentation vs. Engagement): A weak but statistically significant positive correlation was found between documentation comprehensiveness and TER ( $\rho = 0.36, p < 0.001$ ). In repositories with higher PR activity, this relationship strengthened to a moderate correlation ( $\rho = 0.44$ ).
RQ2 (Specific Categories & Metadata):
- The strongest correlations with testing engagement came from documentation on "How to Run Tests" ( $\rho = 0.34$ ) and "How to Write Tests" ( $\rho = 0.36$ ).
- Other significant categories included Unit Testing and Integration Testing.
- Repository popularity (stars, forks, watchers) had no significant relationship with TER, suggesting that testing engagement is driven by project guidance rather than project fame.
RQ3 (Metric Validation): TER showed a moderate positive correlation with the standard Test Code Ratio ( $\rho = 0.52, p < 0.001$ ), validating its utility as a measure of testing activity.

Significance

This research shifts the focus of software quality maintenance from reactive correction (fixing tests after a PR is opened) to proactive guidance (providing clear instructions before a PR is even started). The findings suggest that maintainers can improve software reliability by investing in specific types of documentation—specifically instructions on how to execute and author tests. This provides a low-cost, high-impact strategy for OSS projects to foster a better testing culture among a diverse and often transient contributor base.