What Do Contribution Guidelines Say About Software Testing?

Imagine open-source software projects as massive, bustling community kitchens. Anyone can walk in, pick up a knife, and start chopping vegetables (writing code) to help cook a better meal. But to keep the kitchen running smoothly, the head chefs (project maintainers) leave a Rulebook on the counter. This Rulebook tells new helpers how to wash their hands, where to find the ingredients, and how to submit their chopped veggies without messing up the soup.

This paper is like a team of researchers who went into 200 of these famous community kitchens (specifically those using Python and JavaScript languages) to read the Rulebooks and see what they actually say about testing.

In the world of cooking, "testing" is like tasting the dish before serving it to make sure it doesn't taste like soap. The researchers wanted to know: Do the Rulebooks actually teach new helpers how to taste the food, or do they just assume everyone knows how to do it?

Here is what they found, broken down simply:

1. Most Kitchens Have a "Tasting" Section (But Not All)

The researchers found that 78% of the kitchens had a specific section in their Rulebook dedicated to tasting (testing).

• Where is it? Most of the time (58%), it's in a file literally named "CONTRIBUTING" (the main instruction manual). Sometimes it's in a separate, fancy brochure (external docs), and rarely, it's just scribbled on the main menu (README).
• The Gap: About 22% of the kitchens had no instructions on tasting at all. If you walked into those kitchens, you'd have to guess how to check if your food was good.

2. The "How-To" Imbalance: Running vs. Writing

When the Rulebooks did talk about tasting, they were very good at one thing but bad at another.

• The "How to Run" (83.5%): Most Rulebooks clearly said, "Here is the magic spell (command) to taste the whole pot." It's like saying, "Press this button to check the flavor."
• The "How to Write" (37%): Far fewer Rulebooks explained how to actually create a new taste test. It's like saying, "Press the button," but not teaching you how to make a new spoon or how to know what a "bad" flavor looks like.
• The Result: Helpers know how to check the existing food, but they are often left guessing on how to create their own tests for the new ingredients they added.

3. The "Test Pyramid" Problem

In software, there are different levels of testing, like different types of taste tests:

• Unit Tests (71%): These are like tasting a single ingredient (e.g., "Does this carrot taste sweet?"). The Rulebooks talked about this a lot.
• Integration Tests (20.5%): These are like tasting how the carrot and the onion work together in the pot. The Rulebooks rarely mentioned this.
• End-to-End Tests (15.5%): This is like tasting the final, fully cooked meal to see if the whole dish works. The Rulebooks almost never talked about this.

The Metaphor: The chefs are very worried about the carrots tasting right, but they rarely tell the helpers how to check if the whole stew is going to burn or if the flavors mix well.

4. The Missing "Secret Weapons"

The researchers also looked for advanced cooking tips that help make testing easier and more reliable:

• Mocking (9.5%): Sometimes, you can't taste the real ocean water in your soup; you have to use a fake salt shaker to simulate it. This is called "mocking." Only 1 out of 10 kitchens had instructions on how to use these fake tools.
• Coverage (25.5%): This is a scorecard showing how much of the recipe was actually tasted. Only about a quarter of the kitchens told helpers what score they needed to aim for.
• Best Practices (9%): General tips like "Always taste before serving" were very rare.

The Bottom Line

The paper concludes that while most open-source projects want their helpers to test their work, the instructions they leave behind are unbalanced.

They are great at saying, "Here is how you run the test," but they are often silent on:

• How to write a new test.
• How to test complex interactions (integration).
• How to test the whole system (end-to-end).
• How to use advanced tools (mocking) or set quality goals (coverage).

The Takeaway: If you are a new helper in one of these kitchens, you might know how to press the "taste" button, but you might be left alone to figure out how to actually create a taste test for your new recipe, or how to ensure your new dish won't ruin the whole meal. The authors suggest that project leaders need to write clearer, more complete instructions so helpers don't have to guess.