How Developers Use Relation Chains in Gerrit-Based Review Ecosystems: An Empirical Study Across Three Open-Source Ecosystems

Imagine a world where building software is like constructing a massive, intricate castle. In this world, developers don't just throw bricks at a wall and hope they stick; they use a rigorous system called Code Review. Before any new brick (or line of code) is permanently added to the castle, a team of inspectors checks it for cracks, ensures it fits the design, and makes sure it doesn't break anything else. This process is vital for keeping the castle standing tall and safe.

Sometimes, however, a project is too big to be one single brick. It's a whole tower that needs to be built. In the past, developers might have tried to build the whole tower at once, but that's hard to inspect. So, they started breaking it down into a sequence of smaller, connected steps. In the software world, specifically within a tool called Gerrit, these connected steps are called Relation Chains. Think of a relation chain like a set of dominoes standing in a line: while you can't knock over the third one until the second falls, and the second until the first falls, the inspectors can check all of them at the same time, but the castle can only be completed in order. The whole chain is linked; if the first domino (the "base") is shaky, the whole line is in trouble. Understanding how these chains work is crucial because if the system is too slow or confusing, developers might get stuck waiting for hours, or the castle might get built with hidden cracks.

---

The Domino Effect: How Developers Actually Use Code Chains

This paper is a deep dive into how developers in three massive open-source communities (OpenStack, Wikimedia, and ONAP) use these "Relation Chains" to build software. The researchers looked at nearly 30,000 chains and over 400,000 individual code changes to see how these linked dominoes behave in the real world. They wanted to know: Are these chains common? Do they make the review process faster or slower? And what happens when you try to fix one domino in the middle of the line?

The Chains Are Everywhere (and Getting Bigger)

First, the study found that these chains are not a rare, niche trick; they are a standard way of working. Depending on the project, anywhere from 5% to 49% of all code changes are part of a chain. In fact, in 14 out of the 15 projects they studied, the use of these chains is actually increasing over time. Developers are realizing that breaking big tasks into linked, smaller pieces is the way to go.

Most of these chains are short, usually just a pair of dominoes (a base change and one dependent change). However, some projects have chains that stretch incredibly deep. The researchers found chains with up to 98 members! One project even had a single auto-generated chain with nearly 60,000 members, though that was a special case of automated configuration, not human writing.

The "Middle" is the Bottleneck

Here is where it gets interesting. The researchers discovered that being in the middle of a chain is the hardest job. If you are the first domino (the "base"), you just have to wait for your own review. If you are the last one (the "top"), you just wait for the ones before you. But if you are in the middle, you are stuck in a squeeze. You are blocked by the domino before you (waiting for it to be approved) while simultaneously blocking the dominoes after you.

Because of this "squeeze," changes in the middle of a chain take significantly longer to get approved. The study found that, on average, chain members take 2.6 times longer to merge than single, isolated changes of the same size. This delay isn't because the code is worse; it's because of the "synchronization overhead." Even though the inspectors (reviewers) can check the bricks in parallel, the actual merging into the castle must happen one by one, from the bottom up. Every time a change in the chain is tweaked, it often forces the whole line to be re-checked, re-tested, and re-ordered, creating a bottleneck where the middle changes wait for the ones below them while also holding up the ones above.

The "CI Amplification" Monster

The paper also highlights a phenomenon they call the CI amplification effect. "CI" stands for Continuous Integration, which is like a robot army that automatically tests every new brick to make sure it doesn't break the castle. In projects with strict rules (like OpenStack), every time a developer updates a change in a chain, the robot army has to re-test that change and all the changes that depend on it.

The study found that chain members trigger 10 to 23 automated test jobs, whereas a single, isolated change might only trigger fewer than two. It's like if you had to re-test your entire house every time you changed a single lightbulb. This creates a massive amount of extra work for the computers and delays for the humans.

The "Foundation Effect"

One of the most fascinating findings is what the authors call the Foundation Effect. They discovered that the amount of effort spent on the first domino (the base) predicts how much effort will be spent on all the dominoes after it.

If the base change gets a lot of attention, lots of comments, and many rounds of revision, the entire chain tends to follow suit. The researchers found a strong link (a correlation of 0.43 to 0.61) between the activity on the base and the activity on the descendants. It's as if the "vibe" of the first domino sets the tone for the whole line. If the foundation is shaky and requires a lot of fixing, the whole tower takes longer to build. Conversely, if the base is solid and gets approved quickly, the rest of the chain tends to flow smoothly.

Chains Are Not Static

Finally, the paper reveals that these chains are not rigid structures. About 33.5% of the changes in a chain undergo a "structural evolution" before they are finally merged. This means the connection between the dominoes changes while they are being reviewed. A developer might decide to detach a change from its parent and link it to a different one, or the whole chain might get reorganized.

This adds another layer of complexity: the map of the chain is constantly shifting. Sometimes, a chain can sit dormant for a long time. The study found that the gap between when one part of a chain is submitted and when it finally merges can stretch up to 2.85 years in some cases!

What This Means for the Future

The authors conclude that current tools for reviewing code often treat every change as an isolated event, like looking at a single brick without seeing the wall it belongs to. This paper suggests that we need to change our tools to understand the "chain" as a whole unit.

They suggest that if we focus our attention on the base of the chain (the first domino), we can save a massive amount of time for the rest of the chain. If the foundation is solid, the whole structure moves faster. They also propose that tools should be smarter about the "middle" changes, perhaps prioritizing them to unblock the rest of the line.

In short, building software with relation chains is like conducting a complex orchestra. If the conductor (the base) is off-beat, the whole orchestra struggles. But if the conductor is clear and the musicians (the tools) understand how the instruments are linked, the music flows much faster. The study suggests that by understanding these connections, we can stop waiting in line and start building castles much more efficiently.