Feature Toggle Dynamics in Large-Scale Systems: Prevalence, Growth, Lifespan, and Benchmarking
This paper presents a longitudinal analysis of over 4,000 feature toggle events in Kubernetes and GitLab, revealing that removals lag behind additions leading to growing technical debt and de facto permanent toggles, and proposes a new benchmarking framework with empirically derived metrics to help practitioners assess and improve their toggle management practices.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are building a massive, complex city (your software system). To keep things running smoothly while you are still constructing new buildings, you put up temporary scaffolding. In the software world, this scaffolding is called a "Feature Toggle."
Think of a toggle as a light switch in the wall.
- Switch ON: The new feature is visible to users.
- Switch OFF: The feature is hidden, even though the code is already there.
Developers love these switches. They let them build a new bridge, test it with a few cars, and hide it from the public if it wobbles, all without tearing down the whole city.
The Problem:
The paper argues that while these switches are great for building, they are terrible if you forget to take them down. Just like leaving scaffolding up on a building for 10 years, forgotten switches clutter the city. They make the streets confusing, increase the risk of accidents (bugs), and make it hard to know which roads are actually open.
The researchers asked: "How many of these switches are we leaving up? How long do they stay up? And are we cleaning them up fast enough?"
The Study: Two Giant Cities
To find the answers, the researchers looked at the "construction logs" of two massive digital cities:
- Kubernetes: A huge city built with Go (a strict, organized language). It's like a city with a very slow, quarterly construction schedule.
- GitLab: A massive city built with Ruby (a fast, flexible language). It's like a city that renovates its streets every single month.
They tracked over 4,000 switches over many years. Here is what they found, explained simply:
1. The "Add More Than You Remove" Trap
In both cities, the construction crews were adding new switches faster than they were removing old ones.
- Kubernetes: For every 100 new switches added, they only removed about 65.
- GitLab: For every 100 new switches added, they removed about 87.
- The Result: The "junk drawer" of switches is slowly filling up. In Kubernetes, the pile of unused switches is growing by about 35% every year; in GitLab, by 13%. It's like buying a new toy every week but only throwing away one old toy every month. Eventually, you run out of room.
2. The "Time Travel" Difference
The researchers found that "how long is too long?" depends entirely on the city's rhythm.
- In Kubernetes (Slow City): A switch staying up for 2 years is considered "normal." The median lifespan is 734 days.
- In GitLab (Fast City): A switch staying up for 6 months is considered "normal." The median lifespan is only 185 days.
- The Analogy: If you left a "Road Work Ahead" sign up for 2 years in a city that updates its traffic lights every month, that sign is a disaster. But in a city that only updates its traffic lights once a year, that same sign might be perfectly fine. You can't compare the two cities using the same clock.
3. The "Ghost Switches" (The Permanent Ones)
The most worrying discovery was the "Ghost Switches." These are switches that have been left on so long that they have effectively become permanent parts of the building, even though they were supposed to be temporary.
- In Kubernetes, 8 switches have been up so long they broke the record for the longest time a switch was ever supposed to stay up.
- In GitLab, 25 switches are in the same boat.
- The Metaphor: Imagine a "Do Not Enter" sign that was put up for a road closure in 2019. It's still there in 2026. The road has been open for years, but the sign is still blocking the view. These are the "de facto permanent" toggles that turn into technical debt.
The Solution: A "Health Check" Dashboard
Since every city is different, you can't just say "All switches must be removed in 6 months." The researchers created a Benchmarking Framework—think of it as a Health Check Dashboard for your software city.
They proposed 5 Key Metrics to tell you if your switch management is healthy:
- Churn Rate: How busy are you? (Are you adding/removing switches constantly, or is everything static?)
- Net Accumulation: Is your junk drawer getting fuller? (Are you adding more than you remove?)
- Cleanup Ratio: What percentage of your switches do you actually take down?
- Toggle Density: How many switches are there per mile of road? (Too many switches make the city confusing).
- Normalized Lifespan: How long does a switch stay up relative to your city's construction schedule? (This is the most important one, as it accounts for the "Slow City" vs. "Fast City" difference).
The Takeaway
The paper concludes that there is no single "right" way to manage switches.
- Kubernetes uses a "Slow and Steady" approach: Fewer switches, but they stay up for a long time (like a multi-year construction project).
- GitLab uses a "Fast and Furious" approach: Hundreds of switches, but they are removed very quickly (like a pop-up event).
The Lesson for Everyone:
Don't just look at the number of switches. Look at the rhythm of your team.
- If you are a fast-moving team, you need to clean up fast.
- If you are a slow-moving team, you can keep switches up longer, but you must still have a plan to remove them.
The researchers built a free, interactive dashboard (like a car's dashboard) where teams can plug in their own numbers. It tells them: "Hey, your 'junk drawer' is getting too full," or "Your cleanup speed is great, keep it up!"
In short: Feature toggles are great tools, but like any tool, if you leave them lying around the workshop forever, they become a hazard. This paper gives us the ruler and the clock to measure when it's time to clean up.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.