Multiverse Analyses and International Large-Scale Assessments: An example with TALIS 2018

Imagine you are a detective trying to solve a mystery, but instead of a single crime scene, you have a massive, glittering warehouse filled with millions of clues. This is what happens in the world of "big data" science, where researchers study huge groups of people—like thousands of teachers from around the globe—to understand how things work. But here's the tricky part: when you have so many clues, you can arrange them in a million different ways to tell a story. You might decide to focus only on the red clues, or ignore the ones from the left side of the room, or group them by size instead of color. Each of these choices is like taking a different path through a garden. If you take one path, you might find a hidden treasure; if you take another, you might find a dead end. This idea is called the "garden of forking paths," and it suggests that sometimes, the story we tell depends more on which path the detective chose to walk than on the actual clues they found.

This paper, written by Mark White from the University of Oslo, dives deep into this problem using a massive dataset called TALIS 2018, which surveyed teachers from many countries. The author wanted to see if the "story" told by a famous, highly-cited study about teacher training (called "induction") was actually true, or if it just happened to be the result of one specific path through the garden. The paper doesn't just look at one way of analyzing the data; instead, it tries out nearly half a million different ways to crunch the numbers. It's like asking 500,000 different detectives to solve the same case using slightly different rules and seeing if they all find the same culprit. The goal isn't to say the original study was "wrong," but to show that the results can change wildly depending on the choices made along the way, making it hard to know for sure what the "real" answer is.

The Garden of a Million Paths

Think of the original study the author is investigating as a map to a treasure chest. The map claimed that certain teacher training activities—like team teaching or keeping a diary—were the keys to making teachers happier and better at their jobs. The researchers who drew this map used a specific set of rules to find the treasure: they looked at American teachers, they included only full-time workers, and they used a specific math formula to connect the training to the results. They found a shiny treasure chest and said, "Look! Training works!"

But Mark White wondered: "What if we changed the rules just a little bit?" What if we looked at teachers in Australia or New Zealand instead of just America? What if we only looked at teachers in their very first year of teaching? What if we changed the math formula slightly? To answer this, White didn't just guess; he built a "multiverse." In science, a multiverse analysis is like running a simulation where you test every single reasonable combination of rules at once.

In this case, White identified 13 different "knobs" or choices that a researcher could turn. These included decisions like:

• Who counts? (Should we include only new teachers, or all teachers? Should we only look at America, or other English-speaking countries too?)
• What numbers do we use? (Should we adjust the math to account for how many people were surveyed, or just count everyone equally?)
• How do we measure the training? (Should we look at the whole training program, or break it down into tiny pieces?)

When you multiply all these choices together, you get a staggering number: 497,664 different possible ways to analyze the data. That's nearly half a million different paths through the garden. White ran the analysis for every single one of these paths.

The Shifting Treasure

The results were a bit of a shock. When White looked at the "multiverse" of results, he found that the treasure chest wasn't in the same place for every path.

In some of the 497,664 analyses, the training activities looked like a magic wand that made teachers incredibly happy and effective. In other analyses, the exact same training activities looked like they were actually hurting teachers, making them less effective. In many other paths, the training seemed to do nothing at all.

The paper shows this with a giant graph (Figure 1 in the original text). Imagine a line of dots stretching from far left to far right. The dots on the far left represent analyses where training was a disaster. The dots on the far right represent analyses where training was a miracle. The dots in the middle represent analyses where it didn't matter. The scary part is that there are green dots (meaning "statistically significant" or "real") on both sides. This means that for almost every specific training activity and every specific teacher outcome, a researcher could find a "reasonable" way to prove it was a huge success, and another "reasonable" way to prove it was a huge failure.

The original study had picked one specific path through this garden and found a positive result. But White's multiverse analysis suggests that this result wasn't necessarily the "true" truth. It was just one of hundreds of thousands of possible truths, and the choice of which path to take was somewhat arbitrary.

Why This Matters

This paper doesn't say that teacher training is useless. It doesn't say the original authors were incorrect. Instead, it highlights a subtle but huge problem in how we do science. When a research question is a bit vague—like "Does training help teachers?"—there are so many ways to answer it that the answer often depends on the choices the researcher makes before they even start looking at the data.

The author calls this the "garden of forking paths." If you don't know which path you're on, you can't be sure if the treasure you found is real or just a result of the specific path chosen. The paper suggests that when we see a study with a single, clear result, we should be a little more curious. We should ask, "What if they had chosen a different path?"

By running nearly half a million analyses, this paper shows us that the "truth" about teacher training is much more complex and uncertain than a single headline suggests. It doesn't give us a new, better answer; instead, it gives us a much clearer picture of how unsure we really are. It's a reminder that in the garden of science, there are many paths, and sometimes, the view changes completely depending on which one you choose to walk.