Bridging SCED and Group Designs: Demonstrating the Ratio-Metric BC-IRR Effect Size in a Meta-Analysis

In the world of education and therapy, researchers face a persistent puzzle: how to measure whether a new method actually helps a person learn or behave better. For decades, the scientific community has relied on two very different ways of gathering evidence. One approach, known as group design, gathers large numbers of people, splits them into teams, and compares the average results of the team receiving a new treatment against a team receiving standard care. This method is powerful for seeing broad trends across populations. The other approach, called single-case experimental design, focuses intensely on one individual at a time. Researchers carefully observe a single person, introduce a specific intervention, and track how that person's behavior changes over time, often looking for immediate shifts in specific actions like making a request or staying on task. While both methods are rigorous, they have historically spoken different languages. The group studies produce numbers that describe how a whole group changed, while the single-case studies produce numbers that describe how an individual changed. Because these numbers are calculated differently, it has been difficult to combine them into a single, clear picture of what works best for students with autism spectrum disorder.

This difficulty creates a blind spot for educators and policymakers. If a new technology helps a group of children improve their social skills by a certain amount, but helps a different set of children improve their communication by a different amount, how can we know if the technology is truly effective overall? The problem is not just that the studies look different, but that the tools used to measure success are incompatible. Traditional tools often confuse the size of the improvement with how much the results vary from person to person, or they change their meaning depending on how long the observation lasted. To solve this, a team of researchers led by Wen Luo at Texas A&M University set out to build a new measuring stick. They wanted a way to translate the results from single individuals and groups into the same language, allowing them to be compared side by side. They focused on a specific type of measurement called the incidence rate ratio. In plain terms, this is a simple comparison of rates: it asks how many times a behavior happened during the treatment compared to how many times it happened before the treatment started. By using this ratio, the researchers could express the effect of an intervention as a straightforward percentage increase or decrease, rather than a complex statistical score that requires specialized training to understand.

The researchers tested this new approach by revisiting a large collection of previous studies on technology-aided instruction for students with autism. This original collection, compiled by other scientists in 2017, included dozens of studies using both group designs and single-case designs. The team re-examined the data from thirty-six specific outcomes, ranging from how well children recognized facial expressions to how often they asked for help or engaged in conversation. They applied their new ratio-based method to the single-case studies and a matching ratio method to the group studies. The goal was to see if the two types of research would tell a consistent story when viewed through this new lens. The results revealed a clear, quantifiable difference in how the two designs measured success. The group studies showed that the technology interventions led to an average improvement of about 23 percent over the control conditions. In other words, students in the treatment groups performed roughly one-quarter better than those who did not receive the specific technology aid.

However, the single-case studies told a much more dramatic story. When the researchers applied the same ratio-based logic to the individual cases, they found that the interventions produced a more than fourfold increase in positive outcomes compared to the baseline. This means that for the individual students in these studies, the behavior they were trying to improve happened more than four times as often after the intervention began. The difference between a 23 percent increase and a fourfold increase is striking, and it initially seemed to suggest a contradiction. Yet, the researchers did not dismiss this gap as an error or a flaw in the data. Instead, they used the new metric to explain why the numbers looked so different. They found that the two types of studies were essentially starting from different places. The group studies often compared the new technology against a standard classroom environment or a different type of instruction, where students already had a moderate level of skill. The single-case studies, by contrast, often started with students who had very low levels of the target behavior before the intervention began. Because the starting point was so low, even a modest absolute improvement resulted in a massive percentage increase.

This distinction is crucial because it changes how we interpret the success of these interventions. The new method showed that the large numbers seen in single-case studies were not necessarily a sign that the treatment worked better for individuals than for groups, but rather that the individuals started with a much lower baseline. The ratio-based measure made this clear by focusing on the relative change rather than the absolute difference. It also highlighted a key weakness in the older methods used to compare these studies. The traditional tools often mixed the size of the improvement with the variability of the data, making it hard to tell if a large effect was real or just a result of how spread out the scores were. The new ratio approach avoided this trap entirely, offering a result that remained stable regardless of how much the scores varied or how long the observation sessions lasted. It also provided a number that anyone could understand: a fourfold increase is a tangible concept, whereas a complex statistical score is not.

The study also addressed the question of whether these different results were caused by a bias in how studies are published. It is a common concern that researchers are more likely to publish single-case studies that show huge improvements, while ignoring those with small or no effects. While the researchers acknowledged that this bias might exist, their analysis suggested it was not the whole story. The data indicated that the difference in results was largely due to the fundamental differences in how the studies were set up and what they were measuring. The group studies were testing whether the technology was better than what was already available, while the single-case studies were testing whether the technology could create a major change from a state of little to no ability. Both findings are valid, but they answer different questions. The group studies suggest that the technology offers a modest but meaningful advantage over standard practices, while the single-case studies demonstrate that the technology can be transformative for students who are starting from a very low level of functioning.

By bridging these two worlds, the researchers provided a clearer path for synthesizing evidence. They showed that it is possible to combine findings from different types of research without forcing them into a single, misleading number. The new method allows scientists to see the full picture: that technology-aided instruction is effective, but its impact looks different depending on the context and the starting point of the students. The study concludes that no single number can capture the entire truth of an intervention's effectiveness. Instead, researchers and practitioners should look at both the relative change and the absolute change to understand the full scope of an intervention's power. This approach ensures that decisions about education and therapy are based on a complete and accurate understanding of the evidence, rather than on a partial view that might miss the nuances of how different students respond to help. The work demonstrates that when we use the right tools to measure change, we can see not just that something works, but exactly how and for whom it works best.