How can digital and AI-enabled behaviour change interventions be evaluated while they are changing? HUMMBER: identifying what changed, for whom, why and with what effect
This paper introduces HUMMBER, a human-centred, reusable evaluation framework designed to continuously assess digital and AI-enabled behaviour change interventions as they evolve, demonstrating its computational feasibility and ability to trace changes, identify affected groups, and determine causal effects through a wholly synthetic proof-of-concept study.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine trying to steer a ship through a storm while the map, the wind, and the crew are all changing at the same time. This is the reality for modern health programs that live on smartphones and wearables. These digital tools are designed to help people adopt healthier habits, like exercising more or eating better. Some of the newest versions even use artificial intelligence to guess what a person needs and to change their advice in real time. But this constant shifting creates a problem for the people who try to measure if these tools actually work. If a program changes its features every week, or if a user's life circumstances shift, a simple "before and after" check becomes misleading. A person might end up in a good place, but the path they took to get there could be different from someone else's, or the tool might have worked for one group but failed for another. To truly understand these moving targets, scientists need a way to watch, listen, and ask questions while the program is running, rather than just waiting for the final result.
A team of researchers has proposed a new way to handle this complexity, called HUMMBER. This approach is not a new app or a medical treatment; it is a set of rules for how to study these changing digital tools. The researchers wanted to create a method that could track what changes, who is affected, under what conditions, and why. Crucially, this method keeps a human in charge of the final decisions. The artificial intelligence can offer suggestions and gather data, but the method ensures that a human evaluator decides what the data means and whether to act on it. The team tested this idea by building a completely computer-generated simulation. They created a fake digital health program and populated it with 600 virtual people, then ran the program for 26 weeks. They introduced specific problems, such as some people having poor internet access or the program becoming too difficult, to see if their new method could spot these issues accurately.
The simulation revealed that looking only at the final outcome is often a trap. In the computer world, the researchers created five different groups of virtual people. By the end of the 26 weeks, all five groups had reached almost the exact same level of success, with roughly 70 percent of them sticking to the program. However, the journey each group took was completely different. One group struggled because the technology failed them, another because the goals felt too hard, and a third because they simply lost interest. If the researchers had only looked at the final numbers, they would have missed these vital differences. The HUMMBER method, however, successfully tracked these separate paths. It showed that while the destination was the same, the reasons for getting there—or failing to get there—were unique to each group. This proves that a method is needed to look at the whole story, not just the ending.
The study also highlighted how fairness can be hidden in the data. The researchers programmed a scenario where people with limited digital access had fewer chances to receive help from the tool. The method caught this immediately. It showed that these participants had fewer opportunities to get support and more missing data, even though the help itself worked just as well for them as for everyone else. Without this detailed tracking, a researcher might have wrongly concluded that the tool didn't work for people with poor internet. Instead, the method clarified that the problem was not the tool's effectiveness, but the lack of access to it. This distinction is vital for ensuring that health tools help everyone, not just those with the best technology.
Another key finding was about how to ask people for information without annoying them. The simulation tested different ways of checking in with users: asking them immediately, waiting a while, or letting them choose when to answer. The results showed that letting the user pick the time was the most efficient. It gathered the most useful information for every minute the user spent answering questions. This suggests that respecting a person's time and choice can actually improve the quality of the data collected, rather than just being a nice gesture. The method also tested whether the system could figure out why things changed. When the simulation started with a wrong guess about the cause of a problem, the HUMMBER method slowly corrected itself as new evidence came in. It did not force a quick answer; instead, it updated its understanding as the story unfolded, eventually identifying the true cause in most cases.
Perhaps most importantly, the method learned when to stop and say "we don't know yet." In the simulation, the researchers tried to prove that a specific feature caused a change in behavior. When the data was shaky or the comparison was unfair, the method refused to make a claim. It labeled the result as "insufficient evidence" rather than forcing a conclusion. This is a deliberate design choice. The researchers built the system to be cautious. If the evidence was not strong enough, or if the data was missing in a way that could skew the results, the method flagged the uncertainty. It did not pretend to have an answer. This protects decision-makers from acting on false hopes. The system also checked for situations where the effect of a tool changed over time, ensuring that a conclusion drawn in week one was not blindly applied to week twenty.
The researchers were careful to state that this was a test of the method itself, not a test of a real health program. The people and events were entirely made up by a computer to see if the rules worked under controlled conditions. They did not test this on real patients or in a real hospital. The goal was to prove that the logic of the method holds up when the variables are known and the outcomes are pre-set. The simulation showed that the architecture could keep different types of evidence connected without confusing them. It could separate the story of what happened from the story of why it happened, and it could keep the burden on the user visible.
This work suggests a new way forward for the future of digital health. As artificial intelligence becomes more common in health apps, these tools will change and adapt constantly. The HUMMBER method offers a way to watch these changes closely, ensuring that we understand who benefits, who is left out, and what is actually driving the results. It places the responsibility back on human evaluators to interpret the evidence and make the final call. While the simulation proved the concept works in a controlled environment, the researchers note that the next step is to try it in the real world. They need to see if doctors, patients, and developers can use these rules together to make sense of complex, changing health tools. Until then, the method remains a promising blueprint for keeping human judgment at the center of an increasingly automated world.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.