Evaluating national machine-learning predictions of household biogas use across time and local contexts in Nepal

In the hills and valleys of Nepal, millions of households have spent decades trying to move away from cooking with smoke and soot. For generations, families relied on burning wood, crop residue, or dung, a practice that harms health and depletes forests. To help, the government and international groups have spent years installing biogas plants. These systems take animal waste and turn it into a clean gas for cooking, offering a promise of a cleaner, healthier life. But a clean stove in the yard does not guarantee a clean fire in the kitchen. Families might install the technology but stop using it if they lack water, if the animals run out of manure, or if the machine breaks and no one can fix it. This gap between having the technology and actually using it is the central puzzle for anyone trying to understand how energy transitions really work.

To track this progress, researchers often rely on large national surveys that ask thousands of families about their fuel choices. In recent years, scientists have started using powerful computer programs, known as machine learning, to analyze these surveys. These programs look for patterns in the data to predict which households are likely to use biogas. The hope is that these predictions can act as a map, showing officials where the energy transition is succeeding and where it is stalling. However, a map drawn for a whole country might not fit a specific village, and a map drawn for one year might not fit the next. The question is whether these national computer models remain useful when applied to different places or different times, or if they simply become outdated guesses.

A team of researchers set out to test exactly this. They took machine-learning models built from national data in Nepal and tried to use them to predict biogas use in a specific local area, the Chitwan Valley, across several different years. They wanted to see if the models could still tell the difference between households that would use biogas and those that would not, even as time passed and the local situation changed. They also wanted to see if the models treated different social groups fairly, or if they consistently missed the mark for certain communities.

The researchers gathered data from three national surveys conducted in 2011, 2016, and 2022, which covered the entire country. They then compared these national predictions against independent surveys they had conducted themselves in the Chitwan Valley in 2014, 2017, and 2023. To make the test fair, they only used information that was available before each local survey took place. For instance, when predicting for the 2014 local survey, they used only the national data from 2011. This ensured they were testing how well the models held up over time, rather than just repeating what they already knew.

The results revealed a fascinating split in how these models perform. When the researchers looked at whether the models could rank households correctly—identifying which families were more likely to use biogas than others—the models remained surprisingly strong. Even years later, the computer programs were still good at spotting the households with a higher chance of using the technology. However, when the researchers looked at the actual numbers, the models became less accurate. The models kept predicting that more families would use biogas than actually did. Over time, the gap between what the models predicted and what was observed in real life grew wider. In the most recent comparison, the models predicted that nearly three percent of households would use biogas as their main fuel, but the actual number was only about one percent.

This discrepancy became even clearer when the researchers looked at the different stages of using biogas. Having a plant installed is just the first step. A plant might be active for a year, then stop working. A family might use the gas occasionally, or they might use it as their main fuel. The national models were designed to predict the most advanced stage: using biogas as the primary cooking fuel. When the researchers compared these predictions to local data, they found that the models worked best when looking at that specific, advanced stage. But when they looked at simpler stages, like just having a plant installed, the models vastly overestimated how many families were actually using the technology. In 2023, for example, while thirteen percent of households had a plant, only six percent reported it was active, and fewer than three percent used it as their main fuel. The models, trained on the national goal of primary use, could not easily translate that success down to the messy reality of partial use or broken equipment.

The study also uncovered important differences based on social identity. In Nepal, society is structured by caste, a system that has historically determined access to resources and opportunities. When the researchers compared the model's predictions to the actual outcomes for different caste groups, they found a consistent pattern of under-prediction for marginalized communities. Specifically, households from the Dalit caste, who have faced long-standing discrimination, were significantly less likely to use biogas than the national models predicted. The models, built on broad national averages, failed to account for the specific barriers these families face. In contrast, the models did not show the same kind of gap for households involved in community forest groups, suggesting that while social status was a major factor in the prediction error, local institutional membership was less of a differentiator in this specific context.

The researchers concluded that these national models should not be treated as fixed forecasts that tell us exactly how many people will use biogas in a given year. Instead, they should be viewed as adaptive benchmarks. They are useful tools for spotting relative differences—telling us which households are more likely to adopt the technology than others—but they drift when it comes to predicting exact numbers. The study suggests that to truly understand the energy transition, officials need to repeatedly check these national predictions against local realities. By doing so, they can identify where the models are failing, such as in specific social groups or at specific stages of technology use, and target their support where it is needed most.

Ultimately, the work shows that while technology can be delivered to a household, keeping it running requires a much deeper understanding of local life. A computer program can tell you who is likely to use a biogas plant, but it cannot easily see the broken pump, the lack of water, or the social barriers that stop a family from using it. The path to a clean energy future in Nepal, and perhaps elsewhere, lies not just in installing more stoves, but in constantly re-evaluating our predictions to ensure they reflect the complex, changing lives of the people they are meant to serve.