Experimentally Validated Digital Twins: Evaluating Workforce Training Programs in the Age of Big Data
This paper demonstrates that machine learning models trained on large-scale administrative data can construct "experimentally validated digital twins" to accurately estimate workforce training program impacts, achieving error rates comparable to randomized controlled trials while revealing significant variations in program effectiveness across locations and time.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the author. For technical accuracy, refer to the original paper. Read full disclaimer
For decades, the gold standard for knowing if a program actually works has been the randomized controlled trial. In this approach, researchers take a group of people, flip a coin to decide who gets the new service and who does not, and then compare the results. Because the assignment is random, any difference in outcomes can be confidently blamed on the program itself, not on the fact that the people who signed up were more motivated or had better jobs to begin with. This method has become the bedrock of modern policy, guiding decisions on everything from education to job training. However, running these trials is expensive, slow, and often impossible to do for every single program that exists. As a result, many programs operate without rigorous proof of their value, leaving policymakers to guess whether their investments are helping or hurting.
The alternative has traditionally been to look at observational data, comparing people who chose to join a program with those who did not. But history has shown this to be a dangerous path. Because people who choose to join are often different from those who stay away in ways that researchers cannot see—such as their drive or family support—simple comparisons often produce misleading results. For years, the consensus was that without a randomized trial, you simply could not trust the numbers. This paper, however, revisits that conclusion in an era of massive data. The researchers asked a simple question: if we have access to the complete work and tax history of every American adult, combined with powerful computer models, can we build a virtual version of a person that is so accurate it acts just like a randomized control group?
The team, led by economists at Harvard and the Opportunity Insights organization, set out to test this idea using the specific case of workforce training programs. They gathered a massive dataset containing the tax records of 327 million Americans, tracking their earnings from 1999 through 2025. They linked these records to information on over 100,000 people who had participated in various job training programs. Instead of trying to find a perfect match for each participant in a small survey, as was done in the past, they used machine learning to build a sophisticated prediction model. This model learned from the entire population to predict what a participant's earnings would have been if they had never joined the program. The researchers called these predictions "digital twins"—statistical stand-ins for the participants that represent what would have happened to them had they not received the training.
To see if these digital twins were any good, the researchers needed a benchmark. They turned to 37 existing randomized trials of job training programs that had already been conducted between 2000 and 2019. These trials provided the true, experimentally verified answers for how much money the participants actually earned compared to those who were randomly assigned to not join. The researchers then applied their digital twin method to these same groups of people. They asked the computer model to predict the earnings of the participants based on their past work history, demographics, and where they lived, and then compared those predictions to the actual results from the randomized trials.
The results were striking. The predictions made by the digital twins matched the results from the randomized trials with remarkable precision. The difference between the computer's estimate and the real experimental result was so small that it was comparable to the natural statistical noise found in the trials themselves. In fact, the error in the digital twin estimates was slightly smaller than the error in the randomized trials, largely because the computer model had access to a much larger pool of data to learn from. The researchers found that the old way of doing things, which relied on smaller surveys and simpler math, produced errors that were more than twice as large. The key to the success was not just having more data, but using flexible computer models that could handle the complexity of real human lives, such as tracking a person's earnings over many years and across different industries.
With the method validated, the researchers used it to look at job training programs in places and times where no randomized trial had ever been done. They discovered that the success of these programs was not a fixed number; it varied wildly depending on where the program was located and the year it took place. For example, a specific program called Year Up United showed a wide range of results. In some cities and for some years, the program boosted participants' annual earnings by over $12,000, while in other locations or years, the boost was much smaller. These differences were too large to be random chance, suggesting that the effectiveness of a program depends heavily on local conditions and how the program is run at a specific time.
The study also uncovered what makes these programs work. The biggest gains in earnings came from programs that did more than just teach skills; they actively placed participants into internships and jobs in specific, high-wage industries. The researchers found that the more a program succeeded in placing people into these target-sector jobs, the higher the earnings gains. Furthermore, these gains were not temporary. Participants who saw large earnings increases two years after the program continued to earn significantly more than their digital twins even ten to fifteen years later. This persistence held true even for the 90 percent of participants who eventually left their initial jobs. The data suggests that these programs do not just get people a paycheck; they put them on a career ladder that pays off for the rest of their working lives, regardless of where they end up working.
The authors conclude that while randomized trials remain a vital tool for checking the accuracy of these methods, the era of big data has changed the landscape of program evaluation. It is now possible to build "digital twins" that provide reliable, near-real-time evidence on whether a program is working, even in the absence of a formal experiment. This approach allows policymakers to see how a program performs in a specific city or year, rather than relying on a single average from a trial conducted years ago in a different place. The findings suggest that by using rich data and flexible models, we can move beyond the old debate of whether observational data is trustworthy and instead use it to understand the specific details of what makes interventions succeed or fail.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.