Adoption Telemetry: Measuring Enterprise AI Adoption from Production Signals
This paper introduces "adoption telemetry," a framework and open-source implementation (NANTE) that measures enterprise AI adoption by mapping production usage signals to a five-stage change-management progression, while explicitly framing its thresholds as testable constructs for future empirical validation.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern workplace, a quiet paradox has taken hold. Companies have rushed to install artificial intelligence tools, hoping to automate tasks and reshape how work gets done. Yet, despite nearly universal deployment, the results have been underwhelming. Studies show that the vast majority of these projects fail to deliver measurable financial value, and many are abandoned entirely. The consensus among experts is that the technology itself is not the problem; the machines work as designed. The failure lies in the human element: organizations struggle to change the way their employees actually work. They cannot tell the difference between a team that has merely tried a new tool and one that has fundamentally reorganized its daily routine around it. Current measurement tools are blind to this distinction. They can count how many people clicked a button or how often a system was opened, but they cannot see whether the work itself has changed. This gap leaves leaders guessing, unable to diagnose why adoption stalls or how to fix it.
A new approach called "adoption telemetry" seeks to solve this by turning the invisible process of behavioral change into something visible and measurable. Instead of relying on surveys or simple activity counts, this method treats adoption as an engineering problem. It proposes that organizations can understand how their people are adapting to AI by analyzing the digital footprints left behind in the systems they use every day. The researchers behind this work, led by Damon A. Young, have built a framework that reads these production signals—such as how many times a tool is used, how long a session lasts, and whether the tasks completed successfully—and maps them to specific stages of change. The goal is not just to count usage, but to pinpoint exactly where a group of employees gets stuck in the journey from knowing a tool exists to weaving it into the fabric of their daily work.
The core of this new framework is a five-stage model called NANTE, which stands for Notice, Attempt, Navigate, Transform, and Embed. Each stage represents a distinct milestone in how a person relates to a new capability. The first three stages—Notice, Attempt, and Navigate—measure breadth. They track whether people know the tool exists, whether they have tried it at least once, and whether they have returned to use it repeatedly. The final two stages—Transform and Embed—measure depth. This is where the real change happens. "Transform" means the tool has become part of complex, multi-step workflows and is being used successfully to get real work done. "Embed" means the integration is so deep that removing the tool would disrupt the team's output. The researchers found that most organizations get stuck right at the boundary between the third and fourth stages. People are using the tools, but they are not changing their work habits. The new system is designed to make this specific cliff visible, distinguishing between a team that is simply exploring and one that has truly integrated the technology.
To test whether this idea works, the researchers built an open-source toolkit and created a series of simulated populations to see if the system could correctly identify different types of adoption failures. They designed six distinct scenarios: one healthy group that successfully adopted the tool, and five groups representing common ways adoption goes wrong. These failures included teams that tried the tool but never went deep, teams that knew about the tool but never started, teams that used it frequently but failed at the tasks, and teams where only a few "champions" used the tool while the rest did not. The simulation results showed that the system could mechanically distinguish between these groups. For instance, it could tell the difference between a healthy team and a "shallow plateau" team. On a standard usage dashboard, both groups might look identical because they both have high numbers of active users. However, the new system saw that the healthy team had moved into the deep stages of workflow integration, while the other team had stalled at the surface level. It also correctly identified when a team was failing because they lacked skills versus when they were failing because they lacked motivation.
The researchers are careful to state that while the system works in these simulations, the specific numbers used to define each stage are not yet proven facts. The thresholds—such as how many weeks of use are needed to count as "recurring"—are proposed starting points that can be tested and adjusted. The current evidence comes from synthetic data, not from real-world companies yet. The author explicitly notes that they have not validated these rules against actual business outcomes. They are inviting organizations to partner with them to test the system on real data, which would allow them to calibrate the thresholds and confirm that the diagnoses match reality. Until then, the system should be viewed as a structured hypothesis rather than a final verdict. The researchers also acknowledge that the system has limits; it cannot measure what people feel or think, only what they do. It cannot yet detect if a team is pretending to use the tool to satisfy a manager, nor can it fully measure the success of autonomous agents that work without human initiation.
What makes this work significant is not just the tool itself, but the argument that such a tool was missing. The researchers explain that the failure to measure adoption properly is not an accident but a structural problem. The different groups that usually handle these measurements—those who build the AI, those who sell the software, and those who manage organizational change—operate in separate silos. The AI builders focus on whether the code works. The software sellers focus on whether people are logging in. The change managers focus on what people say in surveys. None of these groups has built a system that combines the technical data with a model of human behavior change. This new framework attempts to bridge that gap by unifying these perspectives into a single instrument. It suggests that the next step in enterprise AI is not better algorithms, but better ways to see how humans are actually changing their work. By making the invisible visible, organizations may finally be able to move past the initial excitement of deployment and address the real challenge of lasting adoption.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.