UX in the Age of AI: Rethinking Evaluation Metrics Through a Statistical Lens

This paper proposes the Adaptive Dynamic UX Statistical Framework (ADUX-Stat), a novel evaluation model that replaces static usability metrics with probabilistic constructs—specifically the Interaction Entropy Index, Temporal Drift Coefficient, and Bayesian Usability Confidence Score—to effectively assess the stochastic and context-sensitive nature of AI-mediated systems.

The Gist

Imagine you are trying to grade a student's performance. In the old days, if you asked a student to solve a math problem, they would always give you the exact same answer. You could give them a simple score: "10 out of 10." This is how we used to test computer software. We asked users to click a button, and if it worked, they got a point. If it didn't, they didn't. The system was predictable, like a vending machine that always gives you a soda when you press "A1."

But today, computers are different. They use Artificial Intelligence (AI). An AI isn't a vending machine; it's more like a chatty, creative friend. If you ask your friend the same question twice, they might give you two slightly different answers depending on their mood, the time of day, or what they were just talking about.

The problem, according to this paper, is that we are still trying to grade this "chatty friend" with the old "vending machine" tests. It doesn't work. The old tests assume the computer will always do the same thing, but AI is messy, unpredictable, and changes over time.

To fix this, the author, Harish Vijayakumar, proposes a new way to measure how good an AI feels to use. He calls it ADUX-Stat. Instead of giving a single number, this new system uses three "tools" to understand the AI's personality.

Here is how the three tools work, using simple analogies:

1. The "Surprise Meter" (Interaction Entropy Index)

The Problem: Sometimes an AI is helpful and consistent. Other times, it's wild and unpredictable. If you ask a voice assistant for the weather, and it gives you a different answer every time, you get frustrated.
The Solution: This tool measures how much the AI "surprises" you.

• Low Surprise (Good): The AI acts like a reliable librarian. You ask for a book, and it always hands you the right one.
• High Surprise (Bad or Chaotic): The AI acts like a magician pulling random rabbits out of a hat. Sometimes it's great, sometimes it's nonsense.
  This tool doesn't just say "it worked"; it measures how much the AI's behavior varies from your perspective.

2. The "Time-Travel Compass" (Temporal Drift Coefficient)

The Problem: AI isn't static. It learns. An AI might be terrible when you first meet it, but get smarter the more you talk to it. Or, it might start out great and slowly get worse as it gets confused.
The Solution: This tool looks at the AI's performance over time, like watching a movie instead of a single photo.

• Positive Drift: The AI is getting better, like a student who studies hard and improves their grades week by week.
• Negative Drift: The AI is getting worse, like a car engine that starts making weird noises after a few months.
  This helps us see if the AI is a "slow learner" or a "slow decliner," which a single test can never tell you.

3. The "Honesty Bubble" (Bayesian Usability Confidence Score)

The Problem: Old tests give you a single number, like "85% satisfaction." But that number feels too precise. It's like saying, "I am exactly 5 feet 10.00 inches tall." In reality, measurements have errors, and with AI, there is a lot of uncertainty.
The Solution: This tool gives you a range instead of a single number. It's like saying, "I am probably between 5 feet 9 inches and 5 feet 11 inches."

• It uses a special math method (Bayesian statistics) to admit, "We aren't 100% sure, but here is the most likely range."
• If you don't have much data, the range is wide (honest about not knowing). If you have lots of data, the range gets narrow (more confident).
  This stops us from pretending we know more than we actually do.

How They Tested It

The author didn't test this on real people yet. Instead, he did a "thought experiment." He imagined how these three tools would work on five different types of AI products:

1. Chatbots: He predicted they would have high "Surprise" because they can say many different things.
2. Recommendation Engines (like Netflix): He predicted they would get better over time ("Positive Drift") as they learn your taste.
3. Form Fillers: He predicted they would have low "Surprise" because they just fill in known data fields.

The Bottom Line

The paper argues that we need to stop treating AI like a simple machine. We need new tools that understand that AI is unpredictable, changes over time, and uncertain.

The author admits this is just a new map; he hasn't gone on the journey with real travelers yet. He hopes that in the future, researchers will use these three tools to actually test AI products with real people, so we can finally measure the experience of talking to a machine the way it really is: a dynamic, evolving conversation, not a fixed button press.

Technical Summary

Technical Summary: UX in the Age of AI: Rethinking Evaluation Metrics Through a Statistical Lens

Problem Statement

The rapid integration of artificial intelligence (AI) into consumer-facing digital products has rendered classical User Experience (UX) evaluation frameworks structurally insufficient. Legacy metrics such as the System Usability Scale (SUS), Net Promoter Score (NPS), and task completion rates were engineered for deterministic, rule-based interfaces where identical inputs yield identical outputs. In contrast, AI-mediated systems—including conversational agents, generative interfaces, and recommendation engines—operate as stochastic, context-sensitive, and temporally variable systems. In these environments, a single query may produce multiple distinct responses, and user satisfaction is a probabilistic phenomenon rather than a fixed state. Consequently, existing instruments, which rely on assumptions of test-retest reliability and interface stability, fail to capture the inherent unpredictability and longitudinal evolution of AI-driven user experiences.

Methodology: The ADUX-Stat Framework

To address this epistemic gap, the paper proposes the Adaptive Dynamic UX Statistical Framework (ADUX-Stat). This model reconceptualizes usability not as a static scalar score, but as a probabilistic signal distribution. The framework integrates three original statistical constructs designed to measure distinct dimensions of AI interface behavior:

1. Interaction Entropy Index (IEI):

   • Purpose: Quantifies the degree of perceived output variability from the user's standpoint.
   • Mechanism: Drawing on Shannon's information entropy theory, IEI treats user satisfaction responses as a probability distribution over a discrete response space.
   • Formula: $IEI = -\sum p(r) \log_2 p(r)$, where $p(r)$ is the probability of a specific satisfaction rating $r$.
   • Interpretation: A high IEI indicates broad distribution of user responses (high unpredictability), while a low IEI indicates convergent responses (predictability).

2. Temporal Drift Coefficient (TDC):

   • Purpose: Measures the rate and direction of change in perceived usability across longitudinal interaction sessions.
   • Mechanism: Operationalizes usability as a time-series variable using linear regression to detect systematic improvement or degradation as the AI system evolves.
   • Formula: $TDC = \beta_1$ in the equation $U(t) = \beta_0 + \beta_1t + \epsilon(t)$, where $U(t)$ is the mean usability score at time $t$.
   • Interpretation: A positive $\beta_1$ signals improving UX over time; a negative $\beta_1$ signals deterioration. Stable estimation requires a minimum of five longitudinal measurement points.

3. Bayesian Usability Confidence Score (BUCS):

   • Purpose: Replaces point-estimate paradigms with probabilistic ranges to acknowledge measurement uncertainty.
   • Mechanism: Employs a Beta-Binomial model for task completion assessments. It updates a prior distribution (e.g., non-informative Beta(1,1)) with observed data to generate a posterior distribution.
   • Output: Reports the 95% Highest Density Interval (HDI) of the posterior distribution, providing a credible interval of plausible usability values rather than a single point estimate.

Key Results (Conceptual Validation)

The paper validates ADUX-Stat through a conceptual application across five AI product categories: (1) LLM-based conversational assistants, (2) AI-powered content recommendation engines, (3) generative image interfaces, (4) voice assistants, and (5) intelligent form auto-completion systems.

• IEI Discriminant Validity: The framework successfully differentiated between product types. Conversational assistants and generative image interfaces exhibited high IEI values (high unpredictability), recommendation engines showed moderate IEI, and structured form auto-completion systems demonstrated low IEI.
• TDC Sensitivity: The model aligned with literature suggesting conversational AI often exhibits negative drift in early deployment (due to learning curves) followed by positive drift as personalization improves. Recommendation engines showed consistent positive drift, while voice assistants demonstrated high sensitivity to environmental variables.
• BUCS Uncertainty Propagation: When applied to task completion data, BUCS produced 95% HDIs substantially wider than frequentist confidence intervals on the same data (using non-informative priors). This reflects "honest" uncertainty propagation, with intervals narrowing predictably as simulated sample sizes increased.

Significance and Claims

The paper claims ADUX-Stat offers a necessary statistical reorientation for the field of UX research, addressing a critical gap at the intersection of HCI, statistical modeling, and AI product evaluation. Its significance is defined by three core properties:

1. Epistemic Honesty: Unlike classical metrics that imply false precision through scalar point estimates, ADUX-Stat utilizes credible intervals and entropy distributions to acknowledge the inherent uncertainty of AI evaluation.
2. Temporal Sensitivity: The framework treats UX quality in AI systems as a trajectory rather than a static state, asserting that longitudinal measurement is epistemologically necessary for valid evaluation.
3. User-Perception Centricity: The IEI measures entropy as experienced by users rather than as computed from system logs, preserving the phenomenological orientation of UX research while incorporating statistical rigor.

The authors position ADUX-Stat as a reproducible, field-deployable methodology that can be integrated into existing workflows using standard statistical software, serving as a supplement to established instruments like SUS.

Limitations and Future Directions

The paper maintains a modest stance regarding its current scope. It explicitly acknowledges that the validation presented is conceptual and does not substitute for controlled experimental studies with real user populations. The authors state that future work must:

• Establish normative ranges for IEI, TDC, and BUCS across product categories.
• Develop standardized elicitation procedures.
• Assess inter-rater reliability across evaluator cohorts.
• Conduct empirical validation to confirm the framework's efficacy in real-world settings.