📊 statistics

Better Assumptions, Stronger Conclusions: The Case for Ordinal Regression in HCI

This paper critiques the inconsistent and often inappropriate statistical methods currently used to analyze ordinal data in HCI research and advocates for the adoption of cumulative link (mixed) models as a more robust alternative, supported by practical R-based examples.

Original authors: Brandon Victor Syiem, Eduardo Velloso

Published 2026-02-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Brandon Victor Syiem, Eduardo Velloso

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Ruler" Mistake

Imagine you are asking people to rate how much they like a new video game. You give them a survey with five options:

Hated it
Disliked it
It was okay
Liked it
Loved it

This is ordinal data. We know the order (5 is better than 1), but we don't know the distance between them. Is the jump from "Hated it" to "Disliked it" the same size as the jump from "Liked it" to "Loved it"? Probably not. One person might hate a game slightly, while another might be absolutely furious. The "gap" in feelings isn't a straight line.

The Mistake:
For years, many researchers in Human-Computer Interaction (HCI) have treated these survey answers like a ruler. They've added up the numbers (1+2+3+4+5), calculated the average, and used math tools designed for continuous measurements (like temperature or height).

The Analogy:
Imagine you are measuring the height of a group of people, but instead of a tape measure, you only have a ladder with rungs labeled "Short," "Medium," and "Tall."

The Ruler Approach (Parametric): You assume the distance between "Short" and "Medium" is exactly the same as "Medium" and "Tall." You calculate an "average height" of 1.5 rungs. This is mathematically convenient, but it's fake. You can't be 1.5 rungs tall on a ladder.
The Reality: The gap between "Short" and "Medium" might be 2 inches, but the gap between "Medium" and "Tall" might be 10 inches. Treating them as equal distorts the truth.

The Current Chaos: A Kitchen Without a Recipe

The authors looked at 94 recent papers from the top HCI conference (CHI). They found a massive mess:

Some researchers used "Ruler Math" (Parametric tests like ANOVA) on the ladder data.
Some used "Rank Math" (Non-parametric tests like Friedman) which ignores the gaps entirely but is less sensitive.
Some used a mix of both in the same study, creating a "Frankenstein" analysis where the rules changed halfway through.

The Consequence:
It's like two chefs trying to compare their recipes. Chef A measures ingredients in cups, Chef B measures in grams, and Chef C just guesses. They can't compare their cakes to see which is actually better. This makes it hard for science to build on previous work because the results aren't consistent.

The Solution: The "Hidden Dial" (Ordinal Regression)

The authors propose a new way to look at the data called Cumulative Link Models (CLMs).

The Analogy: The Hidden Dial
Imagine that inside every person's brain, there is a hidden, continuous dial that measures their true feeling. This dial is smooth and continuous (like a volume knob).

When the dial is at the very bottom, they click "1" (Hated it).
As they turn the dial up, they stay at "1" until they hit a specific threshold (a cut-off point).
Once they cross that threshold, they click "2" (Disliked it).
There is another threshold for "3," and so on.

How CLMs Work:
Instead of treating the survey numbers (1, 2, 3) as rigid steps on a ladder, CLMs acknowledge that there is a smooth, invisible dial underneath.

The Model: It guesses where those "thresholds" are located on the hidden dial.
The Math: It calculates how much an experiment (like a new game feature) shifts the entire dial up or down.
The Result: It tells you, "This new feature shifted the dial significantly higher," without assuming that the distance between "1" and "2" is the same as "4" and "5."

Why This Matters: Finding the Signal in the Noise

The authors ran the old data through this new "Hidden Dial" model and found something surprising.

The Example:
In one study, researchers used the old "Ruler Math" and concluded: "There is no difference between the learning methods." They saw the averages and they looked the same.

But when the authors used the new "Hidden Dial" model, they found: "Actually, there IS a difference!"

Why? The old method was looking at the average. The new method looked at the shape of the answers.
The Metaphor: Imagine two groups of people.
- Group A mostly says "It was okay" (3), but a few say "Loved it" (5).
- Group B mostly says "Disliked it" (2), but a few say "Hated it" (1).
- The average might look similar. But the "Hidden Dial" sees that Group A is clustered near the top of the scale, while Group B is clustered near the bottom. The new model catches this nuance; the old model misses it.

The Takeaway for Everyone

Stop forcing square pegs into round holes: Don't treat survey rankings (1-5) like precise measurements (like weight or speed).
Use the right tool: The authors provide a "recipe" (using R software) to use the Cumulative Link Model. It's like upgrading from a basic calculator to a smart calculator that understands the difference between a ladder and a ruler.
Better Science: By using methods that respect the nature of the data, researchers can find real differences that were previously hidden, avoid false alarms, and make sure their conclusions are actually true.

In short: The paper argues that we need to stop pretending our survey data is a straight line. It's a staircase with uneven steps. If we use a model that understands the stairs, we can finally see the view from the top clearly.

1. Problem Statement

Human-Computer Interaction (HCI) research heavily relies on ordinal measures (e.g., Likert scales, preference rankings, satisfaction scores) to quantify subjective user experiences. However, there is a significant lack of consensus regarding the statistical methods used to analyze this data.

The Conflict: Researchers frequently oscillate between parametric methods (e.g., ANOVA, t-tests) and non-parametric methods (e.g., Friedman, Wilcoxon tests).
- Parametric methods assume data is metric (interval/ratio), normally distributed, and has equal variance. Applying these to ordinal data (which lacks equal intervals) can inflate Type I and Type II errors and produce misleading mean estimates.
- Non-parametric methods avoid metric assumptions but often discard the inherent ordering of data or lack sensitivity (statistical power). Furthermore, some common non-parametric tests (like the Wilcoxon Signed Rank test) implicitly assume metric data by calculating differences between ranks, violating the nature of ordinal data.
The Consequence: This methodological inconsistency leads to erroneous insights, hinders the replication of findings, and prevents the aggregation of results across studies. The field suffers from "inconsistent successive assumptions," where a study might use a non-parametric omnibus test followed by a parametric post-hoc test, creating logical contradictions.

2. Methodology

The authors employed a mixed-method approach combining a systematic literature review with practical re-analysis using advanced statistical modeling.

A. Literature Review

Scope: Analyzed 94 full-text papers from the 2024 ACM CHI Conference proceedings (sampled from 558 papers).
Focus: Identified statistical methods used for ordinal data analysis, categorized into:
1. Predictive Modeling: (e.g., Regression, GLMs).
2. Omnibus Tests: (e.g., ANOVA, Kruskal-Wallis, Friedman).
3. Pairwise Comparisons: (e.g., t-tests, Wilcoxon, Dunn).
Analysis: Categorized methods based on their assumptions regarding metric data, normality, and variance.

B. Proposed Solution: Cumulative Link (Mixed) Models (CL(M)Ms)

The authors advocate for Ordinal Regression using Cumulative Link Models (CLMs) and Cumulative Link Mixed Models (CLMMs).

Theoretical Basis: CL(M)Ms assume an underlying continuous latent variable (unobservable psychological state) that follows a specific distribution (e.g., standard normal).
Mechanism: The observed ordinal response is generated by mapping this latent variable through a series of thresholds (cutpoints).
- $Y = k$ if $\tau_{k-1} < \eta \leq \tau_k$ , where $\eta$ is the latent variable and $\tau$ are thresholds.
Advantages:
- Treats data as categorical with inherent order (avoiding metric assumptions).
- Leverages the ordering information (unlike simple rank tests), increasing statistical power.
- Handles complex designs (fixed and random effects) via Mixed Models (CLMMs).

C. Practical Demonstration

The authors re-analyzed two open-sourced datasets from previous CHI papers using R (specifically the ordinal package):

Fitton et al. [26]: A between-subjects study on VR learning. Re-analyzed "Perceived Usefulness" and "SIM-TLX" sub-scales using CLMs.
Chen et al. [16]: A within-subjects study on display height adjustment. Re-analyzed raw NASA-TLX scores using CLMMs (to account for participant random effects).

3. Key Contributions

Empirical Evidence of Inconsistency: The review revealed that 319 out of 527 statistical tests in the sampled papers imposed metric assumptions on ordinal data, even when using "non-parametric" methods. There is no standard sequence of analysis (e.g., Omnibus $\to$ Post-hoc) that is consistently applied or theoretically sound.
Theoretical Clarification: The paper provides a clear mathematical and intuitive explanation of CL(M)Ms, demystifying the "latent variable" concept and the "proportional odds" assumption for the HCI community.
Demonstration of Superiority:
- In the Fitton et al. re-analysis, the original ANOVA found no significant differences in "Physical Demand" or "Stress" across conditions. The CLM analysis, however, detected significant differences between conditions (Self vs. Active), revealing insights that the metric approach missed due to the skewness of the ordinal data.
- In the Chen et al. re-analysis, the authors demonstrated how CLMMs correctly handle within-subject designs and random participant effects, which standard non-parametric tests (like Friedman) struggle to model flexibly.
Practical Toolkit: The authors provide R code, scripts, and interactive tools to lower the barrier to entry for implementing CL(M)Ms, addressing the historical perception that these methods are too complex for HCI researchers.

4. Results

Literature Findings:
- Predictive Modeling: Rarely used for ordinal data (only 28 instances in the sample).
- Omnibus Tests: Dominated by ANOVA (71 uses) and Kruskal-Wallis (70 uses).
- Pairwise Tests: Dominated by Wilcoxon Signed Rank (61 uses) and Paired t-tests (54 uses).
- Critical Flaw: Many "non-parametric" tests used (e.g., Wilcoxon Signed Rank, ART-ANOVA) actually calculate differences or residuals, implicitly assuming metric data, thus violating the ordinal nature of the data.
Re-analysis Findings:
- CLMs provided more sensitive detection of effects compared to traditional ANOVA, particularly when data distributions were skewed or non-normal.
- CLMs allowed for the estimation of effects on a latent scale (standard normal), offering a more theoretically grounded interpretation of "how much" a condition changed a user's internal state, rather than just comparing means of arbitrary numbers.

5. Significance

This paper serves as a critical intervention in HCI methodology:

Paradigm Shift: It moves the field away from the binary "Parametric vs. Non-parametric" debate toward a more nuanced "Ordinal Regression" approach that respects the data's nature.
Reproducibility & Consistency: By standardizing on CL(M)Ms, the field can reduce the "noise" caused by incompatible statistical assumptions, allowing for better meta-analysis and theory building.
Accessibility: By providing concrete R examples and explaining the assumptions (Proportional Odds, Latent Variance), the authors make a sophisticated statistical tool accessible to HCI practitioners.
Future-Proofing: As HCI research increasingly deals with complex, multi-factorial designs and subjective metrics, CL(M)Ms offer the necessary flexibility (handling random effects, multiple predictors) that traditional tests lack.

In conclusion, the authors argue that adopting Cumulative Link (Mixed) Models will lead to "Better Assumptions" (respecting the ordinal nature of data) and consequently "Stronger Conclusions" (higher statistical power and more accurate insights) for the HCI community.