📊 statistics

ICCDesign: An R Package for the Design and Analysis of ICC-Based Reliability Studies with Continuous Responses

This paper introduces the `ICCDesign` R package, which provides an integrated workflow for designing and analyzing ICC-based reliability studies with continuous responses by unifying sample size planning, point estimation, confidence interval calculation, and automated reliability evaluation within a single framework guided by the McGraw and Wong (1996) classification.

Original authors: Ziyu Liu, Ruilin Ma, Yundan Zhang, Chenge Gao

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ziyu Liu, Ruilin Ma, Yundan Zhang, Chenge Gao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge at a talent show, and you have a panel of five other judges. You want to know: How much do these judges agree with each other? Do they all give the same score to the same singer, or does one judge love rock while another hates it?

In the world of science and medicine, researchers face this exact problem every day. They need a tool called the Intraclass Correlation Coefficient (ICC) to measure how reliable their "judges" (or raters) are. But until now, using this tool has been like trying to assemble a piece of furniture with instructions written in three different languages, using screws from three different boxes.

Here is the story of a new tool called ICCDesign that fixes this mess.

The Problem: A Puzzle with Missing Pieces

For decades, scientists have had to jump between different computer programs to do one job:

One program tells you how many people you need to study (Sample Size).
Another program calculates the agreement score.
A third program checks if the score is "good" or "bad."

Worse, the "agreement" score isn't just one thing. Depending on how your study is set up (Are the judges the same for everyone? Are they a random group or a specific team?), there are 10 different ways to calculate the score. Picking the wrong one is like trying to measure a room with a ruler when you should have used a tape measure—the result looks like a number, but it's the wrong answer.

The Solution: ICCDesign

The authors of this paper, a team from East China Normal University, built a new R package called ICCDesign. Think of it as a "Swiss Army Knife" for reliability studies. Instead of juggling three different tools, you now have one all-in-one station.

Here is what makes it special, explained through simple analogies:

1. The "GPS" for Choosing the Right Formula

The biggest headache in this field is knowing which of the 10 formulas to use.

The Old Way: You had to read a thick textbook, figure out your study design, and hope you picked the right math.
The ICCDesign Way: It comes with a built-in four-step decision framework. It's like a GPS that asks you four simple questions:
1. Are the same judges rating everyone?
2. Are the judges a random sample or a fixed team?
3. Are we looking at one single score or an average?
4. Do we need them to agree on the exact number, or just on the pattern?
Based on your answers, the package automatically picks the correct formula. If you pick a tricky combination that requires a warning, the package gently taps you on the shoulder and says, "Hey, be careful with this one!"

2. The "One-Stop Shop" Workflow

Before, a researcher had to switch between different software windows, copying and pasting numbers, which is a recipe for typos and errors.

The ICCDesign Way: It handles the whole journey in one place. You input your data, it cleans it up (like a chef washing vegetables before chopping), calculates the score, checks the confidence intervals (the "margin of error"), and even writes a plain-English report for you.

3. The "Crystal Ball" for Planning

Before you start a study, you need to know how many people to recruit.

The Old Way: You might guess, or use a calculator that only works for one specific scenario.
The ICCDesign Way: It has a "Crystal Ball" feature. You can tell it, "I want to be 80% sure my results are accurate," or "I want the margin of error to be this small," and it tells you exactly how many people you need. It can also work backward: "I have 30 people; how accurate can my results be?"

4. The "Traffic Light" Report

Once the math is done, the package doesn't just give you a number like "0.72." It translates that number into a simple grade, based on expert guidelines:

< 0.50: Poor (Red Light)
0.50 – 0.75: Moderate (Yellow Light)
0.75 – 0.90: Good (Green Light)
> 0.90: Excellent (High Five)

If the result is right on the edge (like a score that could be "Good" or "Moderate"), the package adds a sticky note saying, "We aren't 100% sure yet; you might need more data."

5. The "No-Code" Dashboard

For people who are scared of computer code, the package includes a Shiny web app. This is a colorful, interactive website you can run on your computer. You can drag and drop your data, click buttons to make choices, and see the results instantly without writing a single line of code.

What It Does (and Doesn't Do)

The paper is very clear about the boundaries of this tool:

It does: Handle continuous numbers (like height, weight, or test scores), assume one rating per person, and follow the standard rules set by famous statisticians (McGraw, Wong, Shrout, Fleiss).
It doesn't: Handle missing data perfectly (it just removes the incomplete rows), work with "Yes/No" answers, or fix studies where the data is unbalanced (like having 5 judges for one person and only 2 for another).

The Bottom Line

ICCDesign is a new tool that takes the confusion out of measuring reliability. It turns a complex, fragmented process into a smooth, guided experience. It helps researchers stop worrying about which math to use and start focusing on what their data actually means.

The tool is free, open-source, and available for anyone to use right now.

Technical Summary of ICCDesign: An R Package for the Design and Analysis of ICC-Based Reliability Studies

Problem Statement

The intraclass correlation coefficient (ICC) is a cornerstone statistic in reliability research across medical, psychological, and behavioral sciences. However, its practical application faces two significant hurdles. First, the theoretical framework established by McGraw and Wong (1996) organizes ICC into ten distinct design combinations (six standard forms and four additional combinations) based on model assumptions (one-way vs. two-way, random vs. mixed effects) and design choices (single vs. average rating, absolute agreement vs. consistency). Selecting the correct form requires researchers to navigate complex design questions, yet existing guidelines are often not operationalized within software interfaces. Second, the current R ecosystem for ICC analysis is fragmented. Tools for point estimation (e.g., irr, psych), sample size calculation (e.g., ICC.Sample.Size), and reliability evaluation are distributed across separate packages. This fragmentation forces researchers to switch between tools, increasing the risk of analytical errors and inconsistent design specifications between the planning and analysis phases.

Methodology

The ICCDesign package (version 0.1.0) provides an integrated workflow for continuous response data, assuming one rating per subject–rater cell. The methodology is grounded in the Analysis of Variance (ANOVA) framework and adheres to the classification system of Shrout and Fleiss (1979) and McGraw and Wong (1996).

1. Statistical Models and Estimation:
The package implements three statistical models:

One-way random effects: For designs where raters are not a common set across subjects.
Two-way random effects: For designs where subjects are rated by the same set of raters sampled from a broader population.
Two-way mixed effects: For designs where raters constitute a specific fixed group.

Point estimates are calculated using ANOVA-based formulas derived from mean squares (MSR, MSC, MSE, MSW). The package supports all ten design combinations, including "use-with-caution" scenarios where computational routines are shared but design interpretations differ (e.g., consistency under two-way random vs. mixed models).

2. Inference and Hypothesis Testing:

Confidence Intervals (CIs): The package computes F-distribution-based CIs. For standard forms, it uses Type 1 intervals. For absolute agreement in two-way models, it employs a Satterthwaite-type approximation to correct degrees of freedom. All reported bounds are truncated to the interpretable range $[0, 1]$ .
Hypothesis Testing: It implements one-tailed F-tests for two scenarios: a default test against zero ( $H_0: ICC = 0$ ) and an optional threshold test against a user-specified minimum reliability ( $H_0: ICC = \rho_0$ ).

3. Sample Size Planning:
Based on Zou (2012), the package offers three planning modes:

Lower Bound Assurance: Determines the minimum sample size ( $n$ ) to ensure the lower bound of the CI exceeds a target threshold with a specified assurance probability.
Confidence Interval Half-Width: Calculates $n$ to achieve a desired CI precision (half-width) with a given assurance.
Inverse Assurance: Computes the assurance probability for a fixed sample size.

4. Reliability Evaluation:
Automated grading is performed using the criteria proposed by Koo and Li (2016), applied to the lower bound of the confidence interval rather than the point estimate. The system flags uncertainty if the CI spans the "good" reliability threshold (0.75).

Key Contributions

The ICCDesign package introduces a unified interface that consolidates the entire reliability study workflow. Its primary contributions include:

Comprehensive Coverage: It implements point estimation, ANOVA-based CIs, and hypothesis tests for all ten ICC design combinations under the McGraw and Wong (1996) framework, including four additional "use-with-caution" combinations.
Four-Step Decision Framework: A structured parameter system guides users from study design specifications (rater consistency, effect type, rating unit, agreement type) to the appropriate ICC form. The system provides explicit feedback (NOTES or WARNINGs) for special cases, such as automatic mappings between computational routines or restrictions on inference for fixed-rater absolute agreement.
Integrated Planning and Analysis: The package unifies sample size planning (via Zou's closed-form formulas) and data analysis, ensuring consistent design specifications throughout the research lifecycle.
Automated Reporting and Visualization: It generates standardized text-format analysis reports and includes an interactive Shiny web application that allows users to perform the full workflow without writing R code.
Data Preprocessing Utility: A dedicated function (icc_preprocess_data) validates data format, dimensions, and missing values before analysis, enhancing transparency and data quality control.

Results and Capabilities

The paper demonstrates the package's functionality through illustrative examples using a built-in simulated dataset. The examples cover:

Preprocessing wide-format data.
Computing various ICC forms (e.g., ICC(1,1), ICC(2,1), ICC(3,k)).
Conducting custom threshold hypothesis tests.
Handling "use-with-caution" combinations with appropriate user notifications.
Performing sample size calculations for lower-bound assurance, CI width, and inverse assurance.
Launching the Shiny application for a code-free workflow.

Comparisons with existing packages (irr, psych, ICC.Sample.Size, irrNA) highlight that ICCDesign is the only tool offering full coverage of the ten design combinations, automated decision guidance, dedicated preprocessing, and a unified interface for both planning and analysis.

Significance and Claims

The authors position ICCDesign not as a proposal for new statistical estimators, but as a consolidation of established procedures into a cohesive, user-friendly interface. The significance of the work lies in addressing the fragmentation of current tools and the lack of operational guidance for selecting the correct ICC form. By integrating design specification, sample size planning, estimation, inference, and reporting into a single package, ICCDesign aims to reduce operational errors and improve the rigor of reliability studies. The package is available on GitHub under the GPL (>= 3) license, with future plans to submit to CRAN and expand support for missing data and bootstrap intervals.