📊 statistics

Discovering the critical number of respondents to validate an item in a questionnaire: The Binomial Cut-level Content Validity proposal

This paper proposes a refined Binomial Cut-level Content Validity method to determine the critical number of respondents needed for questionnaire item validation, addressing paradoxes and probability inaccuracies inherent in traditional Content Validation Ratio (CVR) approaches.

Original authors: Helder Gomes Costa, Eduardo Shimoda, José Fabiano da Serra Costa, Aldo Shimoya, Edilvando Pereira Eufrazio

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Helder Gomes Costa, Eduardo Shimoda, José Fabiano da Serra Costa, Aldo Shimoya, Edilvando Pereira Eufrazio

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to create the perfect new recipe for a soup. You have a list of 20 potential ingredients (like carrots, spices, and mystery powders). Before you serve this soup to a whole city, you need to ask a group of expert food critics: "Is this ingredient essential for the soup, or is it just junk?"

This is exactly what researchers do when they design a questionnaire. They have a list of questions (ingredients) and need to know which ones are "essential" to ask and which ones should be thrown out.

For decades, the standard way to do this has been a method called CVR (Content Validity Ratio). Think of CVR as an old, slightly broken scale that the critics use to weigh the ingredients. It has been used for 50 years, but the authors of this paper argue that the scale is a bit wobbly and sometimes gives confusing results.

Here is a simple breakdown of their new idea, which they call BCV (Binomial Cut-level Validity).

The Problem with the Old Scale (CVR)

The old method (CVR) has three main flaws, which the authors compare to these scenarios:

The "Coin Flip" Mistake:
Imagine the critics have three choices for every ingredient:
- A) Keep it (Essential)
- B) It's okay, but not needed (Important but not essential)
- C) Throw it away (Unnecessary)
The old method assumes that if a critic picks an answer randomly, they have a 50/50 chance of picking "Keep it" or "Throw it away." But that's like flipping a coin! In reality, there are three options. If you pick randomly, you only have a 1-in-3 chance of picking "Keep it." The old math was using the wrong odds, making the results slightly inaccurate.
The "One-Sided" Blind Spot:
The old method only checks if enough people said "Keep it." It never checks if enough people said "Throw it away."
- The Analogy: Imagine a jury that only asks, "Is the defendant guilty?" They never ask, "Is the defendant innocent?"
- The Risk: You might end up keeping an ingredient because 11 people said "Keep it," but 12 people actually said "Throw it away." The old method misses this conflict.
The "Paradox" Trap:
Sometimes, a group of critics is so confused that they say an ingredient is both "Essential" AND "Unnecessary" at the same time.
- The Analogy: It's like a group of people voting on whether a bridge is safe. 60% say "It's safe!" and 60% say "It's unsafe!" (because some people voted twice or the group is split down the middle).
- The old method doesn't have a way to catch this "paradox." It just gives a result, even if the data is contradictory.

The New Solution: The "Double-Check" Scale (BCV)

The authors propose a new, smarter scale called BCV. Here is how it works in plain English:

1. Fixing the Odds (The Dice Roll)
Instead of assuming a coin flip (50/50), the new method uses a three-sided die (or a four-sided die if you add an "I don't know" option). It calculates the math based on the actual number of choices the critics have. This makes the "Essential" threshold much more precise.

2. The "Double-Check" Validation
The new method asks two questions for every ingredient:

Question A: Did enough people say "Keep it" to prove it's essential?
Question B: Did enough people say "Throw it away" to prove it's junk?

3. Catching the Paradoxes
By answering both questions, the new method creates four possible outcomes, like a traffic light system:

🟢 Green Light (Keep It): Enough people said "Essential," and not enough said "Unnecessary." -> Keep the question.
🔴 Red Light (Toss It): Enough people said "Unnecessary," and not enough said "Essential." -> Delete the question.
⚠️ Yellow Light (The "Paradox" - Type A): Too many people said "Essential," AND too many people said "Unnecessary."
- What this means: The group is confused or divided. Maybe the question is worded poorly, or the experts disagree wildly. The system flags this as a "Strong Paradox" so you can investigate before making a decision.
⚪ White Light (The "Paradox" - Type B): Not enough people said "Essential," AND not enough people said "Unnecessary."
- What this means: Everyone is on the fence. The question is too vague or boring. The system flags this as a "Weak Paradox" (or "No Consensus") so you can rewrite it.

Why Does This Matter?

Think of the old method as a crude filter that lets some bad ingredients through and misses some good ones.

The new BCV method is like a smart, high-tech scanner. It doesn't just count votes; it checks for contradictions. It tells you:

"This is definitely good."
"This is definitely bad."
"Wait, the data is fighting itself! We need to look closer."

The Bottom Line

The authors aren't just tweaking a formula; they are upgrading the entire process of deciding what questions to ask. By using better math (Binomial distribution instead of Normal approximation) and checking for "paradoxes," they ensure that the final questionnaire is not just a list of questions, but a reliable, clear, and logical tool that actually measures what it's supposed to measure.

It's the difference between guessing if a soup tastes good and actually tasting every single ingredient to make sure the recipe is perfect.

1. Problem Statement

The research addresses limitations in the existing methods for calculating the Content Validity Ratio (CVR), a metric used to determine if questionnaire items are "essential" based on expert consensus. The authors identify four critical flaws in the current state-of-the-art (Lawshe [1975], Wilson et al. [2012], and Ayre & Scally [2014]):

Incorrect Probability Assumption: Previous methods assume the probability ( $p$ ) of an expert randomly selecting "Essential" is $0.5$ (50%). However, Lawshe's scale offers three options ("Essential," "Important but not essential," "Unnecessary"), making the random probability $p = 1/3$ . If a "Not Applicable/No Answer" option is added, $p = 1/4$ .
Inappropriate Distribution: Wilson et al. [2012] used a normal approximation to calculate critical values for a discrete, non-continuous scale. Ayre & Scally [2014] corrected this by using a binomial distribution but retained the incorrect $p = 0.5$ assumption.
Lack of Paradox Detection: Existing methods only validate if an item is "Essential." They do not check if the same sample simultaneously validates an item as "Unnecessary." This can lead to paradoxical inconsistencies where a sample is split enough to validate an item as both essential and unnecessary, indicating heterogeneous perceptions.
One-Tailed Limitation: Current approaches use one-tailed tests to find the minimum number of experts needed to agree an item is essential. They fail to provide a symmetric mechanism to validate if an item should be discarded ("Unnecessary").

2. Methodology: Binomial Cut-Level Validation (BCV)

The authors propose the Binomial Cut-Level Validation (BCV) method, which utilizes a rigorous binomial distribution approach tailored to the specific number of options in the questionnaire scale.

Core Mathematical Framework

The method calculates the probability that $n$ experts selected a specific option purely by chance.

Formula: $P(n) = \binom{N}{n} p^n (1-p)^{N-n}$ $P (n) = (n N) p^{n} (1 - p)^{N - n}$
- $N$ : Total number of experts.
- $n$ : Number of experts selecting a specific option (e.g., "Essential").
- $p$ : Probability of random selection ( $1/3$ for a 3-point scale; $1/4$ for a 4-point scale including "NA").
- $\lambda$ : The cut-level (significance threshold), typically set at 0.05 or 0.01.

The Algorithm

Define Scale: Determine if the scale is $S_3$ (3 options) or $S_4$ (4 options). Set $p$ accordingly ( $1/3$ or $1/4$ ).
Calculate Critical Value ( $n_{critical}$ ): Determine the minimum integer $n$ such that the probability of $n$ or more experts choosing an option by chance is $\le \lambda$ .
Dual Validation (Double-Check):
- Essential Validation: Check if $n_{Essential} \ge n_{critical}$ AND $P(n_{Essential}) \le \lambda$ .
- Unnecessary Validation: Check if $n_{Unnecessary} \ge n_{critical}$ AND $P(n_{Unnecessary}) \le \lambda$ .
Decision Matrix: Classify items into four categories based on the dual validation results:
- Type A (Strong Retain): Validated as "Essential" but NOT "Unnecessary."
- Type B (Strong Paradox): Validated as BOTH "Essential" and "Unnecessary." (Indicates sample heterogeneity; requires deeper analysis).
- Type C (Weak Paradox): Validated as NEITHER "Essential" nor "Unnecessary." (Indicates lack of consensus; requires deeper analysis).
- Type D (Strong Discard): Validated as "Unnecessary" but NOT "Essential."

3. Key Contributions

Correction of Probability ( $p$ ): The proposal correctly sets $p$ based on the number of scale options ( $1/3$ or $1/4$ ) rather than the arbitrary $0.5$ used in previous literature.
Binomial Precision: It replaces normal approximations with exact binomial calculations, which is statistically more accurate for discrete, small-sample data.
Paradox Detection: It introduces a "double-tailed" validation logic. By testing for both "Essential" and "Unnecessary" simultaneously, it identifies Type B Paradoxes (items deemed both essential and unnecessary), a critical flaw in previous methods that could lead to invalid questionnaire designs.
Discard Capability: Unlike CVR, which only suggests retention, BCV provides a statistically robust method to recommend discarding items (Type D).
Expanded Tables: The authors provide pre-calculated tables (Appendix A) for sample sizes $N=5$ to $100$ (and up to 10,000 in a repository) for both 3-point and 4-point scales at $\lambda = 0.05$ and $0.01$.

4. Results and Comparison

The paper compares BCV against Lawshe, Wilson, and Ayre & Scally:

Sensitivity: BCV is more "cautious" yet more precise. It identifies items as "Essential" that previous methods might reject because it accounts for the lower random probability ( $1/3$ vs $1/2$ ).
Sample Size Impact: As sample size increases, the divergence between BCV and previous methods grows.
Paradox Identification: The study demonstrates that previous methods cannot detect the "Essential vs. Unnecessary" contradiction. BCV explicitly flags these inconsistencies, preventing the inclusion of ambiguous items in final questionnaires.
Flexibility: The method adapts to scales with 3 or 4 options, whereas previous methods were rigid regarding the 3-point scale assumption.

5. Significance

This research significantly advances the field of psychometrics and survey methodology by:

Improving Validity: It ensures that "essential" items are truly essential by correcting the statistical foundation of the CVR.
Enhancing Data Quality: By detecting paradoxical perceptions within a sample, researchers can identify when a questionnaire item is confusing or when the expert panel is too heterogeneous, leading to better study design.
Standardization: It provides a clear, mathematically sound algorithm for determining the critical number of respondents, moving the field away from heuristic tables (like Lawshe's) toward exact probability modeling.
Practical Application: The inclusion of decision tables (A, B, C, D) gives researchers a clear, actionable framework for item retention or elimination, reducing subjectivity in the validation process.

In conclusion, the Binomial Cut-Level Validity (BCV) proposal offers a more rigorous, statistically accurate, and comprehensive alternative to the traditional Content Validity Ratio, specifically addressing the mathematical errors in probability assumptions and the lack of paradox detection in current practices.