Estimated Ranking Tables and Uncertainty

Technical Summary: Estimated Ranking Tables and Uncertainty

Problem Statement
National statistical agencies frequently release tables containing sample estimates for $K$ populations (e.g., states, countries, or districts). Traditionally, these tables are ordered alphabetically or by some non-quantitative criterion, with rows containing primary estimates and associated measures of uncertainty (e.g., standard errors). The author argues that this presentation creates "implicit estimated rankings," forcing users to manually search for the smallest or largest estimates to determine relative standings. This process is inefficient, particularly when $K$ is large (e.g., 51 U.S. states), and fails to explicitly communicate the uncertainty inherent in the ordering itself. While users often re-order data to create "explicit" rankings, these user-generated tables typically lack statistical statements regarding the uncertainty of the new ordering. The paper posits that tables produced by statistical agencies should be presented explicitly as estimated ranking tables, where rows are ordered by the magnitude of the estimates, and the table includes measures of uncertainty for the overall ranking.

Methodology
The paper proposes a framework for constructing and visualizing an Estimated Ranking Table accompanied by a DIFF Joint Confidence Region for the true ranking of $K$ populations.

1. Definitions and Assumptions:

   • Rank: The true rank $r_k$ of population $k$ is defined as $1$ plus the number of other populations with means less than or equal to $\bar{Y}_k$.
   • Estimates: The paper assumes $K$ independent probability samples yielding estimates $\bar{y}_k$ with known standard errors $SE(\bar{y}_k)$.
   • Normality: It assumes $\bar{y}_k \sim N(\bar{Y}_k, SD(\bar{y}_k))$.
   • Ordering Principle: The paper advocates for the "First Quadrant Principle," where higher estimates are visually placed higher (or lower, depending on axis orientation, but consistently correlated) to ensure harmony between the estimate magnitude and the assigned rank order.

2. The DIFF Joint Confidence Region:
   The core methodology relies on constructing a joint confidence region for the vector of true ranks $(r_1, \dots, r_K)$ using differences between population parameters.

   • Pairwise Intervals: The method constructs $\binom{K}{2}$ joint confidence intervals for the differences $\bar{Y}_k - \bar{Y}_{k'}$.
   • Bonferroni Correction: To achieve an overall joint confidence level of $100(1-\alpha)\%$, a Bonferroni correction is applied. The individual confidence level is adjusted to $1 - \gamma^*$, where $\gamma^* = \alpha / \binom{K}{2}$. The intervals are:
     $$(\bar{y}_k - \bar{y}_{k'}) \pm z_{\gamma^*/2} \sqrt{SE(\bar{y}_k)^2 + SE(\bar{y}_{k'})^2}$$
   • Classification of Differences: For each population $k$, the intervals for $\bar{Y}_k - \bar{Y}_{k'}$ are classified into three sets based on their position relative to 0:
     • $AR_k$: Intervals lying entirely to the right of 0 (implying $\bar{Y}_k > \bar{Y}_{k'}$).
     • $AL_k$: Intervals lying entirely to the left of 0 (implying $\bar{Y}_k < \bar{Y}_{k'}$).
     • $AO_k$: Intervals overlapping 0 (implying the order is indeterminate).
   • Rank Sets: Based on Theorem 1, the set of possible true ranks for population $k$ is:
     $$\{ |AR_k| + 1, |AR_k| + 2, \dots, |AR_k| + |AO_k| + 1 \}$$
   • Joint Region: The joint confidence region is the Cartesian product of these individual rank sets, constrained by the requirement that ranks must be unique (a permutation). The visual representation is a grid where columns represent populations (ordered by estimated size) and rows represent ranks. Cells contain the population abbreviation if that population can occupy that rank within the confidence region.

3. Visualization:
   The paper introduces a theory-based visual (a scatterplot-style grid) that simultaneously displays:

   • The estimated ranking (bolded entries).
   • The uncertainty in the ranking (non-bolded entries).
   • Marginal confidence sets for each population's true rank.
   • Marginal confidence sets for each rank (which populations could occupy it).
   • The "tightness" of the region, defined as $T^* = (K^2 - OP) / (K^2 - K)$, where $OP$ is the number of occupied cells.

Key Results and Examples
The paper illustrates the methodology using three real-world datasets:

1. U.S. States (K=51): Using 2011 American Community Survey data on mean travel time to work. The DIFF 90% joint confidence region reveals that uncertainty is lowest at the extremes (top and bottom ranks) and highest in the middle ranks. States with highly urbanized, high-population characteristics (Group 1) showed the least uncertainty, while states with mixed urban/rural characteristics (Group 3) showed the most.
2. Indian States (K=18): Using 2022-23 National Sample Survey data on Monthly Per Capita Consumption Expenditure (MPCE). The visual demonstrates how the method handles varying standard errors and estimate distributions across 18 major Indian states.
3. OECD Countries (K=37): Using 2022 PISA mathematics scores. The region visualizes the uncertainty in the global ranking of education systems, showing which countries could plausibly swap ranks given the sampling error.

In all cases, the paper contrasts the proposed "Estimated Ranking Table" (ordered by magnitude) with traditional "Alphabetical Tables," arguing the former allows for immediate visual comparison of relative standings.

Significance and Claims
The paper claims that presenting data as explicit estimated ranking tables with associated joint confidence regions offers several advantages:

• Efficiency: It reduces the cognitive load on users by eliminating the need to manually sort data to find relative standings.
• Transparency: It explicitly communicates the uncertainty of the ranking, preventing the misinterpretation of small differences in estimates as definitive differences in rank.
• Comprehensive Uncertainty: Unlike methods that only provide confidence intervals for individual ranks, the DIFF joint confidence region provides a coherent view of the entire ranking structure, revealing possible alternative true rankings.
• Visual Clarity: The proposed visualization allows users to see at a glance which populations are statistically indistinguishable in rank and which are distinct.

The author notes that while previous efforts (such as the INDI/KWW method) provided similar insights, the DIFF method offers a "tighter" joint confidence region, providing more precise bounds on the possible true rankings. The paper concludes that statistical agencies should maximize the use of explicit estimated ranking tables and minimize the use of alphabetically ordered tables to better serve data users.