The Marginal Likelihood of two-way tables and Ecological Inference

Technical Summary: The Marginal Likelihood of Two-Way Tables and Ecological Inference

Problem Statement
The paper addresses the fundamental statistical challenge of inferring the association structure (specifically, conditional row distributions) of a two-way contingency table when only the row and column marginal totals are observed. This problem is central to Ecological Inference, a methodology used to estimate individual-level behaviors (e.g., voter transitions between elections) from aggregate data (e.g., polling station results) where individual joint decisions are unobservable due to ballot secrecy.

The paper investigates two distinct scenarios:

1. Single Table: Inferring conditional probabilities from the margins of a single $R \times C$ table.
2. Collection of Tables: Inferring a common set of conditional row distributions across a collection of $s$ tables (local units) that share the same association structure but have different marginal totals.

Methodology and Theoretical Framework

1. The Single Table Case ($R \times C$)
The author generalizes Plackett's (1977) work on $2 \times 2$ tables to general $R \times C$ tables. The analysis focuses on the marginal likelihood under multinomial sampling.

• Parametrization: Conditional row probabilities are parameterized using logits ($\phi_j$) and log-odds ratios ($\lambda_{ij}$) relative to a baseline category.
• Likelihood Equations: The paper derives the likelihood equations, showing that standard unconstrained estimates ($\hat{p}_{j|i} = n_{ij}/n_{i0}$) satisfy the marginal constraints but fail to satisfy the conditional expectation constraints ($M_{ij} = n_{i0}p_{j|i}$) for the observed table.
• Extreme Tables: The core theoretical contribution involves the concept of Extreme Tables. These are frequency tables constructed via specific permutations of row and column categories (Algorithm 1) that represent the upper bounds of the Fréchet class of distributions with given margins.
  • The paper defines a sequence of probability distributions $P(Z, \epsilon)$ approaching an extreme table $Z$ as $\epsilon \to 0$.
  • In this limit, the non-zero log-odds ratios diverge to $+\infty$ (or $2\xi$), creating the strongest possible positive association within the Fréchet class.
• Local Maxima: It is proven that these limiting distributions satisfy the likelihood equations, meaning the marginal likelihood possesses a collection of local maxima achieved at these extreme tables.

2. The Collection of Tables Case
Recognizing that inference from a single table is essentially impossible (as the likelihood is maximized at pathological extreme associations), the paper shifts to a collection of $s$ tables sharing the same conditional row probabilities.

• Likelihood Construction: The exact multinomial marginal likelihood is formulated for the collection, requiring the summation over all possible tables $N_h$ compatible with the observed margins of each unit $h$.
• Fisher Scoring Algorithm: To maximize this complex likelihood, the author proposes an efficient Fisher scoring algorithm.
  • Instead of the expected information matrix (which is difficult to compute), the method utilizes an empirical information matrix derived from score vectors.
  • The score vectors are computed using conditional expectations ($M_{hij}$) under the extended hypergeometric distribution.
  • A necessary condition for identifiability is established: the number of local units $s$ must be at least $R(C-1)$ (assuming non-degenerate data).

Key Results

1. Inconsistency of Single-Table MLE
The paper demonstrates that Maximum Likelihood Estimates (MLE) for a single table are inconsistent.

• As sample size increases, the global maximum of the likelihood is attained at one of the extreme tables.
• These extreme tables correspond to distributions with the strongest possible positive association (after suitable permutation), which generally do not reflect the true underlying association structure.
• Consequently, likelihood-based inference on a single table yields "inconclusive" or misleading results, confirming Plackett's earlier findings for $2 \times 2$ cases.

2. Performance of the Proposed Algorithm
A simulation study was conducted with $s=60$ local units, $R=C=3$, and $n=40$ observations per unit. The proposed Fisher scoring algorithm (Exact Likelihood) was compared against:

• Goodman's linear regression method (1953).
• The modified Brown and Payne method (2009).

Findings:

• The Exact Likelihood method produced estimates ($\hat{\Pi}$) closest to the true generating mechanism ($\Pi$), with a mean squared error ($M_e$) of 0.0537.
• The modified Brown and Payne method yielded $M_e = 0.0549$.
• Goodman's method yielded $M_e = 0.0551$.
• The results indicate that while the proposed method is theoretically superior, the performance gap with established methods in this specific simulation was modest. The author notes that Goodman's method, despite its limitations (e.g., potential for probabilities outside $[0,1]$), performs surprisingly well.

3. Computational Complexity
The paper highlights a significant computational bottleneck: constructing and storing the collection of all compatible tables ($N_h$) for each local unit.

• For a $3 \times 3$ table, the size of $N_h$ grows rapidly with sample size $n$ (approx. $10^5$ tables for $n=100$, $>10^6$ for $n=200$).
• The author notes that applying this method to real-world electoral data (where polling stations often have $n \approx 800$) is currently infeasible without approximations or subset selection strategies.

Significance and Claims
The paper claims to provide:

1. Geometric Insight: A clarification of the geometry of the collection of tables with fixed margins, specifically identifying that local maxima of the marginal likelihood correspond to extreme tables with diverging log-odds ratios.
2. Theoretical Resolution: A proof that MLE for a single table is inconsistent, explaining why standard likelihood inference fails in ecological inference contexts without additional assumptions or data.
3. Algorithmic Contribution: An efficient Fisher scoring algorithm for maximizing the exact multinomial likelihood when data from multiple local units are available.
4. Empirical Validation: Evidence that the exact likelihood approach yields estimates closer to the truth than existing methods, though the practical advantage in small simulations is marginal.

The author concludes that while the exact method is theoretically robust, its application to large-scale real-world data is currently limited by computational constraints, suggesting a need for future work on meaningful subsets of the table collection to approximate the score and information matrices.