Examining the robustness of a model selection procedure in the binary latent block model through a language placement test data set

Technical Summary: Examining the Robustness of a Model Selection Procedure in the Binary Latent Block Model through a Language Placement Test Data Set

Problem Statement
The paper addresses the challenge of analyzing placement test data, specifically the SELF test used at Université Grenoble Alpes, to simultaneously cluster students into homogeneous groups based on proficiency and items into groups based on difficulty or skill. The data is represented as a binary matrix where rows correspond to students, columns to test items, and entries indicate correct (1) or incorrect (0) answers.

The core difficulty lies in the application of Latent Block Models (LBM) to this binary data. While LBM allows for simultaneous row and column clustering, selecting the correct number of row groups ($g$) and column groups ($m$) is non-trivial. Standard criteria like AIC and BIC are not directly applicable because the maximized likelihood cannot be computed for LBMs. Furthermore, estimation algorithms (such as Variational EM or Gibbs sampling) are sensitive to initial values and prone to converging to empty clusters or local optima. The authors note a lack of theoretical results or non-asymptotic concentration inequalities for model selection criteria in this specific context, necessitating a computational approach.

Methodology
The authors propose a model selection procedure based on the Integrated Completed Likelihood (ICL) criterion, extended for binary latent block models. The methodology involves three main components:

1. Estimation Algorithm: The study utilizes a combined V-Bayes Gibbs algorithm. This approach employs a Variational Bayes approximation to avoid empty groups and a Gibbs sampler to mitigate the sensitivity to initial values. The algorithm is run $T$ times (initializations) for each candidate pair of $(g, m)$, retaining the solution that maximizes the free energy.
2. Model Selection: The procedure iterates over a grid of possible $(g, m)$ pairs. For each pair, the ICL criterion is calculated using the partition $(\hat{z}, \hat{w})$ obtained via the Maximum A Posteriori (MAP) rule after the final V-Bayes step. The pair $(\hat{g}, \hat{m})$ maximizing the ICL is selected as the optimal model.
3. Robustness Analysis: The robustness of the procedure is evaluated in two ways:
   • Initialization Tuning: Determining the necessary number of initializations ($T$) to reliably recover the true model, tested via simulation and real data.
   • Sample Size Stability: Assessing the stability of the student groupings as the sample size ($n$) varies. This involves simulating subsamples of students and comparing the resulting partitions to a reference model using misclassification rates and contingency tables with label switching adjustments.

Data Sets
The study utilizes two real-world datasets from the SELF placement test:

• Japanese Test: 137 students, 33 items (first stage).
• English Test: 228 students, 36 items (first stage).
  Both datasets cover three skills: oral comprehension, written comprehension, and written expression.

Key Results

• Initialization Sensitivity: The required number of initializations ($T$) depends heavily on the "difficulty" of the dataset.
  • In the Japanese dataset (considered an "easy" case), the reference model $(3, 4)$ was found in 100% of 170,000 runs with a single initialization ($T=1$). The inter-arrival times for the reference model were all equal to 1.
  • In the English dataset (a "difficult" case), the reference model $(4, 5)$ was found only 16 times in 170,000 runs. The median inter-arrival time was 6,595.5, suggesting that hundreds or thousands of initializations are necessary to reliably find the optimal model in complex scenarios.
  • Simulation studies confirmed that for low noise levels ($\epsilon \leq 0.25$), $T=1$ is often sufficient, but as noise increases ($\epsilon = 0.3$), the distribution of $T$ shifts significantly higher.
• Robustness to Sample Size:
  • As the number of students ($n$) increases, the selected number of groups $(\hat{g}, \hat{m})$ converges toward the reference pair.
  • The rate of misclassified students decreases as $n$ increases when the correct number of groups is selected.
  • For smaller sample sizes and higher noise levels, the distribution of selected models becomes more scattered, though it still tends to concentrate around the reference as $n$ grows.
• Parameter Estimates: In the English dataset, the estimated parameters revealed one item group where the probability of a correct answer was similar across all student groups. The authors suggest this indicates noisy or irrelevant features that may hinder meaningful co-clustering.

Significance and Claims
The paper claims to provide a practical, computational framework for tuning model selection in binary latent block models where theoretical guarantees are absent. The primary contribution is the demonstration that the number of algorithm initializations must be adapted to the specific difficulty of the dataset; a "one-size-fits-all" approach is insufficient.

The authors modestly conclude that while the procedure is robust in terms of group stability as sample size increases, it can struggle with datasets containing noisy features (items with uniform success probabilities). As a future perspective, they suggest that incorporating a "noise class" into the latent block model (as proposed by Laclau and Brault) could improve results by isolating items that do not contribute to student classification, potentially aiding in the refinement of placement tests. The work does not claim to solve the theoretical identifiability issues but offers a validated computational strategy for practitioners dealing with placement test data.