Technical Summary: "Can LLMs Rank? A Tale of Triads and Triage"

Problem Statement

Large Language Models (LLMs) are increasingly deployed as "judges" for high-stakes resource allocation tasks, such as prioritizing households for emergency housing or triaging patients in emergency departments. While LLMs demonstrate sophisticated reasoning, their outputs are sensitive to presentation order and prompt framing, making them fragile for ranking large groups simultaneously. A common mitigation strategy, grounded in social choice theory, is to elicit pairwise comparisons and aggregate them into a total order.

However, a critical gap remains: practitioners lack a reliable method to assess whether an LLM's pairwise judgments are sufficiently consistent to trust the resulting ranking before deployment. Existing literature often treats consistency metrics and ranking accuracy as separate concerns, failing to explore the relationship between intra-run consistency (the internal logical consistency of a single set of judgments) and inter-run variance (the stability of the ranking outcome across multiple runs). This paper investigates whether these two axes of reliability are independent and how they behave in both synthetic and real-world LLM applications.

Methodology

Theoretical Framework

The authors propose a $2 \times 2$ taxonomy of ranking reliability defined by two independent axes:

1. Intra-Run Consistency (IC): Measured by the Coefficient of Consistency ($\zeta$). Originally developed by Kendall and Smith, $\zeta$ quantifies judge reliability by counting circular triads (triples $\{a, b, c\}$ where $a \succ b \succ c \succ a$) in a tournament graph. A value of $\zeta = 1$ indicates perfect transitivity, while $\zeta = 0$ indicates maximum cyclicity. The paper derives the asymptotic behavior of $\zeta$ as the number of candidates $n$ increases, showing it depends on the inconsistency rate $r$ and a scaling factor.
2. Inter-Run Variance (IV): Measured by Kendall's $\tau$ distance between rankings generated by independent runs of the same pipeline. This captures the stability of the final ranking.

The authors theoretically characterize the relationship between $n$, $\zeta$, and $\tau$ under the Bradley-Terry-Luce (BTL) model, a standard generative model for pairwise comparisons with known ground truth.

Experimental Design

The study evaluates these metrics across two domains:

• Homelessness Housing Allocation: Using St. Louis coordinated entry records (VI-SPDAT, VIF-SPDAT, TAY-VISPDAT) for single adults, families, and transition-age youth.
• Emergency Department Triage: Using MIMIC-IV data with Emergency Severity Index (ESI) labels.

Models: Three leading LLMs were tested locally: Meta-Llama-3-8B-Instruct, DeepSeek-R1-Distill-Llama-8B, and Qwen2.5-7B-Instruct.

Procedure:

1. Subtournament Evaluation: For each dataset, a complete subtournament of $m=30$ items was generated ($435$ pairwise comparisons per model). $\zeta$ was computed exactly, and self-consistency ($\tau_{cons}$) was measured via bootstrap subsampling.
2. Full Sparse Pipeline: The pipeline was run on larger sets ($n$ up to 698) with edge sampling probability $p=0.4$ to simulate real-world constraints. The authors measured how $\hat{\zeta}$ (estimated from sparse data) and $\tau_{cons}$ scaled with $n$.

Key Contributions

1. Theoretical Characterization of $\zeta$: The paper proves that under the BTL model with fixed discriminability, the inconsistency rate $r$ decreases as $\Theta(1/n^2)$, causing $\zeta$ to converge toward 1 as $n$ increases. This establishes a theoretical baseline where consistency improves with scale.
2. Decoupling of Consistency Axes: Through synthetic experiments, the authors demonstrate that $\zeta$ and $\tau_{cons}$ are correlated but not redundant. Specifically, $\zeta$ measures the quality of the comparison signal (intransitivity), while $\tau_{cons}$ depends on both signal quality and the quantity of comparisons available per item.
3. Empirical Divergence in Real Data: Contrary to the BTL model, real-world LLM data exhibits a constant inconsistency rate ($r$) as $n$ increases. Consequently, $\zeta$ stabilizes near an asymptotic floor rather than improving, while $\tau_{cons}$ continues to increase with $n$ due to the accumulation of data points.
4. Model-Specific Performance Profiles: The study reveals that no single LLM dominates both axes.
   • LLaMA: Consistently achieves the highest intra-run consistency ($\zeta$) but exhibits lower inter-run stability.
   • Qwen: Achieves the highest inter-run stability ($\tau_{cons}$) but has the lowest intra-run consistency.
   • DeepSeek: Typically occupies a middle ground on both dimensions.

Results

• Synthetic vs. Real Behavior: In synthetic BTL data, $\zeta$ increases with $n$. In real LLM-generated data, $\zeta$ remains roughly constant across different values of $n$, indicating that the inconsistency rate $r$ does not decrease with scale as predicted by idealized models.
• Diagnostic Utility: The sparse estimator $\hat{\zeta}$ is shown to be unbiased across different observation probabilities ($p$), allowing practitioners to estimate comparison quality even with incomplete data.
• Trade-offs: The paper highlights a critical trade-off: a model with high $\zeta$ (LLaMA) may produce rankings that are internally logical but unstable across runs, while a model with high $\tau_{cons}$ (Qwen) may produce stable rankings that are internally intransitive.

Significance and Claims

The paper claims that relying on a single metric (either $\zeta$ or $\tau_{cons}$) is insufficient and potentially misleading for assessing LLM reliability in high-stakes ranking tasks.

• Independent Axes: The authors assert that intra-run consistency and inter-run variance are genuinely independent axes of reliability. A system can be internally consistent but unstable (low IV, high IC), or stable but internally intransitive (high IV, low IC).
• Practical Diagnostic Protocol: The paper proposes a two-step diagnostic for practitioners:
  1. Run a small, complete subtournament to compute $\zeta$. If $\zeta$ is low, the comparison process itself is fundamentally intransitive, and no amount of additional data will fix the ranking.
  2. If $\zeta$ is high but $\tau_{cons}$ is low, the issue is insufficient data quantity, not signal quality; the remedy is to increase the sampling fraction rather than change the model.
• Model Selection: The results suggest that model selection must be domain-specific. If a stakeholder prioritizes the logical coherence of individual judgments, LLaMA may be preferred. If they prioritize the stability of the final output across runs, Qwen may be preferred.

The paper concludes that the proposed taxonomy provides a necessary framework for moving beyond simple accuracy metrics to a more nuanced understanding of AI reliability in consequential decision-making.