Rule-Aware Bayesian Reconstruction and Local Auditing of Hidden Crowd Preferences

Technical Summary: Rule-Aware Bayesian Reconstruction and Local Auditing of Hidden Crowd Preferences

Problem Formulation
The paper addresses the challenge of auditing hybrid expert–crowd decision systems where professional scores and final outcomes (e.g., eliminations) are public, but the underlying crowd preference signal remains hidden. In systems like televised competitions, the aggregation of expert and crowd signals determines the final decision, yet multiple distinct crowd-share trajectories can produce the same observed outcome. Consequently, the inferential goal is not to reconstruct a single "true" vote total, but to characterize the joint posterior distribution of latent crowd utilities compatible with the observed active sets, aggregation rules, and elimination outcomes. The authors frame this as a dynamic inverse-preference problem, specifically investigating how different aggregation rules (percentage-based vs. rank-based) interact with latent preferences and how these systems can be audited for rule sensitivity and potential intervention.

Methodology
The authors propose a Rule-Aware Bayesian Dynamic Inverse Preference Model applied to 33 seasons of a televised competition (2,750 contestant-week records).

1. Latent State Dynamics: The model assumes a weekly latent log-utility ($z_{i,s,w}$) for each contestant that evolves via a Gaussian random walk. Crowd shares ($p_{i,s,w}$) are derived from these utilities using an active-set softmax function.
2. Rule-Specific Likelihoods:
   • Percentage Aggregation: The likelihood models the probability of elimination based on a combined score of judge share and crowd share ($C^P = q + p$), where lower combined scores imply higher elimination risk.
   • Rank Aggregation: The model uses a differentiable descending rank function to approximate discrete ranks, combining judge and fan ranks into a risk score ($C^R$).
   • Multi-Elimination Handling: For weeks with unordered sets of multiple eliminations, the authors introduce an unordered set likelihood. This sums the probabilities over all permutations of the eliminated set, retaining information that would otherwise be lost if only a single elimination were modeled.
3. Inference and Computation:
   • Posterior Sampling: The primary analysis uses the No-U-Turn Sampler (NUTS) to generate joint posterior draws, preserving cross-contestant and cross-week dependencies.
   • Approximation: A Laplace approximation (L-BFGS) is used for synthetic stress tests and rolling-origin predictions to ensure computational efficiency across thousands of fits.
4. Evaluation Framework:
   • Synthetic Recovery: Tests model performance against known ground truth under varying drift levels and rule types.
   • Strict Rolling-Origin Prediction: Evaluates the model's ability to predict the next elimination using only past data (excluding the target week's outcome).
   • Counterfactual Replay: Uses posterior draws to simulate alternative aggregation rules (e.g., swapping percentage for rank) or interventions at recorded decision states.
   • Adaptive Merit-Safe Protocol (AMSP): A proposed intervention that dampens the crowd component for contestants identified as technical outliers (low judge scores) to improve technical alignment while attempting to preserve crowd alignment and decision entropy.

Key Results

• Prediction vs. Consistency: The dynamic filter achieves 39.9% Top-1 accuracy in strict rolling-origin prediction, outperforming the static inverse model (35.5%) but falling slightly short of using judge scores alone (40.8%). The authors emphasize that the framework's primary value lies in its joint uncertainty distribution rather than point prediction accuracy.
• Uncertainty Quantification: Temporal structure significantly improves interval coverage. In synthetic tests, dynamic modeling increased 90% interval coverage by 0.074 to 0.109 across rule families compared to static models, though coverage declined under rapid drift and rank aggregation.
• Rule Divergence: Exact rank and percentage aggregation rules select different candidates in 27.6% of posterior draws (95% interval: 26.1–29.3%), indicating that the choice of aggregation rule materially alters outcomes even with identical inputs.
• Counterfactual Intervention (AMSP): In replay experiments, the AMSP would increase technical alignment by 0.034 (relative to judge rank) while retaining 91.1% of the baseline elimination entropy. It would alter the decision in 7.9% of posterior draws.
• Partner Analysis: Observational analysis suggests that assignment to "prior-elite" partners is associated with a 0.278 standard deviation increase in judge outcomes and a 0.263 increase in crowd support, though the authors note these are associations within the measured-covariate overlap population.

Significance and Claims
The paper claims to provide a measurable framework for hidden preference uncertainty and aggregation trade-offs at recorded decision states. Its primary contributions are:

1. Methodological: The development of a rule-aware Bayesian model that handles partial observability and unordered multi-elimination events, preserving joint posterior dependence via NUTS.
2. Audit Capability: The ability to quantify how often different aggregation rules diverge and to simulate the effects of transparent interventions (like AMSP) without altering the underlying data.
3. Practical Implication: The study suggests that for system designers, the value of such frameworks lies in retaining joint uncertainty objects and reporting trade-offs (technical alignment vs. crowd alignment vs. entropy) rather than seeking a single point estimate.

The authors maintain a modest stance, noting that the empirical evidence is specific to one competition's incentive structure. They caution that applying this to other domains (e.g., grant review) requires domain-specific likelihoods and observation models, and that full-season counterfactuals would require more complex transition models for active sets and audience allocation. The work is presented as a tool for local auditing and understanding the mechanics of hybrid decision systems, rather than a universal solution for all preference aggregation problems.