Cross-City Comparison of Crime-Prediction Models: A Scale Confound in Aggregate Poisson Likelihood, with a Four-City Demonstration

Technical Summary: Cross-City Comparison of Crime-Prediction Models

Problem Statement
The paper addresses a methodological gap in the cross-population evaluation of count-prediction models, specifically within the domain of sub-municipal crime forecasting. While aggregate event-weighted Poisson predictive log-likelihood (PLL) is widely used to compare model performance across jurisdictions with differing crime densities, the authors demonstrate that this metric is fundamentally confounded by zero-rate composition. Specifically, the per-event log-likelihood ceiling is mathematically dependent on the predicted mean ($\mu$), becoming more negative as $\mu$ increases. Consequently, populations with higher event densities (and thus lower zero rates) are systematically penalized in aggregate PLL rankings, regardless of actual model quality. This confound can reshuffle the apparent ranking of cities, leading to misleading conclusions about predictive performance.

Methodology
The study employs a four-city empirical demonstration using a unified LightGBM pipeline applied to crime incident data from Boston (USA), Leeds, Birmingham, and London (UK).

• Data: The dataset comprises H3 resolution-9 spatial cells (approx. 0.105 km²) aggregated at monthly granularity. Boston data (Jan 2023–Apr 2026) is daily-resolution NIBRS data, while UK data (Mar 2023–Feb 2026) is monthly Home Office data.
• Model: Independent LightGBM boosters with a Poisson objective are trained for each city and native crime type (36 total $(city, type)$ cells). The models utilize a 14-feature stack including log-population, temporal lags, rolling means, and neighborhood sums.
• Evaluation Protocol: The study uses an interleaved year-3 split (24 training bins, alternating even/odd bins for validation and testing). Metrics are computed on held-out test sets.
• Metric Analysis: The authors algebraically derive the Stirling approximation limit for Poisson log-likelihood to show the dependency on $\mu$. They compare four summary metrics: event-weighted mean PLL, nonzero_pll_mean (mean PLL conditional on $y>0$), calibration_ratio (observed mean/predicted mean), and zero_rate.
• Negative Results Catalog: The paper documents four specific gradient-space interventions tested against a divergence-detection diagnostic (defined as $\mu_{max} / \mu_{median} > 1000$) for shoplifting events: Tweedie loss ($p=1.5$), Negative Binomial (NB) loss with fixed dispersion, training-target winsorization, and single-type learning-rate reduction.

Key Contributions

1. Identification of Scale Confound: The paper provides both algebraic proof and empirical evidence that aggregate event-weighted PLL is not comparable across populations with differing zero-rate compositions. In the four-city corpus, three different metrics produced three different rank-1 cities (Birmingham on aggregate PLL, Boston on nonzero_pll_mean, London on calibration_ratio), with no city holding rank across all three.
2. Proposed Metric Pair: The authors propose nonzero_pll_mean and calibration_ratio as the primary pair for cross-population evaluation, with zero_rate reported as descriptive context. This approach isolates model performance on event occurrence from the structural properties of the data distribution.
3. Negative Results Catalog: The paper explicitly reports four configuration-specific interventions that failed to improve predictions in the tested regime. Crucially, the authors found that the divergence diagnostic itself was measuring spatial sparsity (a structural property of concentrated crime) rather than predictive pathology, rendering the interventions unnecessary or counterproductive.

Results

• Metric Disagreement: The study confirms that aggregate PLL rankings are driven by zero-rate composition. For instance, the 13.5 percentage-point difference in zero rates between Boston (74%) and London (61%) accounts for the majority of the aggregate PLL gap, despite comparable per-event performance.
• Calibration Improvements: A case study on Boston's calibration ratio (initially 0.849 vs. UK cities' ~0.93) revealed that the discrepancy was resolved not by model architecture changes, but by data-pipeline corrections: applying time-decay weights, excluding partial-month test bins, and using an interleaved split to balance seasonal effects.
• Intervention Outcomes:
  • Tweedie Loss ($p=1.5$): Failed to reduce divergence ratios and caused training pathology in other crime types.
  • Negative Binomial (Fixed $\alpha$): Converged models over-predicted significantly (calibration ratios 0.69–0.83) compared to the Poisson baseline.
  • Winsorization: 99.9th percentile clipping reduced divergence ratios without degrading other types, but was not adopted because the underlying diagnostic was mis-targeted.
  • Learning Rate Reduction: Reducing the learning rate by a factor of 5 for shoplifting cleared the divergence flag but catastrophically degraded prediction quality (calibration ratio $\approx 0.01$), as the model failed to learn the sparse spatial structure.

Significance and Scope
The paper positions itself as a methodological note rather than a generalization of predictive policing architectures.

• Scope Limitations: The authors explicitly state that the four-city sample ($n=4$) consists of high-income, OECD, English-speaking, common-law jurisdictions with similar policing institutions. Therefore, the study does not claim cross-jurisdictional generalization for the LightGBM architecture or the specific negative results.
• Primary Contribution: The core significance lies in the algebraic and empirical demonstration that aggregate PLL is a scale-confounded metric. The paper argues that cross-population comparisons must report nonzero_pll_mean and calibration_ratio alongside zero_rate to avoid misleading rankings.
• Procedural Recommendation: A key takeaway is the recommendation to investigate data-pipeline integrity (e.g., truncated bins, mismatched windows) before altering model architecture or loss functions when cross-population calibration discrepancies arise.
• Negative Results: The reported failures of specific interventions serve as configuration-specific observations for practitioners working with heavy-tailed, spatially concentrated count data, rather than categorical rejections of the techniques (e.g., NB with MLE-estimated $\alpha$ remains untested).

The paper concludes that the scale confound in aggregate PLL is a measurement issue upstream of broader debates on predictive policing ethics, necessitating a shift in how performance is reported in comparative studies.