📊 statistics

The Likelihood Ratio Wall: Structural Limits on Accurate Risk Assessment for Rare Violence

This paper argues that pretrial risk assessment tools face a fundamental "Likelihood Ratio Wall" where the rarity of violent re-offenses and structural biases like over-policing mathematically prevent high-confidence individualized predictions, rendering current instruments prone to high false-positive rates that cannot be fixed by recalibration.

Original authors: Marco Pollanen

Published 2026-05-01

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Marco Pollanen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge trying to decide whether to keep a defendant in jail before their trial. You have a new "Risk Tool" that scans their history and gives them a label: "High Risk for Violence."

The tool is supposed to tell you: "If we let this person go, there is a good chance they will commit a violent crime."

This paper argues that, mathematically, this tool is almost always lying to you when it says someone is "High Risk" for rare violent crimes. Even if the tool looks impressive on paper, the reality is that most people it flags as "dangerous" would actually have stayed peaceful if released.

Here is the breakdown of why this happens, using simple analogies.

1. The "Needle in a Haystack" Problem (The Base Rate)

Violent re-arrests are very rare. In most places, only about 2 to 5 out of every 100 released defendants commit a new violent crime.

Imagine you are a security guard at a massive stadium with 10,000 people. You are looking for one person who is secretly carrying a weapon (the "violent offender").

The "Risk Tool" is a metal detector.
The problem is that the metal detector is not perfect. It sometimes beeps for harmless things like belt buckles or keys (false alarms).

Because the "bad guy" is so rare (1 in 10,000), even a very good metal detector will ring for innocent people far more often than it rings for the actual threat. The paper calls this the Likelihood Ratio Wall.

2. The Wall You Can't Climb

The authors prove a mathematical rule: To be right 50% of the time (meaning, if the tool says "High Risk," there is a coin-flip chance the person is actually dangerous), the tool needs to be incredibly good at spotting the needle.

The Reality: Current tools are like a metal detector that beeps for belt buckles 10 times for every actual gun it finds.
The Requirement: To be right half the time, the tool would need to be 30 to 50 times better at spotting the gun than the belt buckle.

The Analogy: Imagine trying to find a specific grain of sand on a beach. If your "sand finder" picks up 100 grains of sand for every 1 grain of gold, and gold is incredibly rare, you will end up holding a bucket full of sand thinking you found gold. The tool isn't "broken"; it's just that the gold is too rare for the tool to work well.

The paper shows that current tools are nowhere near good enough to clear this wall. When they flag someone as "High Risk," they are usually wrong.

3. Why "Tweaking" the Tool Doesn't Help (Recalibration)

You might think, "Can't we just adjust the tool's settings to make it more accurate?"

The authors say no. They prove that changing the numbers (recalibration) is like repainting the metal detector. You can make it look fancier or change the volume of the beep, but you cannot change the fact that it still beeps for belt buckles.

If the underlying data (the history of arrests, prior charges, etc.) doesn't contain a strong enough signal to separate the violent from the non-violent, no amount of math tricks can create that signal out of thin air. The "Wall" is a structural limit, not a software bug.

4. The "Surveillance Ceiling" (Why Some Groups Get Flagged More Often)

The paper also points out a cruel irony: Over-policing makes the tool worse for certain groups.

Imagine two neighborhoods, Neighborhood A and Neighborhood B.

Neighborhood A is policed normally.
Neighborhood B is over-policed. The police are there constantly, checking everyone's pockets and records.

Even if people in Neighborhood B are no more violent than people in Neighborhood A, they accumulate more "risk markers" (like prior arrests or failed court appearances) simply because they are watched more closely.

The Analogy:

In Neighborhood A, a person might get a "risk marker" only if they actually do something wrong.
In Neighborhood B, a person gets a "risk marker" just for being in the wrong place at the wrong time, or for a minor infraction that police in Neighborhood A would have ignored.

The paper proves that because Neighborhood B has more "false markers" (innocent people flagged by the system), the tool becomes less accurate for them. The tool hits a "Surveillance Ceiling" where it simply cannot be precise, no matter how smart the algorithm is. It ends up flagging innocent people in Neighborhood B at a much higher rate, making the "High Risk" label even less trustworthy for them.

5. The "Number Needed to Detain" (The Human Cost)

The authors translate these math problems into a real-world cost: How many people do we have to lock up to stop one violent crime?

If the tool is right 11% of the time (which is what the math says happens at current performance levels), it means you have to detain 9 people to prevent 1 violent crime.
This implies that 8 of those 9 people were innocent of future violence and were deprived of their liberty unnecessarily.

The Bottom Line

The paper concludes that for rare violent crimes, the data we currently have (criminal records, arrest history) is not strong enough to support high-confidence decisions about locking people up.

When a judge sees a "High Risk for Violence" label, they are being told the person is likely to be dangerous. But the math says that label is actually wrong about 8 or 9 times out of 10.

The Recommendation:
The authors suggest that if we keep using these tools, we must be honest about the uncertainty. Instead of just saying "High Risk," the reports should say:

"This person is flagged as high risk, but based on current data, there is only an 11% chance this flag is correct. To prevent one violent crime, we would likely detain 9 people who would not have committed a crime."

The paper argues that until we have data that is 30 to 50 times better than what we have now, we should stop treating these "High Risk" labels as proof that someone is dangerous.

1. Problem Statement

The paper addresses a fundamental statistical barrier in the use of algorithmic pretrial risk assessment tools (e.g., COMPAS, PSA) for predicting rare violent re-offenses. While these tools classify millions of defendants annually, their utility for making high-stakes detention decisions is compromised by the low base rate of violent recidivism (typically 2–5%).

The core problem is the Positive Predictive Value (PPV) deficit. Judges and policymakers often interpret a "high-risk" flag as implying a high probability that the individual will commit violence. However, the paper argues that due to the rarity of the outcome, even tools with respectable ranking metrics (like AUC) produce "high-risk" flags that are incorrect the majority of the time. The paper seeks to quantify the structural limits of accuracy and demonstrate that post-hoc fixes (like recalibration) cannot overcome these limits.

2. Methodology

The paper employs a combination of theoretical derivation, statistical modeling, and empirical validation:

Bayesian Derivation: The author derives a universal bound linking the Likelihood Ratio (LR), base rate ( $\pi$ ), and PPV using Bayes' theorem.
Theoretical Proofs:
- Recalibration Invariance: Proving that monotonic transformations of risk scores (used in recalibration) preserve the Likelihood Ratio and thus cannot improve the separation of true positives from false positives.
- Surveillance Ceiling Model: Developing a stylized model of "count-threshold" classifiers (common in risk tools) to demonstrate how unequal surveillance (over-policing) inflates false positive rates in specific groups, structurally lowering the maximum achievable PPV for those groups even if base rates are equal.
Empirical Validation:
- Meta-Analysis: Reviewing existing literature on AUC and translating it into estimated Likelihood Ratios.
- Dataset Analysis: Using the ProPublica COMPAS dataset (Broward County, FL) to calculate actual sensitivity, false positive rates, and LRs for violent recidivism.
- Stratified Analysis: Comparing performance metrics across racial groups to test the "Surveillance Ceiling" hypothesis.

3. Key Contributions

A. The Likelihood Ratio Wall (Theorem 1)

The paper formalizes a universal precision bound. It proves that for a classifier to achieve a specific PPV ( $\alpha$ ) at a given base rate ( $\pi$ ), it must satisfy a minimum Likelihood Ratio:
$LR \geq \frac{\alpha}{1-\alpha} \cdot \frac{1-\pi}{\pi}$

Implication: As the base rate ( $\pi$ ) decreases, the required LR increases hyperbolically.
Example: At a 3% base rate, achieving a 50% PPV (the "preponderance of evidence" standard) requires an LR of 32.3. Current tools typically achieve an LR of only 2–6.

B. Recalibration Invariance (Theorem 3)

The paper proves that recalibrating scores (e.g., Platt scaling, isotonic regression) to match observed frequencies preserves the ordering of individuals. Since the Likelihood Ratio depends only on the ranking (sensitivity vs. false positive rate) and not the absolute probability values, recalibration cannot fix the PPV deficit. It improves probability calibration but not the tool's discriminative power.

C. The Surveillance Ceiling (Theorem 4)

The paper introduces a formal model showing that over-policing degrades predictive precision.

Mechanism: In over-policed communities, individuals accumulate administrative "risk markers" (e.g., prior arrests, failures to appear) at higher rates due to increased surveillance, even if they do not commit crimes (inflating the negative class distribution).
Result: This increases the False Positive Rate (FPR) disproportionately. The theorem proves that for count-threshold classifiers, the maximum achievable LR (and thus PPV) is structurally lower for over-policed groups compared to less-polished groups, even if the true underlying offense rates are identical.

D. Policy Metric: Number Needed to Detain (NND)

The author translates statistical findings into a policy-relevant metric: NND = 1/PPV. This represents the number of defendants who must be detained to prevent one violent offense.

Finding: At current performance levels, NND ranges from 6 to 13, meaning 5 to 12 innocent people are detained for every one violent offender prevented.

4. Key Results

The "99-to-1 Rule": At a 1% base rate, a tool must be roughly 99 times more likely to flag a true positive than a false positive to achieve 50% PPV. Current data suggests no such signal exists in criminal history records.
Empirical LR Gap: Analysis of the COMPAS dataset shows an LR of 4.3 for violent recidivism. At a realistic 3% base rate, this yields a PPV of only 12% (i.e., an 88% chance the "high-risk" flag is wrong).
Surveillance Disparity: In the COMPAS data, Black defendants had 1.8x more risk factors on average than White defendants, but their False Positive Rate was 2.9x higher. This superlinear amplification confirms the Surveillance Ceiling mechanism.
AUC Misleading: The paper demonstrates that a tool can maintain a constant AUC (e.g., 0.70) across different base rates, while the practical PPV of a "high-risk" flag drops from ~70% (at 50% base rate) to ~10% (at 3% base rate).

5. Significance and Implications

Structural Impossibility: The inability of current tools to support high-confidence individualized detention decisions is not a bug of specific algorithms but a structural limitation of the data. The available features (criminal history) do not contain enough signal to overcome the base rate of rare violence.
Fairness Beyond Metrics: The paper argues that debates focusing solely on fairness metrics (e.g., equalizing false positive rates across groups) are incomplete. Even if groups are treated equally by the algorithm, unequal surveillance creates a "Surveillance Ceiling" that makes high-precision prediction impossible for over-policed groups.
Policy Recommendations:
1. Transparency: Risk reports should explicitly disclose uncertainty (e.g., "This flag is correct only 1 in 9 times").
2. Metric Shift: Reporting should move from AUC to PPV, LR, and NND.
3. Labeling: "High Risk for Violence" labels should not be used to support high-confidence detention decisions when PPV is below 20%.
4. Caution on Data: The paper warns against using these findings to justify more invasive data collection, arguing instead for transparency about the limits of current data.

In conclusion, the paper establishes that for rare violent outcomes, the "Likelihood Ratio Wall" prevents current risk assessment tools from providing the level of certainty required for liberty-depriving decisions, a problem exacerbated by systemic surveillance disparities that no algorithmic tuning can currently resolve.