Using Fine-Tuned LLMs to Identify Indicators of Vulnerability in UK Police Incident Logs
This study demonstrates that while fine-tuned open-source LLMs can estimate the prevalence of vulnerability indicators in UK police incident logs at a population level, their outputs require extensive human review and statistical correction to become defensible measurements, rendering them unsuitable for individual operational decisions due to persistent instability and error.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a detective trying to solve a mystery, but instead of looking at a crime scene, you are staring at a mountain of handwritten police notebooks. These notebooks are full of stories about people calling the police for help. Some stories are about lost cats, some about noisy neighbors, and some involve people who are very sick, struggling with addiction, or have nowhere to sleep. In the world of data science, these notebooks are called "unstructured text" because they aren't neat lists or checkboxes; they are messy, free-flowing stories. For a long time, computers were terrible at reading these stories to find specific patterns, like "is this person homeless?" or "are they struggling with mental health?" They could only read the neat checkboxes, which often missed the real story.
Enter Large Language Models (LLMs). Think of these as super-smart, digital reading machines that have read almost everything on the internet. They are great at understanding context, nuance, and the difference between a joke and a serious cry for help. But here's the catch: just because a robot can read a story doesn't mean it can always tell the truth about what's happening in it. Sometimes, the robot gets too excited and sees problems where there are none, or it gets confused by slang and shorthand. This paper asks a big question: Can we teach these digital reading machines to accurately count how many people in police stories are dealing with tough life situations, and can we trust the numbers they give us?
The Experiment: Teaching a Robot to Read Police Stories
The researchers in this study decided to test if they could use a fine-tuned LLM (a robot that has been specifically trained on examples) to read nearly 3,000 police incident logs from a UK police force. These logs are the daily diary of police work, written in real-time by officers and control room staff. The goal was to find out how often these logs mentioned four specific struggles: mental ill health, substance misuse, alcohol dependence, and homelessness.
They didn't just ask the robot to read once and give an answer. Instead, they built a careful, multi-step pipeline. First, they trained a smaller, local version of the robot (so the data stayed secure and didn't go out to the cloud) using examples from US police data. Then, they let the robot read each story five times. Why five times? Because, like a human who might be tired or distracted, the robot sometimes changes its mind. If it says "Yes, this person is homeless" five times in a row, that's a strong signal. If it says "Yes" three times and "No" twice, that's a signal of confusion.
The Findings: The Robot is Good at Saying "No," but Bad at Saying "Yes"
The results were a mix of good news and a big warning.
First, the robot was incredibly good at spotting when nothing was wrong. If a story was just about a traffic accident or a lost wallet, the robot confidently and correctly said, "No vulnerability here." It acted like a reliable filter, weeding out the boring stuff.
However, when the robot tried to find the problems, it got a bit too eager. It started seeing "homelessness" in stories about people staying in hotels, or "mental health issues" in stories about hospital visits that had nothing to do with mental health. It was like a detective who sees a shadow and immediately thinks it's a monster, even when it's just a coat rack.
To fix this, the researchers brought in human reviewers. They had a human look at a sample of the robot's work to see where it went wrong. They found that the robot was systematically over-estimating the problems. For example, the robot initially thought 12.8% of the stories involved substance misuse. But after the human review and some statistical math (a fancy way of correcting the robot's mistakes), the real number was likely much lower, around 5.0%.
The Final Numbers
After all the corrections and double-checking, the study produced some new estimates for how often these vulnerabilities appear in police work:
- Mental ill health: Found in about 23% of incidents (roughly one in five).
- Alcohol dependence: Found in about 8% of incidents.
- Substance misuse: Found in about 5% of incidents.
- Homelessness: Found in about 3% of incidents.
The Big Lesson: Don't Trust the Robot Alone
The most important takeaway from this paper is that you cannot just let a robot read police logs and trust the numbers it spits out. If you did that, you would get the wrong idea about how big the problem is. The robot is a powerful tool, but it needs a human teacher to check its homework.
The researchers showed that while the robot can read thousands of stories in seconds—something a human team would take years to do—it makes a specific kind of mistake: it sees problems where they might not exist. To get a trustworthy answer, you need a "multi-stage pipeline." This means running the robot multiple times, having humans check a sample of its work, and using statistics to correct the errors.
The paper concludes that while we can get a good estimate of the overall picture (like knowing that mental health issues are a major part of police work), we cannot trust the robot to make decisions about individual people. If a robot flags one specific person as "at risk," it might be wrong. But if we use this method to understand the big trends, it gives us a much clearer picture of the world than the old, messy checkboxes ever could. It's a powerful new lens, but one that needs to be held by a steady, careful hand.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.