# Inside an English Region's Secret Algorithm That Scored Thousands of People for Risk, Then Quietly Switched Off the Models No One Could Trust

> Police and a council in southwest England spent years building a vast system that scored thousands of adults and children for risk, but documents show at least two models were quietly abandoned after staff decided their results could no longer be trusted.

**Type:** article · **Category:** Security · **Published:** 2026-06-25 · **Source:** TrendKia
**Canonical:** https://trendkia.com/en/security/hazaron-logon-ko-khatare-ka-skora-dene-vali-pulisa-mashina-jisake-natijon-para-khuda-aphasaron-ka-bharosa-tuta-gaya-2953 · **Language:** English
**Tags:** predictive policing, Avon and Somerset Police, Think Family Database, AI surveillance, Bristol, Offender Management App, algorithmic bias, College of Policing

In a corner of southwest England, the police and a city council spent the better part of a decade quietly assembling something extraordinary: a vast prediction engine that scored thousands of adults and children for the danger they posed, or the danger they faced. Years later, many of the people whose lives it touched still cannot find out what it concluded about them. And in at least two cases, the very staff meant to lean on those scores decided they simply could not be trusted.

The story begins in 2016, when Bristol City Council and the regional Avon and Somerset Police jointly built a database. Into it went all manner of deeply personal information: police intelligence reports, people's housing status, mental health records, teenage pregnancies, enrollment in parenting courses, even free school meals. On top of this sensitive material, officials built machine-learning models that assigned scores to thousands of people. Their goal was to build what they described as a picture of threat, harm, and risk across the region.

## The Most Personal Data, Tipped Into a Bucket
At an event in early 2022 to help officials tackle child exploitation crimes, one police data scientist described part of the approach like this: "I essentially dump all that data in a big bucket and stir it with a data-science spatula, and we come out with a lovely risk score for everybody."

This risk scoring inside the Think Family Database was only one slice of Avon and Somerset Police's sprawling predictive analytics program. The force created at least 23 separate models, including algorithms to gauge the risk that a person would commit burglary, fail to appear in court, go missing, or become a victim of domestic abuse. One senior officer spoke of creating a league table of the area's most dangerous criminals, an apparent reference to the Offender Management App, which was designed to hold data on around 300,000 people in the region.

## John Pegram, Who Had No Idea He Was on a List
How the police developed and used these forecasting tools has not always been clear to the public. John Pegram, who leads a local police accountability group in Bristol, says he did not hear about the Offender Management App until 2023, years after it had been built. When he finally did, he began to suspect he might be on it. "I think I knew I was on the app," Pegram says.

In early 2024, Pegram filed a request to find out how the police were using his data. They refused to say. Months later, after he had hired solicitors to work on his case, the police confirmed he was on the app but declined to say anything more. Like others across Bristol, the UK, and, increasingly, the world, Pegram did not know whether an algorithm had scored him, what that score might be, or how it could shape his dealings with the authorities.

## What the Documents Reveal
Hundreds of pages of documentation obtained through public records requests now form the most comprehensive picture to date of Avon and Somerset's regional experiment in data collection and predictive analytics. (Pegram's litigation is backed by the civil liberties organization Liberty, which had early involvement in a potential legal challenge to the program.)

The documents reveal that at least two of these risk-scoring models were quietly abandoned after Bristol City Council staff decided they could no longer trust them. Previously unreported records show government inspectors and independent reviewers flagging a startling lack of transparency about parts of the program and warning that the systems could erode public trust. Police data running to more than 36,000 model performance scores appears, in some cases, to show genuinely poor predictive performance, according to an independent analyst who reviewed it.

The findings land just as the UK appears ready to embrace predictive analytics and artificial intelligence across the criminal justice system. A familiar figure is helping lead the charge: Andy Marsh, the former chief constable of Avon and Somerset, who now heads the national standard-setting body for forces across England and Wales. As CEO of the College of Policing, Marsh has said effective AI should be "injected like heroin" to speed up British police work. In a recent interview, he said his organization was examining around 100 currently deployed AI tools, including for predictive policing. "Our job is to test the ones that work properly, test them with rigorous evaluation, and then spread them like wildfire through policing."

## How the Whole Experiment Began
Two years earlier, Gary Davies, a former police chief superintendent, had moved into a role at Bristol City Council, and he was thinking along similar lines. Davies led a council team that supported children and families. When a family was in crisis, he says, "it was blatantly obvious." The harder task was spotting those just beginning a downward spiral.

Davies believed the answer lay in data. A child's school might have a record of rising absences, while the police might know the child had recently witnessed domestic abuse for the first time. On their own, these signals might not be enough to trigger an intervention from social services. But together? "If you could see the whole picture, you would realize that the trajectory they were on was going in the wrong direction," he says.

Starting in 2015, a small group of Bristol City Council and Avon and Somerset Police staff moved into one of the city's police stations to tackle the problem together. The Insight Bristol team, headed by Davies, began pulling data from across the public sector to give frontline workers everything they might need to know about children and families.

## The Question of Consent
The Insight Bristol team did not seek residents' consent to use their data in the Think Family Database. Instead, Davies explains, the team relied on legal gateways, a term for cases where data sharing is deemed necessary to meet an agency's legal duties, such as protecting children. "If you were to give the impression that people had consent, then it creates a false illusion, because, actually, as a local authority or police or whoever, we have to keep those records." At first, residents could not opt out of the database; later, the council added an opt-out option to its tax letters.

Davies, who recently retired, believes the project did help protect children. "It improved the understanding of risk and vulnerability for children and families," he says. "It provided that information in a far more efficient way." Communicating it to the public, he says, was hard: "It was fairly difficult to get any enthusiasm or interest from groups of people." Those who did engage understood the need to use personal data, he recalls, summing up the feedback as: "We don't mind you using it to support us, but we don't want you to use it against us."

## Ethical Warnings Sounded From the Start
While the Insight Bristol team was building the Think Family Database, Avon and Somerset Police had begun exploring predictive analytics of its own. In March 2016, the force's ethics committee met to consider how to proceed. Members advised that careful consideration had to be given to what data is used and the variables that are used in the process, concluding: "The use of the system must be treated with some caution and it must be ensured that there is no bias." The committee added that, if the work were to proceed, "the public must be informed as to why and how you are carrying out such processes."

Once the Think Family Database was complete, a police data scientist spearheaded the development of predictive risk models for the project. One aimed to identify children at risk of sexual exploitation. The CSE model, as it was known, drew on a wide range of datasets held by the police, council, and other public agencies. The child protection charity Barnardos provided anonymized data on 1,000 children known to have been sexually abused, and the model was built to detect children with similar characteristics. The system analyzed children's social connections to see whether they were linked to anyone deemed vulnerable to, or a likely perpetrator of, exploitation. Being flagged as "in need," "persistently" absent from school, or having mental health concerns would push up the scores the CSE model produced, documents say.

## Variables That Became Proxies for Poverty
The police's reliance on such a wide sweep of data raised early concerns. In 2018, researchers at Cardiff University's Data Justice Lab reviewed several UK citizen-scoring programs, including the Bristol work, noting, "The variables being used can in practice be proxies for poverty." Davies recalls that most of the children with the highest risk scores were already on the authority's radar. "They were complicated children already being worked with by social workers and family workers," he says. "Most of the output told you what you already knew."

Even so, the police's appetite for predictive analytics held firm. "We want to make choices today that will prevent the crime from happening in the first place," a police business intelligence manager said in 2018. A model to predict child criminal exploitation, introduced in 2019 according to Bristol City Council, again drew on data from a wide range of agencies, including whether a family was getting housing support or was in rent arrears and whether a child received free school meals. That year, chief constable Andy Marsh announced: "In 12 months every part of Avon and Somerset Constabulary will be driven through predictive analytics and visualization."

## A 'Messy' System Behind the Scenes
Behind the scenes, though, the work looked decidedly messy. Elle Pearson, a researcher at Royal Holloway University of London who is completing a PhD on the programs, says, "When I started, nobody could tell me what data they had or where it came from or what system was using which data." Pearson has interviewed more than a dozen staff from the police and local councils.

Over time, she observed a function creep, with systems growing more expansive, folding in more data, and drifting beyond their original purposes. While conscientious teams oversaw the development of data analytics, Pearson says, transparency was thin. "In some instances it might just be one person who's creating these risk models that are making decisions affecting potentially hundreds of thousands of people," she says.

By 2021, according to a government review of Insight Bristol, officials from the Centre for Data Ethics and Innovation, since dissolved, were hearing of ethical tensions tied to the project. Large amounts of sensitive data had gone into the risk scores, the reviewers noted, gathered through legal gateways rather than by building trust with local people. "Legality is not the same as legitimacy," they said.

## An Independent Review Finds the 'Weakest Element'
Two years later, the nonprofit Social Finance carried out an independent review of the Think Family Database and Insight Bristol's data work. The review, running to more than 100 pages and apparently made public only after a records request, was commissioned by Bristol City Council and the nearby Somerset Council, which was planning a similar system. It found that the Think Family Database and its visualizations had been useful for child protection staff and could lead to timelier responses.

But it called the risk-scoring models the project's weakest element, noting that a lack of accuracy had undermined their usefulness. Council staff had voiced doubts about the models built to assess the risk of child sexual exploitation (CSE) and child criminal exploitation (CCE). The review was completed around the same time the council stopped using the models, which staff had recently described as not fit for operational use.

Where the CSE and CCE models had largely been confirming what staff already knew, according to Gary Davies, the former head of Insight Bristol, those same social workers told the Social Finance reviewers they increasingly found the algorithms inaccurate. In an email about the CSE model, one staffer wrote: "There are people who've been victims of sexual offenses in the last month scoring below those who have been perpetrators of burglaries."

## Why the Quality Suddenly Collapsed
According to the review, there was a reason for the sudden drop in perceived quality: the police had stopped using Bristol City Council data. Officials wanted to profile children across the entire Avon and Somerset Police boundary, covering five separate councils, using the same algorithmic approach. The force tried to strike data-sharing agreements with other local authorities, but those efforts stalled, leaving it only its own data to build the algorithm. That meant perpetrators and victims of crime, but not the array of sensitive social factors the model had drawn on before.

After the switch, Bristol City Council staff said children who should have been identified as vulnerable were not listed in the results. "Personally, I feel uncomfortable using it to guide our work, because of the lack of transparency on where the numbers come from and how it was developed," one staff member told Social Finance.

Another said, "I wouldn't go into a meeting saying I've seen this on TFD, because I wouldn't be confident that that is accurate enough." One told the reviewers, "We know there's young girls that get criminally exploited, but they don't come up, we don't talk about them cause they don't fit." Another added, "I used to spend a lot of time methodically going through the 30 names, emailing people, and checking all the details, but it took up so much of my time that I kind of stopped doing that."

## Missing Records, Missing Accountability
When the Social Finance reviewers tried to run their own tests on the risk-scoring models, they found a startling absence of information. "Source code and variables that detail how these models were created was unable to be found, which prevented us completing this element of the evaluation," the report says. Likewise, responses to public records requests suggest neither the council nor the police kept any record of the decision to stop using the CSE and CCE models by June 2023.

Rob Procter, professor of social informatics at the University of Warwick, acted as an expert consultant on the review. "The process to build the models was not documented in anything like sufficient detail," he says. For him, the Bristol work shows the critical need for transparency and public debate whenever such an approach is even considered. "This really raises the question of how you involve the public in deciding to develop and deploy these kinds of tools, and addressing people's rightful concerns that this could lead to people being wrongly targeted," he says. "You have to consider the impact that even one false positive has on a family if a child is flagged as at risk of criminal or sexual exploitation."

Others share the worry. Debbie Watson, a professor of child and family welfare at the University of Bristol, has led a team researching the Think Family project since 2022. She says she is concerned about historic harms the risk-scoring models may have caused. "Whilst they may no longer be in operation, their use appears to have been significant in ways that have seriously impacted some young people in the city."

Bristol City Council declined interview requests about its use of predictive risk-scoring systems and did not answer detailed questions. "This administration does not use any predictive analytics apart from helping to identify children who are at risk of becoming not in education, employment, or training after finishing school (NEET)," councillor Christine Townsend, chair of the Children and Young People Policy Committee, said in a written statement. "The use of analytics has never replaced professional human judgment or decision-making."

Davies says any impact would have been minimal, because his staff never relied on the risk scores. "They weren't really using it to support their judgments, because they didn't understand it and they didn't value it," he says. But the absence of any record of how the models worked, or precisely why they were scrapped, makes it impossible to know for sure. Anyone affected would likely never learn why. As one council worker told the reviewers, "Something that's always on my mind, do the people we're talking about know we have this data?"

## Pegram's Story and the Question of Bias
When John Pegram received confirmation in 2024 that he was in the Offender Management App, he remembers thinking, "I've been here before, and I know what you're trying to do to me."

As a teenager, Pegram had grown used to police attention. He recalls being stopped dozens of times, something he attributes to being a mixed-race kid in a largely white town. Why he had ended up in the app was less clear to him. At a 2017 anti-fascist protest in Bristol, he was arrested for striking a police officer in the face. Although the officer conceded it appeared to be an accident, Pegram was convicted of assault. Seven years had passed, but did that incident mean he was still being flagged as a likely offender? He had little faith in the accuracy of any prediction made about him or anyone else. "There's a lot of bias in the police's data," he says. "There's too many issues for it to be done ethically and fairly."

## An Independent Audit's Damning Verdict
In response to public records requests, Avon and Somerset Police handed over a huge trove of performance data for 13 risk models used between 2017 and 2024, including ones meant to predict missing people, antisocial behavior, and who was most likely to commit or fall victim to crime. The data, along with other context about the force's data science program, was passed to the independent AI auditing firm Eticas for review. The verdict was damning.

"Most of these models produce low precision scores, meaning a high proportion of the individuals they flag as risks are incorrectly identified," the review found. A model used to help predict burglars appeared to run with a precision rating below 10 percent for more than three years, the police data showed. According to Eticas, that meant fewer than one in 10 flagged as high risk would actually offend. Other concerns included performance metrics for various models swinging sharply. "This is not typical of well-governed models in operational use," the audit observed.

A spokesperson for Avon and Somerset Police said the force chose not to deploy some of the models it built, including the burglary one. Asked why it held years of audit and performance data for models it never used, the spokesperson said the audit process was automated and drew on a static file that was not deleted when the decision was made not to deploy the model.

The force declined interview requests about its data science work and did not fully answer a detailed list of questions. "Each model is scored based on its performance, and where issues are identified, they will be updated or turned off," the spokesperson said in a statement, adding that models are reviewed by a police subject expert before deployment.

## An Incomplete Check on Bias
It is not clear what steps the force took to address the risks raised by its own ethics committee in the early days of the work. The committee did not appear to discuss predictive analytics again after 2017, according to records disclosures. And while the police website says every product and project pursued under its data science work is reviewed by a dedicated ethics group, the spokesperson said "there has so far been no meeting held," because "no model has been produced for which potential ethical issues have been identified."

In response to one records request, the force supplied a screenshot of a bias check app that appeared to monitor and compare average risk scores for white individuals and people of color, concluding there was "no significant difference between the two." The Eticas review responded: "Simply including ethnicity as a monitoring variable is not equivalent to testing whether the model produces discriminatory outcomes," calling the absence of more detailed testing by ethnicity, gender, and socioeconomic status "a significant omission."

## The Road Ahead, and the Rise of AI
Asked whether predictive analytics has any role in policing or social work, Davies says more work is needed. "When we were trying to do it, we were trying to do it for the right reasons, in the right way, but we didn't have the capacity that it probably needed." Part of that work, he says, should examine how risk models can inform workers without nudging them toward foregone conclusions. "There is a risk that staff see the computer say something and then don't use their own judgment."

Predictive analytics still plays a major role in policing and public services in the region. Bristol City Council continues to use a risk-scoring model to gauge the chance of a child falling out of education, employment, or training. Avon and Somerset Police's latest audit data, provided last July, indicates that the model behind the Offender Management App correctly predicts just one in three people who actually offend, while one in four people flagged as likely offenders do not.

Last year, the force told Pegram that, although he had a profile on the Offender Management App, he had not been given a risk score, because he had not been linked to any offense in the past two years. He still does not know what other data is held or how it might affect his dealings with the police. In July 2025, his lawyers wrote to the force again, notifying it of his intention to mount a legal challenge. The spokesperson declined to comment on Pegram's case or any proceedings, though they said the force is now identifying an independent party to review its models.

Pegram wants his details removed from the app, but he also wants Avon and Somerset Police to scrap the program entirely. "It's not just me," he says. "I don't think an AI model should have that kind of power over people's lives."

Yet the direction of travel seems clear. The UK government has just created PoliceAI, a body backed by £75 million that will help roll out a range of AI tools to 43 police forces across England and Wales. It sits within the College of Policing, led by Andy Marsh. Launching the project earlier this month, the UK's policing minister, Sarah Jones, said, "This is the future of policing, and it is happening now."

## What this means for you
- **For ordinary citizens:** The case shows how your most personal information (mental health, housing, school records) can be fed without consent into a scoring system, where a flawed risk score can wrongly mark an innocent person as a suspect.
- **In the UK:** AI tools are being rolled out rapidly across 43 police forces in England and Wales, so residents arguably have a right to know whether an algorithm has flagged them and how it shapes police treatment.

## Questions & Answers

### 1. What is the Think Family Database?
It is a database created in 2016 by Bristol City Council and Avon and Somerset Police that pooled sensitive information, from police intelligence reports to mental health records, to assign risk scores to adults and children.

### 2. Which models were shut down, and why?
At least two models, used to assess the risk of child sexual exploitation (CSE) and child criminal exploitation (CCE), were quietly abandoned after council staff found them inaccurate and not fit for operational use.

### 3. Who is John Pegram?
He leads a police accountability group in Bristol and learned in 2024 that he was on the Offender Management App, but the police would not tell him how his data was being used.

### 4. What did the independent audit find?
Eticas found most models produce low precision; a burglary-prediction model ran below 10 percent precision for more than three years, meaning fewer than one in 10 flagged as high risk would actually offend.

### 5. Are these systems still in use?
Yes. Bristol City Council still uses a model to gauge the risk of a child falling out of education, employment, or training (NEET), and the Offender Management App remains active.

### 6. What is PoliceAI?
It is a new UK government body backed by £75 million that will help roll out AI tools to 43 police forces across England and Wales, hosted by the College of Policing.

---
_TrendKia — Har trend, sabse pehle.. Machine-readable view; canonical HTML at the URL above._