National and state-level datasets of United States forensic DNA databases 2001-2025

This paper presents a comprehensive, harmonized series of national and state-level datasets (2001–2025) detailing the structure, composition, and policy context of United States forensic DNA databases to facilitate longitudinal and cross-sectional analyses of their historical development and growth.

The Gist

Imagine the United States has a massive, digital library. But instead of books, this library stores DNA profiles—genetic blueprints collected from crime scenes, convicted criminals, and, in many places, people who have been arrested but not yet convicted.

For the last 25 years, this library has been growing wildly. However, trying to study it has been like trying to read a book where the pages are scattered, some are written in different languages, and the librarian keeps changing the filing system every few years.

This paper is essentially a master key and a new map for that library. The researchers have spent years cleaning up the mess, organizing the scattered pages, and creating three new, easy-to-use datasets so anyone can finally understand how this system works, how big it is, and who is in it.

Here is a breakdown of what they did, using simple analogies:

1. The Problem: The "Shifting Sand" Library

Think of the FBI's national DNA database (called NDIS) as a giant scoreboard that updates every month.

• The Issue: In the past, if you wanted to know how many profiles were in the database in 2010, you'd have to find a specific webpage from that time. But the FBI kept redesigning their website. Sometimes the data was on one page, sometimes split across fifty different pages. If you didn't save the screenshot exactly when you needed it, that data was gone forever.
• The State Level: It was even worse at the state level. Some states published their numbers; others didn't. Some listed the types of people in the database (by race or gender); most didn't. It was like trying to count the apples in 50 different orchards, but only 10 orchards had signs, and the signs were written in different fonts.

2. The Solution: The "Time-Traveling Librarian"

The researchers acted like digital archaeologists. They didn't just look at the current website; they went back in time using the Internet Archive (Wayback Machine).

• Dataset 1: The National Time Machine (NDIS)
  They downloaded 11,359 snapshots of the FBI's website from July 2001 to August 2025. Imagine taking a photo of a scoreboard every single day for 24 years. They then wrote computer programs (parsers) to read these photos, even though the scoreboard looked different in 2005 than it did in 2020.

  • The Result: A perfect, continuous timeline showing exactly how many profiles were added every month, broken down by state and type (criminal, arrestee, or crime scene).

• Dataset 2: The State Policy Map (SDIS)
  They went on a "treasure hunt" across all 50 states to find current rules.

  • The Result: A cheat sheet that tells you: "Does this state collect DNA from people just arrested?" "Can they use DNA to find family members of suspects?" (Familial searching). It maps out the rules of the game for every single state.

• Dataset 3: The Demographic Decoder (FOIA)
  This was the hardest part. For years, no one knew the racial or gender makeup of the database because states rarely reported it.

  • The Result: The researchers found old letters from 2018 where seven states finally answered a Freedom of Information Act (FOIA) request. They digitized these handwritten or scanned reports into a clean, computer-readable format. It's the first time this specific demographic data has been organized for easy study.

3. Cleaning the Data: The "Spot the Fake" Game

When you scrape data from the internet over 24 years, you get glitches.

• The Glitch: Sometimes the FBI website would show "10,000" profiles, and the next month show "100,000" (a mistake), then go back to "10,000." Or, the website might get stuck and show the same number for three months in a row because it was "cached" (stuck in memory).
• The Fix: The researchers built a "truth detector." They created a set of rules to flag these weird jumps.
  • Analogy: Imagine a teacher grading a test. If a student's score jumps from 50 to 500 in one second, the teacher flags it as a "decimal error." If the score stays exactly the same for a year when it should be changing, the teacher flags it as "stale data." They didn't delete the weird numbers; they just put a "warning sticker" on them so researchers know to be careful.

4. Why Does This Matter?

Before this paper, studying the US DNA database was like trying to drive a car with a cracked windshield and a missing map. You could guess where you were going, but you couldn't be sure.

Now, researchers, journalists, and policymakers have:

1. A Clear History: They can see exactly how the database grew over time.
2. A Policy Guide: They can compare how different states treat arrestees or family searches.
3. Demographic Insight: They can start to analyze if the database disproportionately affects certain racial or gender groups (based on the limited data available).

In a nutshell: This paper is the "User Manual" for the US Forensic DNA Database. It takes a chaotic, 25-year history of scattered web pages and turns it into a clean, organized, and trustworthy toolkit for understanding one of the most powerful tools in modern criminal justice.

Technical Summary

Here is a detailed technical summary of the paper "National and State-Level Datasets of United States Forensic DNA Databases 2001–2025".

1. Problem Statement

Despite the substantial expansion of forensic DNA databases in the United States over the last two decades, there is a critical lack of comprehensive, harmonized, and longitudinal data regarding their structure and composition.

• Data Fragmentation: Federal statistics (National DNA Index System - NDIS) are published by the FBI as individual "snapshots" on their website rather than in a continuous, machine-readable time series. This limits the ability to track changes over time.
• State-Level Inconsistency: State-level data (State DNA Index Systems - SDIS) is highly inconsistent. Some states publish detailed breakdowns, while others provide only general summaries or no data at all.
• Demographic Gaps: Data regarding the racial and sex composition of database profiles is sparsely documented. Previous attempts (e.g., Murphy & Tong, 2020) relied on Freedom of Information Act (FOIA) requests that yielded responses from only 7 out of 50 states.
• Reporting Artifacts: Historical web archives contain formatting inconsistencies, naming errors, and potential data anomalies (e.g., stale cache data, decimal errors) that hinder direct analysis.

2. Methodology

The authors developed a three-pronged data collection and processing strategy to reconstruct and standardize forensic DNA data from 2001 to 2025.

A. Federal Statistics Reconstruction (NDIS Time Series)

• Source: Archived FBI webpages via the Internet Archive's Wayback Machine API.
• Search Strategy: A two-phase algorithm queried over 50 jurisdiction-specific URLs (pre-2007) and unified page patterns (post-2007) to capture 11,359 unique snapshots spanning 255 URL patterns.
• Parsing: Four era-specific parsers were developed to handle evolving FBI reporting formats:
  • Pre-2007: Extracted data from individual state HTML files.
  • 2008–2011: Processed consolidated pages with "Back to top" dividers.
  • 2012–2016: Captured the introduction of arrestee profile reporting.
  • Post-2017: Used regex patterns for modern standardized formats.
• Standardization: A mapping table was created to normalize inconsistent jurisdiction names (e.g., "Alabama" vs. "Alabama Stats Alabama") and include non-state entities (e.g., DC/FBI Lab, U.S. Army).

B. State-Level Policy and Counts (SDIS Cross-Section)

• Collection: Targeted web searches of official state government websites using keywords like "SDIS," "CODIS," and "DNA Database."
• Coding:
  • Counts: Aggregated reported totals for arrestees, offenders, and forensic profiles.
  • Policies: Binary coding for arrestee collection laws (Yes/No) and categorical coding for familial search practices (Permitted/Prohibited/Unspecified).
• Scope: Covers all 50 states, representing the best available recent snapshot (2013–2025).

C. Demographic Data Standardization (FOIA)

• Source: Original FOIA response letters and supplementary data from Murphy & Tong (2020), provided directly by the authors.
• Processing: Transcribed scanned PDFs into machine-readable CSVs.
• Harmonization: Standardized terminology (e.g., using "sex" instead of "gender" due to biological relevance in DNA profiling) and preserved both raw agency-reported counts and the authors' derived estimates.
• Limitation: Demographic data is limited to seven states that responded to the 2018 FOIA requests.

D. Technical Validation and Anomaly Detection

To ensure data integrity, the authors implemented a sequential flagging pipeline for the NDIS time series:

1. Decimal Spike-Dip: Flags values >8x or <0.125x the baseline (indicating decimal entry errors).
2. Spike-Dip: Flags sudden doublings or halvings (>2x or <0.5x baseline).
3. NDIS Labs Spike: Specific lower thresholds for small integer lab counts (>3x or <0.33x baseline).
4. Stale Exact Reappearance: Flags when an exact earlier value reappears after higher values (indicating cached data).

• Outcome: 898 anomalies were flagged across the dataset. Instead of deleting data, boolean flags were added to allow users to filter based on their own criteria.
• External Calibration: Reconstructed totals were cross-referenced with FBI brochures (2000–2015) and academic literature estimates (2012–2024), showing high fidelity.

3. Key Contributions

The paper releases three distinct, versioned datasets available on Zenodo and GitHub:

1. NDIS Time Series (2001–2025): A monthly reconstructed dataset including offender, arrestee, and forensic profile counts, participating laboratory totals, and investigations aided, with anomaly flags.
2. SDIS Cross-Section: A current snapshot of all 50 states including profile breakdowns, arrestee collection policies, and familial search authorization status.
3. FOIA Demographics & Collection Rates: Structured data from seven states (2018) regarding sex/racial composition, plus standardized annual collection rate estimates for all 50 states derived from Murphy & Tong's work.

Supporting Infrastructure:

• Reproducibility: All parsing scripts, Quarto notebooks, and workflow documentation are open-source on GitHub.
• Documentation: A comprehensive data dictionary and README file explain organizational terminology and processing stages.

4. Results

• Data Volume: The NDIS dataset captures 11,359 unique web snapshots, providing a continuous monthly record from July 2001 to August 2025.
• Anomaly Distribution: The majority of flagged errors (743/898) were "stale exact reappearances" (cached data), followed by spike/dip events (77) and decimal errors (68).
• Validation: The reconstructed time series aligns closely with independent reference points, including the 2015 FBI CODIS brochure and estimates from Ge et al., Wickenheiser, and Greenwald & Phiri.
• Policy Landscape: The state-level dataset reveals significant variation, with only 29 of 50 states reporting raw arrestee DNA totals in recent years.

5. Significance

This work provides the first foundational, machine-readable infrastructure for studying the historical development of U.S. forensic DNA systems.

• Longitudinal Analysis: Enables researchers to model database growth, policy impacts (e.g., the 2017 expansion of CODIS core STR loci), and reporting practices over 25 years.
• Demographic Equity: Facilitates the study of racial and sex disparities in DNA database composition, a critical area for criminal justice research.
• Transparency: By preserving raw data and flagging anomalies rather than discarding them, the dataset allows for transparent, reproducible, and customizable analysis by the research community.
• Policy Evaluation: The compiled policy metadata allows for cross-jurisdictional comparisons of arrestee collection laws and familial search practices.