Tools to Assess Crime Risk for Young Cohorts Are Likely to Fail Over Time if They Ignore Social Change
The study, by researchers at Carnegie Mellon University (CMU), Harvard, and University of Pennsylvania, appears in PNAS. The researchers analyzed criminal histories of individuals in Chicago over a 25 year period, 1995-2020. They found that regardless of predictors, a “machine learning” tool predicting the likelihood of arrest between ages 17 and 24 for older cohorts born in the 1980s substantially overpredicted the likelihood of arrest for younger cohorts born in the mid-1990s. This suggests that RAIs are likely to fail over time for more recent cohorts if social change dynamics are ignored.
“Our aim is to improve the science and use of RAIs in high-stakes contexts,” says Daniel Nagin, professor of public policy and statistics at CMU’s Heinz College who goes on to point out that “Cohort bias can generate inequality in criminal justice that is distinct from racial bias.”
In predicting criminal behavior and justice system involvement, RAIs typically predict risk using a combination of features that measure individual characteristics, family background, and prior criminal history.
Past research has identified early-life, psychosocial, and neighborhood predictors of criminal involvement in adolescence and beyond. These include family instability and poverty, low self-control and growing up in poor neighborhoods. While the association between many of these features and later criminal involvement is well established, whether the predictive strength of these features is constant across historical periods is uncertain.
In this study, researchers examined whether an RAI trained on individual-level features of an older age cohort accurately predicted the likelihood of criminal justice involvement of a younger cohort. Specifically, they studied the performance of models trained to predict arrests in early adulthood across multiple birth cohorts born between 1979 and 1995.
Their data came from the longitudinal Project on Human Development in Chicago Neighborhoods and included more than 1,000 individuals from four age cohorts. Researchers also examined criminal records from the state of Illinois between 1995 and 2020. Among the study’s findings:
- Cohort bias was significant: RAIs trained on an older cohort overpredicted the probability of arrest of the younger cohort by up to 89%.
- Cohort bias was substantial within racial-ethnic groups (Whites and others, Latinos, and Blacks): This establishes cohort bias as an underappreciated mechanism generating inequality in criminal justice that is distinct from racial bias.
- Cohort bias persisted: It lasted even when measures of arrest from immediately before the ages for which the study predicted arrest were included as predictors and even when analysis was limited to high-risk participants.
In the context of high-stakes decisions, RAIs have the potential to do harm if not well calibrated. While human judgment is also prone to biases, an RAI can make millions more judgments, amplifying its impact.
“Social science research and policy increasingly rely on predictive RAIs, including those that use machine learning algorithms,” explains Erika Montana, a Ph.D. student in machine learning and public policy at CMU’s Heinz College, who coauthored the study and took the lead role in conducting the analyses. “Our findings show that the relations between risk factors and future arrest are not stable over time. As a result, prediction models that rely on these risk factors are prone to systematic and substantial error.”
The study was funded by the Office of Juvenile Justice and Delinquency Prevention and managed by the National Institute of Justice, Office of Justice Programs, U.S. Department of Justice.
About Heinz College of Information Systems and Public Policy
The Heinz College of Information Systems and Public Policy is home to two internationally recognized graduate-level institutions at Carnegie Mellon University: the School of Information Systems and Management and the School of Public Policy and Management. This unique colocation combined with its expertise in analytics set Heinz College apart in the areas of cybersecurity, health care, the future of work, smart cities, and arts & entertainment. In 2016, INFORMS named Heinz College the #1 academic program for Analytics Education. For more information, please visit www.heinz.cmu.edu.