New Risk Assessment Tool for Criminal Justice Seeks to Predict Risk of Harm to Defendants, Reduce Sentencing Disparities

The criminal justice system uses algorithmic risk assessment instruments to predict the risk a defendant poses to society—for example, the risk of recidivating or failing to appear at future court dates. However, defendants are also at risk of harm from the criminal justice system. In a new study, researchers developed a risk assessment instrument that considers the risk the system poses to individuals. In doing so, they created a tool that predicts defendants’ risk of receiving an especially lengthy sentence based on factors that should be legally irrelevant in determining the length of sentences, such as race and gender.

The study was conducted by researchers at Carnegie Mellon University (CMU), the University of Pennsylvania, the American Civil Liberties Union, and the Idaho Justice Project. It appears in a forthcoming paper that will be presented in June 2022 at FAccT, a conference on fairness, accountability, and transparency organized by the Association for Computing Machinery.

“The intended purpose of currently used risk assessment instruments is to estimate the risk posed by the defendant,” says Mikaela Meyer, PhD candidate in statistics and public policy at CMU’s Heinz College, who coauthored the study. “Our instrument flips the script, offering a model that can be used by defense attorneys to defend their clients, warn individuals likely to be treated unjustly by the system, notify judges about the risk to ensure that cases receive appropriate review, and inform post-sentencing motions for compassionate release.”

Researchers built their risk assessment tool using a dataset of more than 57,000 U.S. District Court sentences from 2016 to 2017. They considered the process by which individuals are sentenced, as well as the laws and policies governing sentencing decisions, including U.S. Sentencing Guidelines.

The study considered legally relevant factors in sentencing, including defendants’ criminal history, offenses, and the statutory sentence lengths determined by the nature of the charge. It also considered legally irrelevant factors (i.e., those not pertinent to the charges for which individuals were sentenced and that should not have been considered in sentencing decisions), including defendants’ socio-demographic information (e.g., race, ethnicity, U.S. citizenship status); judges’ socio-demographic information (e.g., race, gender, political party of appointing president); and information about the location, timing, and status of the case (e.g., whether there was a plea agreement, when and where the sentence was handed down, whether certain administrative documents were obtained).

The instrument achieved comparable predictive accuracy to risk assessment instruments used in pretrial and parole contexts, the authors say. It can be used by defense attorneys, defendants, and others to advocate for reducing over-sentencing in a federal context or appeal for clemency.

Among the limitations of the study, the authors note that policy decisions and limits in the processes and management of the criminal justice system affect which data are available. For example, the study did not consider the presence of plea bargaining. In addition, the data used may exclude data that were available to decision makers (such as judges). Finally, the individuals in the dataset they used are not necessarily representative of all sentenced people. The authors note that traditional risk assessment instruments suffer from many of the same limitations.

“Our goal was for the tool we developed to predict disproportionately harsh sentences prior to sentencing so such sentences could be avoided, and to reduce disparities in the system,” explains Meyer.

###

Summarized from an article to be presented at FAccT, Flipping the Script on Criminal Justice Risk Assessment: An Actuarial Model for Assessing the Risk the Federal Sentencing System Poses to Defendants by  Meyer, M (Carnegie Mellon University), Horowitz, A (American Civil Liberties Union), Marshall, E (Idaho Justice Project), and Lum, K (formerly at the University of Pennsylvania). Copyright 2022 Association for Computing Machinery. All rights reserved. 

About Heinz College of Information Systems and Public Policy
The Heinz College of Information Systems and Public Policy is home to two internationally recognized graduate-level institutions at Carnegie Mellon University: the School of Information Systems and Management and the School of Public Policy and Management. This unique colocation combined with its expertise in analytics set Heinz College apart in the areas of cybersecurity, health care, the future of work, smart cities, and arts & entertainment. In 2016, INFORMS named Heinz College the #1 academic program for Analytics Education. For more information, please visit www.heinz.cmu.edu.