When an Algorithm Sets the Sentence — Is AI Recidivism Prediction Fair?

When making decisions in a courtroom where someone's freedom is at stake, people have long trusted human judges. However, humans are vulnerable to bias. Suspicions that outcomes vary even for the same case have constantly followed. The judge's mood, the defendant's skin color, and the condition of that day are said to be variables. Thus, one idea emerged. What if we have an objective and consistent algorithm calculate the risk of recidivism? It was the idea that human caprice and bias could be stripped away.

American courts actually did so. COMPAS is software that scores a defendant's risk of committing a crime again. This tool has been used as auxiliary material for pretrial release (bail), sentencing to determine the penalty, and probation decisions. The fact that recidivism risk assessment tools are increasing in helping such judgments is pointed out by a cited study (DOI: 10.36645/mtlr.30.2.fairness). The expectation was that since it is based on data, it would be fairer.

This article traces how that expectation was shaken. To give the conclusion first, the phrase "fair algorithm" itself was a much trickier promise than thought. First, I clarify which law this article is written about. Almost all the evidence cited is legal and machine learning academic debates centered on the U.S. criminal justice system. Therefore, it should be read as "this is how it was debated mainly in the U.S.," not "it is like that in any country." This is not a story about Korean law.

Common belief — Algorithms will be fairer than human judges

The logic for introducing algorithms was simple. Human judges can be inconsistent and biased. The expectation was that tools trained on data would reduce that caprice and bias. The trend of bringing predictive analytics into the criminal justice system spread with a single promise. It is the promise to "correct human biased or capricious judgment to improve outcome equality and fairness" (DOI: 10.24908/ss.v17i3/4.10410).

The predictive power of the tools was also presented as evidence. In one validation study (2,328 cases), the discriminative performance metric of COMPAS (AUC, where 0.5 is chance level and 1 is perfect) was .66 to .80 depending on the subgroup and outcome criterion. It was on par with or better than tools of a similar generation (DOI: 10.1177/0093854808326545). The common belief that "data is better than people" looked plausible.

Exposure — Racial Bias in COMPAS Pointed Out by ProPublica

However, as validations accumulated, a different picture emerged. The starting point was an analysis by the investigative journalism outlet ProPublica. This outlet reported that COMPAS's predictions were racially biased against Black defendants.

There is a student abstract that re-analyzed this using the Framework for Causal Inference (FACT) (DOI: 10.1609/aaai.v34i10.7192). This abstract reports that it presented strong evidence that COMPAS exhibits racial bias against African American defendants. It states that the estimate is robust even if there are unmeasured confounding factors.

The core of the problem was how errors are distributed. There is an SSRN working paper that examined COMPAS data with a sensitivity analysis (DOI: 10.2139/ssrn.6168631). It states that criminal history characteristics were structurally amplified, and the variable of age showed an asymmetry that differed greatly by racial group. The authors viewed this as practically equivalent to different judgment rules operating for each group. They conclude that this violates the principle of treating people under the same conditions equally.

There is yet another study that added to the weight of the criticism. Previous research reported that commercial recidivism risk assessment software is no better than the judgments of non-experts gathered at random. There is work that reproduced this result. When several machine learning models were used and even bias-mitigation techniques were applied, the accuracy improved slightly. The authors summarize that while bias-mitigation techniques help up to a limit, biases embedded in the data can remain inside AI risk assessment tools (DOI: 10.1007/s43681-026-01116-0). This reads as a signal that there is something that cannot be solved simply by refining the tool.

Why Bias Does Not Disappear — The Trap of Data

Here a deeper problem is revealed. Algorithms mirror the data they were trained on. However, there has long been a concern that arrest records, the starting point of criminal justice data, are a biased proxy for crime.

Most research auditing algorithmic bias effectively uses arrest data as an unbiased measure of crime. That is a practice that clashes with that concern. A review examining the racial disparities of arrests as a measure of crime pointed this out (DOI: 10.1146/annurev-criminol-022422-125019). As long as arrest data is the core of the tool, the racial differences in how arrests proxy crime could be the core pathway biasing the tool. However, this review adds that almost no studies have measured how much biased arrest records create algorithmic bias.

The same trap is seen in predictive policing, the method of using data to predict the next crime location and deploying police. A systematic review of predictive policing fairness found that one of the main drivers of model discrimination was police activity itself (DOI: 10.1007/s43681-024-00541-3). One policy brief calls this a "structural" limitation. Policing data reflects where police patrol, not where crime happens, and tools inherit bias from there (DOI: 10.2139/ssrn.6713159). According to this argument, we must examine not only the algorithm but also the data the algorithm consumed.

When Fairness Standards Clash with Each Other—Fairness Impossibility

Up to this point, one might retort, "Can't we just fix the data?" Yet at the deepest layer of this story lies a mathematical fact that cannot be solved no matter how much the data is tweaked. It is fairness impossibility.

Intuitively, we tend to think that a fair tool must satisfy two things simultaneously. One is calibration of risk scores. This means that people who receive the same score must have the same actual recidivism rate, regardless of race. The other is error rate balance. This means that the rate of wrongly classifying someone who will not reoffend as risky (the false positive rate) and the rate of missing someone who will reoffend (the false negative rate) must be equal across groups. The COMPAS debate split into three paths depending on which standard is used to measure bias (DOI: 10.36645/mtlr.30.2.fairness).

The problem is that these standards cannot be held simultaneously when the recidivism rates differ between groups. The fairness standards proposed by scholars conflict with one another and are mutually exclusive. It is mathematically impossible for a single system to satisfy those standards at the same time. A study formalizing the fairness of recidivism prediction tools summarized it this way (DOI: 10.36645/mtlr.30.2.fairness).

There is also a more general proof. The base recidivism rate is the proportion of people who actually reoffend in each group. If the base recidivism rates differ by group, no classifier can satisfy three criteria simultaneously (barring degenerate cases). The three criteria are statistical parity, equalized odds, and predictive parity. Statistical parity means that the proportion of high scores is the same across groups. Equalized odds means that the two error rates are equal, and predictive parity means that the recidivism rate for the same score is equal. A preprint analyzing the limits of fairness and accuracy presented this impossibility theorem (DOI: 10.21203/rs.3.rs-8598316/v1).

The meaning this fact holds in criminal justice is heavy. The theorem in a policy brief goes like this. If the baseline rates of recidivism differ by group, no algorithm can satisfy all fairness metrics simultaneously. Achieving predictive parity makes the error rates unequal across groups, and equalizing the error rates breaks predictive parity. Therefore, this brief views the choice between metrics not as a technical optimization, but as a value judgment (DOI: 10.2139/ssrn.6713159). The formalization study we saw earlier also views that we ultimately have to choose which statistical measure to adopt as the standard of fairness. This study suggests false positive parity as the best standard (DOI: 10.36645/mtlr.30.2.fairness).

Some philosophical analyses go further to ask whether "algorithmic fairness" is an inconsistent concept. It is the hypothesis that it is a concept that can apply and simultaneously not apply to the same situation (DOI: 10.5406/21521123.62.1.04). If this hypothesis is correct, "fairness," which is called by a single word, may be a bundle of several mutually incompatible demands. Rather than confirming this, the author puts it forward as an option to be scrutinized further.

The Procedural Issue — Can It Be Explained, Can It Be Refuted?

If fairness is a matter of choosing values, the procedural question of who justifies that choice and how follows. Here, issues unique to criminal justice emerge.

First is transparency and due process. Due process is the principle that a person receiving a penalty must know the basis for it and have the opportunity to challenge it. There is the case State v. Loomis, in which COMPAS was used in sentencing. One paper views this case as an example of risk, arguing that handing public power over to an AI system without any checks can threaten human rights and the rule of law (DOI: 10.47348/sacj/v34/i1a2). Meanwhile, the Wisconsin Supreme Court ruled in this 2016 case that the sentencing court merely referencing the COMPAS score itself does not violate due process. However, it attached the condition that the score cannot be used as the decisive basis for determining whether to detain or the sentence length, and that the limitations of the score must be provided in writing.

A study analyzing automated recidivism risk assessment systems pointed out a similar problem. It views the explainability and non-discrimination of decisions made by programs like COMPAS as core challenges of due process. This study also proposes permissible ways of using them that meet due process requirements (DOI: 10.18572/2072-4152-2024-12-16-20). According to this critique, an "opportunity to challenge" remains merely formal to a defendant who cannot see the basis for the judgment.

Second is constitutional legitimacy. There is a paper analyzing COMPAS through the framework of the U.S. Constitution and international human rights. The author bases their argument on studies showing that Black defendants receive higher risk scores than white defendants with similar criminal histories. They further argue that current application is inadequate in light of U.S. due process and equal protection (a constitutional principle guaranteeing equality before the law) jurisprudence. However, they suggest that strong regulation like the EU AI Act is a more realistic path than the complete abolition of the tool (DOI: 10.2139/ssrn.6387640).

Third is human dignity and moral status. According to one paper, sentencing laws in many legal systems require judges to make normative judgments about a person and their actions in addition to "how dangerous they are." The argument is that replacing this with simple empirical prediction is not only a category mistake, but also risks failing to recognize the defendant as a moral agent (DOI: 10.2139/ssrn.6536358). There is also research dealing with "individualized justice." Judging each case by its own circumstances fundamentally conflicts with algorithms that reason by predefined rules. Therefore, the thesis is that human intervention and supervision are necessary to protect that value (DOI: 10.31228/osf.io/kz4s2).

Does Dropping the Algorithm Solve It — A Comparison with Human Judgment

So should we just scrap algorithms and go back to human judges? The story is not that neat. Research directly addressing fairness, accountability, and transparency (FAT) issues points out that those concerns are serious but not unsolvable. More importantly, it argues that the exact same problems apply even more urgently to traditional human decision-making. So the question is ultimately comparative. It is whether algorithms can improve the current state of criminal justice. The authors state there is no definitive answer, but there is room for cautious optimism (DOI: 10.1017/s1744552319000077).

Empirical evidence is also mixed on both sides. First is a pre-trial bail study. Simulating a policy to release in principle defendants whom the U.S. federal Pretrial Risk Assessment (PTRA) tool deems low risk, the effect was estimated through causal analysis. Using data from roughly 146,000 U.S. federal defendants, it was compared with judges' current decisions. This policy was estimated to reduce detention by 34.2% and increase public safety risk by 1.6%. It also reported that Black defendants benefit more than white defendants. The authors view that this tool should act to strongly guide rather than replace judges' discretion (DOI: 10.1111/1745-9133.70015).

There is also research advocating for pre-trial risk assessment tools. According to this study's synthesis, even with differences between racial and ethnic groups, the tool predicts recidivism well. If those results are actually reflected in decision-making, it can be an effective strategy to reduce detention for both people of color and white people (DOI: 10.1177/00938548211041651). In a predictive bias test of a tool other than COMPAS (the PSA), the authors report that valid and consistent predictions emerged across six race-gender groups (DOI: 10.1177/00938548241233934).

Academia is also exploring ways to bypass the impossibility. One study argues that formal fairness falls into impossibility because it traps analysis within isolated decision-making procedures. The solution is to broaden the analytical scope to substantive fairness, which leans on theories of substantive equality from law and philosophy. The proposal is that this can escape that trap (DOI: 10.1007/s13347-022-00584-6). Another study attempted counterfactual adjustments—treating disadvantaged groups like advantaged groups—using optimal transport techniques. In a sample of 300,000 people in a large city, it was shown that the fairness of legally protected groups could be significantly increased. This improvement is said to be compatible with Pareto improvement. Pareto improvement is an improvement achieved without making one side worse off. This result is within that sample and the fairness metric defined by the study (DOI: 10.1177/00491241231155883).

Remaining Questions — Which Fairness to Choose

The COMPAS debate shows that a "fair algorithm" is not an engineering problem with a single correct answer. Conventional wisdom assumed that algorithms would strip away human bias. However, ProPublica pointed out the racial disparity in false positives. There is also the impossibility theorem of fairness. It showed the mathematical limit that when recidivism rates differ by group, attempts to completely eliminate that disparity break other conceptions of fairness.

Which fairness to choose is a value judgment. To justify that judgment, one must clear the procedural thresholds of transparency, due process, dignity, and responsibility. There is a counterargument that human judgment suffers from the same problems, so its superiority or inferiority compared to human judgment remains controversial.

Therefore, what this article leaves behind is acknowledging trade-offs. The question of whether to let algorithms determine sentences starts from two questions. Which fairness is valued more? Who can take responsibility for and explain that choice?

This article is for informational purposes, and many of the cited sources are legal and machine learning academic analyses from specific jurisdictions such as the United States. It is not legal advice, and conclusions may vary depending on the jurisdiction.

Source Papers

COMPAS and ProPublica Exposé

• COMPAS recidivism risk dataset reproduction, bias, and fairness — DOI: 10.1007/s43681-026-01116-0
• COMPAS racial bias viewed through causal inference (FACT) — DOI: 10.1609/aaai.v34i10.7192
• COMPAS audit: Structural amplification and asymmetry of risk scores — DOI: 10.2139/ssrn.6168631
• COMPAS predictive validity validation (AUC .66–.80) — DOI: 10.1177/0093854808326545
• COMPAS debate: Fairness, transparency, and data (German-speaking world) — DOI: 10.1007/s11757-022-00741-9
• COMPAS and the erosion of human decision-making authority — DOI: 10.22197/rbdpp.v12i1.1287

The Trap Called Data

• The problem of arrests as a biased proxy for crime — DOI: 10.1146/annurev-criminol-022422-125019
• Fairness in predictive policing: How police activity causes discrimination — DOI: 10.1007/s43681-024-00541-3
• Predictive policing: The paradox of being both the cause and solution of discrimination — DOI: 10.24908/ss.v17i3/4.10410

Impossibility of Fairness

• Formalizing fairness in recidivism prediction tools (mutually exclusive criteria) — DOI: 10.36645/mtlr.30.2.fairness
• The fairness-accuracy impossibility theorem — DOI: 10.21203/rs.3.rs-8598316/v1
• Structural and technical limitations of pre-trial tools and value judgments — DOI: 10.2139/ssrn.6713159
• Is algorithmic fairness an inconsistent concept? — DOI: 10.5406/21521123.62.1.04

Procedural Issues

• The Loomis case and the dangers of algorithmic sentencing — DOI: 10.47348/sacj/v34/i1a2
• Explainability and non-discrimination challenges in automated recidivism assessment — DOI: 10.18572/2072-4152-2024-12-16-20
• Constitutional and human rights challenges of COMPAS and the strong regulation argument — DOI: 10.2139/ssrn.6387640
• Algorithmic sentencing and human dignity — DOI: 10.2139/ssrn.6536358
• Individual definition and the necessity of human judgment — DOI: 10.31228/osf.io/kz4s2

The Honest Current State

• The comparative question of FAT: Can it improve upon the human status quo? — DOI: 10.1017/s1744552319000077
• The effects of risk-based pre-trial release policies (causal analysis) — DOI: 10.1111/1745-9133.70015
• Empirical evidence defending pre-trial risk assessment tools — DOI: 10.1177/00938548211041651
• Predictive validity of the PSA across race-gender groups — DOI: 10.1177/00938548241233934
• From formal fairness to substantive fairness — DOI: 10.1007/s13347-022-00584-6
• Improving fairness based on optimal transport (Pareto improvement) — DOI: 10.1177/00491241231155883