The Grader Is Also a Language Model — How Far Can Its Verdict Be Trusted?

Asking Fifty Times

We placed two answers side by side and asked, "Which one is better?" This was done fifty times with the same question, the same grader, and the same settings. The grader was a language model, meaning a model that takes text and produces the text that follows.

On average, the winner flipped in 13.6% of the attempts. For 28% of the questions, the flip rate exceeded 20%, and for one question, it reached 56%. This is why the paper is titled "A Coin-Flip Judge?".

One of the two tested graders chose the answer presented first in the majority vote for a given question 72% of the time. When the prompt was changed to one with the same meaning but different wording, the majority vote result changed in 25% of the tested cases. The judgments of two different graders overlapped only 76% of the time. Turning off random sampling merely reduced the wobble; it did not eliminate it.

To recover a baseline judgment obtained from asking fifty times with a 95% probability, this study's data required asking an average of eleven times and deciding by majority vote. The authors note that a single-attempt grade is often too unstable to use for important evaluations. However, the authors explicitly note that both graders belonged to the same company, leaving replication with models from other companies for the future (arXiv:2606.13685, preprint).

A model judge is a language model assigned to grade work. It is the term used interchangeably in "The Score Went Up. What Went Up?", also called an LLM judge after its English name. Throughout this article, "judge" always refers to a model judge, and "grader" refers collectively to both humans and models. A judge chooses the better of two answers or assigns a score to a single answer. Those judgments are used to rank model answers, build scoreboards used when training models, and fill public leaderboards (arXiv:2606.13685). In safety evaluations measuring how well models withstand attacks, a judge also sits in the seat determining whether an answer is harmful (arXiv:2603.06594).

Studies that have model judges do the grading also appear in "Does Retrieval Cure Hallucination? — What RAG Fixes, and What It Does Not" and "Multiplication Won't Tell You — Where Agents Break on Long Tasks". This article questions that grading itself.

Instead of picking a good judge model, this article looks at what shakes a judge and which direction it leans. Many of the cited studies are preprints. A preprint is a paper made public before undergoing review by other researchers. Papers that have passed review and been published in academic journals are marked as "peer-reviewed".

Even High Apparent Agreement Can Be a Fluke

In the Turkish story dataset, a rule-based program automatically marked whether six creative techniques were used in each scene, and the researchers asked humans whether they agreed with those marks. The sticking point was a technique that turns emotions into visible objects. Out of 100 scenes, the number of scenes where five machine graders (one rule-based program and language models) marked this technique as used was 0, 1, 40, 72, and 78. Humans looking at the same 100 scenes answered that 9 did.

Yet, across the two scene bundles and two human standards in this study, the raw rate at which machine and human judgments overlapped fell between 74.7% and 84.5%. This apparent agreement is mixed with shares that were correct purely by chance.

Kappa is the degree of agreement calculated by taking the rate at which two graders' judgments overlapped and subtracting the share that would have been correct by chance even if someone just guessed randomly (DOI: 10.2139/ssrn.7217178, preprint). At the level of chance, it is near 0, and with complete agreement, it is 1. Sticking to the principle alone, out of 100 scenes, when humans marked only 9 as applicable, if a machine grader simply closed its eyes and picked "not used" every time, the two judgments would still overlap by more than 90%. Once that share was subtracted, across both the scene bundles and the two human standards, the kappa for five of the six machine graders was 0.004, 0.015, 0.000, 0.019, and 0.027. These values are indistinguishable from chance. The authors do not view high apparent agreement as a sign of capability, but rather as a product of one side's answers being overwhelmingly frequent.

If the 76% from the previous section is also corrected using kappa, it comes out to 0.51, which is better than chance but still roughly half (arXiv:2606.13685). The point that simple agreement rates inflate discriminative power is also made in "The Score Went Up. But What Went Up?".

The authors of the Turkish study do not synthesize these results into a single narrative. Two interpretations remain consistent with the data. One is that the technique has a quality that can only be caught by reading between the lines, meaning current automated grading cannot reach it; the other is that the definition of the technique is not yet precise enough for anyone to apply it consistently. If the latter interpretation is correct, even human graders failed to catch it properly. Distinguishing between the two requires a second, independent human grader, which this study lacked, and thus the authors claim neither.

Human graders do not agree completely with each other, either. In a systematic review of 66 clinical language model studies published between 2024 and 2025, a significant number did not report inter-rater agreement among humans at all. Even in the studies that did report it, kappa commonly fell between 0.40 and 0.80. The authors therefore write that many models reported to have reached "expert level" may simply be matching the noise mixed into expert criteria. The conclusion is that current evaluation standards cannot back up claims that models have surpassed humans (DOI: 10.21203/rs.3.rs-10047888/v1, preprint, review). The authors propose making the reporting of such agreement mandatory in clinical AI research, arguing that unless human judgment sway is measured with agreement rates, the upper limit models need to reach cannot be set either.

Most tools used to measure graders were originally built to measure human graders. These include agreement measures like kappa, alongside methods that separately gauge whether a grader is lenient or harsh, or how well they discriminate. All of these have been built up by psychometrics, the academic discipline of measuring human abilities and judgments with numbers. The measuring stick was not newly created; rather, the subject being measured changed from humans to models. The exception is a strand that looks directly inside the model, which we will see later.

Five model judges and six clinicians graded the answers to clinical questions posted by Rwandan health workers across eleven items. Even the best judge produced judgments comparable to humans on only four out of the eleven items, and when multiple judges were combined into a jury, the number increased by just one. Some models were consistently lenient while others were consistently harsh. When the input language was switched from English to Kinyarwanda, performance and cost-effectiveness both dropped. The authors conclude that while judges showed potential, their inability to handle linguistic and cultural context is a decisive limitation (DOI: 10.1038/s41746-026-02992-w, peer-reviewed journal). The preprint of the same study adds that what is needed may not be a better judge, but a fundamental rethinking of what counts as "correct" in clinical AI evaluation, because those standards currently rest on the judgments of clinicians who disagree sharply with one another (DOI: 10.1101/2025.10.27.25338910, preprint).

What Shakes the Verdict Is Already on the List

First is the position where the answer is placed. If the bias in 「How AI Gets Things Wrong — The Test-Taker Who Stumbles Outside the Exam Hall」 is a property where results lean in one direction, the judge's bias is that leaning showing up in the scoring. Position bias is the tendency for judgments to lean toward whatever is placed at the front or the back, regardless of content, when two answers are placed side by side and asked to be chosen. The "72% of the time the first-presented answer is chosen in majority voting" seen earlier is an example of this.

Breaking two answers into pieces, matching similar contents together, and then having them graded reduced this tendency. In an experiment where six models graded 11,520 pairs of answers, consistency—the degree to which they picked the same side even when the order of the two answers was swapped—increased by an average of 47.46% compared to the original value, and rose up to 98% in one model. However, the abstract does not disclose what the pre-correction consistency was (arXiv:2310.01432, preprint).

Second is the wording of the grading instructions. A single judge model was instructed to grade 90 reports—which rephrased intensive care unit mortality risk prediction results using a language model—by changing only the design of the instructions into three variations. Agreement with clinicians was measured using the intraclass correlation, which measures how closely the scores of two raters move together (larger means a better match). While all three methods showed 'substantial' agreement with clinician scores, two methods yielded 0.94 while the remaining one was significantly lower at 0.82 (p<0.001). The difference from the clinician average score also ranged from 0.1 to 4.7 depending on the method (DOI: 10.5114/ait/221943, peer-reviewed journal). Shaking also appeared in a benchmark, a collection of test questions that have multiple models solve the same problems to compare scores. In a tool study where four judges were subjected to four benchmarks covering safety, persuasion, misuse, and agent behavior, none of the tested judges were consistently reliable across all four benchmarks. In preliminary experiments, verdicts were shaken by formatting changes, paraphrasing, increases or decreases in verbosity, and flipping answer labels (arXiv:2603.05399, preprint).

Third, temperature unexpectedly contributed almost nothing. Temperature is a setting value that determines how boldly to pick even candidates with low probabilities when drawing the next token. In a classification task labeling hateful content, mental health, and drug review posts, 4 types of judges × 3 types of instructions × 6 temperature levels × 3 seeds were fully crossed to collect 64,800 observations. Seeds are the starting values for random draws. The degree to which scores were scattered (variance) was then broken down by cause. The difficulty of the questions themselves was overwhelming at 74.6–93.0%, the chemistry of which question was handled by which judge came next at 2.65–14.75%, and the contribution of temperature was practically 0 (DOI: 10.2139/ssrn.6432598, preprint).

This seems to clash with the "shaking remains even when random draws are turned off" seen earlier, but one side was picking between two answers while the other was a classification task. If tasks differ, the place where shaking emerges can also differ.

Fourth is format and style, and here the direction of skewing first becomes clear. When judging models trained to match human preferences were shown original answers paired with answers where length, structure, jargon, flattery, or ambiguity was intentionally inflated, they chose the inflated side in over 60% of cases. More importantly is the sign of the correlation. Correlation is the degree to which two values move together, and a negative number means they move in opposite directions. These five features showed a weak negative correlation (average -0.12) with human preference labels, but a medium-strength positive correlation (average +0.36) with labels produced by strong judge models. This means humans slightly dislike these features while strong judge models are drawn to them, and the authors read this as a signal that models may overly rely on superficial appearance cues. The training method proposed by the authors reduced the mismatch with human preferences from 39.4% to 32.5% and the difference in inflated-preference compared to humans from 20.5% to 10.0%, but could not eliminate it (arXiv:2506.05339, preprint).

Fifth is intentionally targeted phrasing, the same idea as adversarial examples—inputs chosen to make the model fail. In a study published in 2024, simply appending a short phrase to the end of an answer inflated the judge's score. When the judge model was unknown, a phrase obtained by first attacking a proxy model was transferred and attached, and even then, the score was inflated enough to get the highest score regardless of what the graded text was. And judges were much more vulnerable to this attack when asked to score than when asked to pick between two things (arXiv:2402.14016, preprint).

Regulatory phrasing tricked judges in the same way. In a benchmark translating two principles from the UK financial regulator into 168 cryptoasset advertising scenarios, the accuracy of a 120-billion-parameter (the numbers inside a model determined by training) judge that was most robust to normal inputs dropped from 0.74 to 0.27 on inputs packed with regulatory keywords. The authors call this "compliance theater." Agreement between this and other families of judges on the same inputs reached a kappa of only 0.16, and based on this value, the authors trace the cause of the failure not to the data, but to that model (arXiv:2608.14329, preprint).

Is It Too Generous to Its Own Answers — and Is That Wrong?

Self-preference bias is the tendency of a model tasked with grading to give higher scores to answers it generated itself than to answers generated by others. In a 2024 measurement, a model clearly showed this bias, but the same study found the cause in an unexpected place. The model gave significantly higher scores than humans to texts that were less unfamiliar to it—specifically, texts where the model was good at guessing the next token—regardless of whether it had written those texts itself. The authors view the foundation of this bias not as narcissism, but as familiarity (arXiv:2410.21819, preprint).

So, is it wrong to view one's own answers more favorably? When this question was posed across seven model families on tasks where correct answers can be verified, such as math, factual knowledge, and code, the results were threefold (arXiv:2504.03846, preprint). First, the stronger the model, the greater the self-preference, but a significant portion of that preference was justified, as it actually selected better answers. Second, when the grader made mistakes on the side of generating answers, a harmful self-preference remained, and this share was more pronounced in stronger models. The authors interpret this as a sign that stronger models have a harder time recognizing when they are wrong. Third, having them write a long chain of thought before grading reduced that harmful share. A chain of thought is writing out intermediate solving steps in sentences before producing an answer (Does AI Really 'Think'? — A Question Researchers Still Fighting Over).

Harmful self-preference remained even when grading criteria were objectively nailed down. In benchmarks where correctness can be verified programmatically for each rubric item, the probability that the judge would incorrectly mark an item as passed—when the answer-generating model had actually failed it—could be over 50% higher when that answer was the judge's own output. In a medical conversation benchmark where the rubrics were subjective, self-preference pushed model scores by up to 10 points. This is a margin that can flip rankings when lining up top-tier models. Grouping multiple judges reduced it, but did not make it disappear (arXiv:2604.06996, preprint).

Evidence on the other side comes in three branches.

First, self-preference was not detected in settings asking whether to accept a correction—whose correctness had been confirmed by a rule checker—into one's own draft. The rejection rate difference between a model keeping the exact context in which it wrote the draft and a new model seeing the same draft for the first time like someone else's writing was -5.1 percentage points. The 95% confidence interval, which represents the plausible range for the true value, was [-12.9, +2.7], spanning both negative and positive numbers, meaning no difference was detected. This is the result of testing four mid-tier model families, and the authors pin down that this sample size cannot rule out effects smaller than about 13 percentage points. A single signal from a small pilot experiment was not reproduced when the scale was increased. The remaining observation is a qualitative one. When models rejected verified corrections, 97% of the reason was flaw-pointing, not taste (arXiv:2606.20093, preprint).

Second, erasing writing styles mostly makes self-preference vanish, but just attaching name tags brings it back. Ten models were made to grade targets stripped of individual model writing styles—specifically, lists of selected constraints to place on a story. Once the quality of the selected items and the strictness of the judges were controlled while keeping who made them hidden, self-preference disappeared on three out of four grading axes and reversed on the remaining one. The judges saw their own items as less original. However, when items of equal quality were placed side by side and only name tags were attached, scores went up if the label "mine" was attached and down if the label "someone else's" was attached, even though no model identity was revealed. This was independent of the actual source (arXiv:2608.18091, preprint).

Third, shifting fields can flip the direction. In Quebec French automotive insurance legal questions, 52 judges were compared against expert-created standards. Even the strongest judge was only moderately reliable, and the judges frequently disagreed with one another. And contrary to the "favoritism toward AI-written answers" reported in general domains, these judges were, if anything, stricter toward AI-written answers. "If anything" is a reservation added by the authors, serving as a clue that the effect is not large (DOI: 10.21203/rs.3.rs-10304807/v1, preprint).

To sum up, self-preference was not explained by mere narcissism. It is being generous to familiar text, reacting to the tag "mine," and having a hard time recognizing when one is wrong. The last of these is the worst. This is because the moment a judge is most needed is precisely the moment when the answer is wrong.

There is a Direction to the Shaking

A study where eight models and human experts graded 1,726 essays together measured two things separately. One is the ability to tell good writing from bad writing (discriminative power), and the other is the location of the threshold, meaning from which point onward high scores are given. On both counts, the models differed from humans. The human discriminative power estimate was about twice that of the models, and the models squeezed scores toward the middle and narrowed the range, systematically withholding top-tier grades. The authors conclude that the low agreement between humans and models is partly because the models failed to tell them apart and partly because the threshold shifted (DOI: 10.1177/01466216261471171, peer-reviewed).

In a different setting, a different kind of compression emerged. This was a study, examined through a pre-registered experiment, of a cycle where survey response texts were generated by a model and then graded back onto a numerical scale by a model from the same family. When a model turned its own writing into scores, the scatter of errors (variance) was compressed fourfold (0.21 versus 0.87), and a systematic bias toward one side remained. The independent alternative proposed by the authors actually lagged in accuracy (65% versus 87%). Therefore, the authors frame the contribution of their method not as accuracy, but as methodological independence (DOI: 10.2139/ssrn.6208718, preprint).

A study that measured how lenient or harsh each grader was using a multi-faceted Rasch model obtained results showing that a few among the ten models were better. At the same time, it explicitly states that because the sample is small, it should not be read as a model ranking (DOI: 10.1111/emip.70018, peer-reviewed).

This "direction" may not just be a metaphor. This is a study that looked into internal states—the numbers models produce during computation—across seven judges, seven bias types, and nine benchmarks. Inputs caught in bias were pushed away in a few fixed directions specific to each type. Pushing the internal state in that direction moved scores in both directions, making it possible to reproduce biased grading on clean inputs or reverse the original grading on biased inputs. The change when pushing by the same magnitude in any random direction was about ten times smaller. By looking only at this direction, they anticipated in advance where judges would stumble across three unseen benchmarks (arXiv:2607.11871, preprint). The exception mentioned earlier is a new measurement tool absent from psychometrics. A study attempting to fix bias using the same tool reduced unfair self-preference by up to 97%, but immediately added that that adjustment was unstable in front of justified self-preference (arXiv:2509.03647, preprint).

When Judges Agree, It Does Not Mean They Are Right

Eight models created 922 patches fixing security vulnerabilities, and judges graded whether those patches blocked the vulnerabilities while preserving original functionality. The authors compared these grades against execution results from a separate patch bundle where the code could actually be run and verified. Two judges from different families agreed well with each other on functionality verdicts with a kappa of 0.75, but their agreement with the execution results was a kappa of 0.26 or lower.

You have to separate two properties to read this number. Reliability is the degree to which measuring the same thing yields the same value again. Validity is the degree to which that value actually measures what it claims to measure. "The Score Went Up. But What Went Up?" threw the same question about construct validity to evaluators that it asked of benchmarks. Think of a ruler to make it simple. If an object always measures 27 centimeters every time you measure it, this ruler has high reliability. But if that object is actually 30 centimeters, this ruler is consistent but fails to measure correctly. The authors' conclusion is this. What agreement among judges measures is reliability, not validity. Both judges reported "over-corrections"—where fixing a vulnerability ruined working functionality—far more than execution confirmed, and only 5% to 10% of those reports were correct. In actual Java code, the two judges' accuracy estimates diverged so that neither could be trusted (DOI: 10.3390/jcp6050153, peer-reviewed journal).

Why do judges agree well with each other? They might have captured quality together, or they might share the same blind spots. The agreement number alone cannot tell these two apart. A study that measured the scores of 42 judges across benchmarks in eight Indian languages and four domains by angles separated these two. When grading criteria were subjective rubrics, judges agreed with each other as well as humans did with each other, but agreement with humans reached only 58% to 66% of human-to-human agreement. Judges were clustered on axes that humans do not weight heavily. Bundling multiple judges together converged not on the human axis, but on the axis shared by the judges. However, on a single rubric where answers could be verified, that gap mostly closed (arXiv:2606.03043, preprint). The earlier variance decomposition study also touched on this same distinction separately. Rubric-style instructions increased consistency among models but did not always increase agreement with human labels, and the authors call this the distinction between reliability and validity that commonly used metrics squash (DOI: 10.2139/ssrn.6432598, preprint).

What if you hand them the correct answers outright? The authors extracted 258 questions from financial reports, created verified ground truths using a graph-based method, and set up judges under various conditions. Even a strict judge armed with the correct answers disagreed with those answers 5.9% of the time, and false acceptance—passing a wrong answer as correct—was 2.9%. A judge making judgments without the correct answers disagreed 22.3% of the time. A judge given the full source text but no correct answers achieved an agreement of only 54.3% and passed one out of every three wrong answers (false acceptance of 33.1%). False acceptance clustered in finding exact values (50%) and multi-step lookups (46%), which are precisely the types where accuracy matters most. The authors conclude that judges cannot be trusted for verifying structured information and that graph verification is a necessary alternative. We must keep in mind that this is a study proposing its own alternative (DOI: 10.2139/ssrn.6482162, preprint).

Where We Can Delegate, and Where We Cannot

None of the cited papers state that we should never use a judge. What varies is the role we give to judges.

Rankings hold up, but thresholds collapse. Yet both are values measured between judges. This is a study where two judges graded the faithfulness of network security measure reports written by six models. The ranking agreement of the scores—in other words, how similar the two lineups were—was strong at 0.875. However, the agreement on threshold judgment, meaning whether the report could be deployed, was a Kappa of 0, no better than chance. In the context of the abstract, this 0.875 is a value between the two judges, not a value compared against human-ranked orders. In a test where the same reports were cross-checked with a program, not a single fabricated identifier was found (DOI: 10.2139/ssrn.6910008, preprint). The prescription from the vulnerability patch study earlier is the same. Let codes run and be re-evaluated, and let judges be used only for lining up models (DOI: 10.3390/jcp6050153, review paper page).

Read the effect measured by a judge as a direction and a relative value, not as an absolute rate. The most complete case came from the clinical setting. When instructions were added telling models not to over-confidently guess when evidence is scarce, dangerous over-confident answers measured by the reference judge dropped from 49.3% to 24.7% (drop measured in pairs: 24.7 percentage points, 95% confidence interval [21.8, 27.7]). This is the result of four models answering the same set of questions, and the direction was maintained even when changing models and phrasing. Yet the size of the effect depended on the judge. A different family of judges agreed on the direction while cutting the effect by nearly half (13.1 percentage points).

Three blinded clinicians defined the reference judge as follows: it is a screening tool that catches almost everything that needs to be caught (sensitivity 1.00) but also frequently filters out harmless things (specificity 0.55), and not an instrument with calibrated scales. Sensitivity is the ratio of things that needed to be filtered out that actually were, and specificity is the ratio of harmless things left alone as harmless. There was also a price to pay. The same instructions dropped the correct diagnosis rate from 80.3% to 50.3%, and this loss was nearly 0 for one model and 58 percentage points for another. That does not mean the improvement was an optical illusion created by the judge. Actual behavioral changes were confirmed compared against a control condition that matched only the instruction framework, and the authors note this was not a loop where the judge fed its own criteria back into itself (arXiv:2607.18086, preprint).

The authors' conclusion is this. The clinical safety effect measured by the judge must be reported as a direction and relative value, using human review as a baseline, alongside usefulness, and must never be reported as an absolute rate with calibrated scales. They also firmly state that this result does not prove readiness for clinical deployment.

Deterministic tests—in other words, program checks that always produce the same result for the same input—are checked with that test for things that can be checked. A case study covering software test automation baked this boundary into its design. Language models are useful for broadening the scenarios to test, but dangerous as judges. This is because they invent impossible states, misread meaning, and inflate weak evidence. Therefore, in this design, while models can write drafts, they cannot write observational values, pass-fail judgments, or deployment readiness declarations (DOI: 10.2139/ssrn.7173619, preprint/single case study).

The judge itself sometimes becomes the target of an attack. This is the result of auditing a safety evaluation using 6,642 human-verified labels. Distribution shift—meaning when data seen during training differs from data actually encountered—occurs here between the answers seen when validating the judge and the answers from attack experiments. When different writing styles for each attacked model, outputs distorted by attacks, and ambiguities varying by scenario overlap, judge performance frequently drops to levels close to chance. Moreover, many attacks were not actually eliciting harmful content, but rather poking at the judge's blind spots and inflating success rates (arXiv:2603.06594, preprint). The attack may not have grown stronger; rather, the yardstick may have wobbled.

A prescription that raises agreement is not necessarily a prescription that raises validity. This is a study testing a three-round deliberation method across two text collections (corpora), where three models grade separately, review each other's scores and rationale, and then recheck. Disagreement between models dropped significantly across all questions, and in one corpus, the coefficient of variation comparing score dispersion to the average dropped from 7.29% to 2.78% (a 61.9% relative improvement). The degree of alignment with human-standard scores varied by task, and in one corpus, rank matching actually worsened slightly. And in both corpora, scores systematically rose after deliberation. The authors add that this upward shift can be reversed by linear calibration adjusted to human standards (DOI: 10.21203/rs.3.rs-10367743/v1, preprint).

There are also low-cost interventions. Adding graded example answers to the instruction prompt brought agreement with humans close to the agreement among humans themselves (DOI: 10.35542/osf.io/cbhgz_v1, preprint). Making models write out reasons instead of just outputting scores consistently increased the correlation between ChatGPT's scores and human grading in a 2023 test (arXiv:2310.05657, preprint). Having models write a long chain of thought before grading reduced harmful self-favoritism (arXiv:2504.03846). In a Quebec legal study, selecting judges based on their ability to catch wrong answers, providing authoritative reference materials, and bundling a few strong judges each reduced dangerous false approvals. Still, those authors conclude that a single judge alone cannot be trusted enough to evaluate high-risk legal answers, and propose using them in a human-in-the-loop manner (DOI: 10.21203/rs.3.rs-10304807/v1, preprint). The approach of handing questions over to a stronger judge when confidence is low crossed the 80% human-agreement line—which a strong single model could barely reach—even on question bundles, by using a smaller model instead. In return, one must accept the cost of forgoing some judgments (arXiv:2407.18370, 2024, preprint).

There is no escaping this by changing formats, either. Scoring was more vulnerable to wording attacks (arXiv:2402.14016), and choosing between two options fluctuated wildly with every repetition (arXiv:2606.13685). Among the studies gathered in this article, none concluded that a single format is safe.

What Is Certain Right Now and What Is Not

① A single grading is unstable. When asked the same question fifty times, the winner flipped by an average of 13.6%, and it took an average of eleven attempts to recover the baseline verdict. However, since both judges were models from the same company, replication with models from other companies remains to be seen (arXiv:2606.13685, preprint).

② Apparent agreement is not evidence of capability. In settings where apparent agreement was 74.7% to 84.5%, the kappa for five out of six machine graders was 0.027 or lower (DOI: 10.2139/ssrn.7217178, preprint), and kappa between humans was also commonly between 0.40 and 0.80 (DOI: 10.21203/rs.3.rs-10047888/v1, preprint).

③ Instability includes directional bias, not just noise. Models crammed scores toward the middle and rarely gave top grades, and their discriminative power was about half that of humans (DOI: 10.1177/01466216261471171, peer-reviewed), while biases appeared as directional patterns inside the models (arXiv:2607.11871, preprint).

④ Self-preference exists but is not simple. A substantial portion was justified (arXiv:2504.03846), it was not detected when models were asked whether to accept corrections validated as correct by a verifier (arXiv:2606.20093), and it largely disappeared when authorship was masked, only to return with just a name tag (arXiv:2608.18091). Still, the harmful self-preference when the model that generated the answer was wrong remained (arXiv:2504.03846), and even in program-verified scorecards, the probability of marking its own output's failure as a pass could increase by over 50% (arXiv:2604.06996). All four are preprints.

⑤ Agreement among judges is reliability, not validity. A verdict that matched with a kappa of 0.75 among judges had a correlation of 0.26 or lower with execution results (DOI: 10.3390/jcp6050153, peer-reviewed), and bundling multiple judges together gathered them not around a human axis, but around an axis shared by the judges (arXiv:2606.03043, preprint).

Some things are still not certain.

There are areas where even what the correct answer is has not been established. In domains where human graders diverge sharply, the suggestion from clinical authors is that we must rebuild the definition of "correctness" from scratch (DOI: 10.1101/2025.10.27.25338910, preprint), and the Turkish study's failure to settle on an interpretation is for the same reason (DOI: 10.2139/ssrn.7217178, preprint).

Evidence that interventions bundling and deliberating among judges improve validity is mixed. Bundled judges gathered around the judges' shared axis (arXiv:2606.03043), but in a Quebec legal study, bundling a few strong judges reduced dangerous false approvals (DOI: 10.21203/rs.3.rs-10304807/v1), whereas a jury in a Rwandan study only managed to add one item comparable to humans (DOI: 10.1038/s41746-026-02992-w). Deliberation systematically pushed scores upward (DOI: 10.21203/rs.3.rs-10367743/v1), and internal directional adjustment was unstable in the face of legitimate self-preference (arXiv:2509.03647). Conversely, inserting example answers, having models write out reasons, and escalating to higher-tier judges improved agreement with humans. These results are also specific to the tasks and models tested by each study.

Most of the cited research notes that we should keep using judges but narrow the roles assigned to them. There is evidence for reading directions and primary screening. Reporting safety effects as directions and relative values, and defining judges as screening tools with a sensitivity of 1.00 and specificity of 0.55, are also phrases from the same papers (arXiv:2607.18086). Ranking is conditional. "Judges for rankings only" is an author's prescription, not a result measuring the validity of rankings, and the 0.875 seen earlier is also a figure among judges. Among the studies gathered in this article, only one deliberation study directly compared rankings against a human standard, and even there, one corpus was maintained while another slightly worsened (DOI: 10.21203/rs.3.rs-10367743/v1). There are three conditions attached to ranking. A single grading will not suffice, self-preference can dictate the top rankings, and judges rarely award top grades themselves. Absolute values such as pass/fail judgments, deployment-readiness judgments, or exact safety percentages are not yet places to entrust to judges. The cost of guarding that boundary is paid for not with better judges, but with human review as a benchmark and deterministic tests.

Source Papers

• "The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation" (2026, preprint) — arXiv:2606.13685 — Pairwise preferences flip by an average of 13.6% across repetitions, cross-judges agree 76% ($\kappa=0.51$), requiring 11 trials for recovery
• "Inter-Rater Reliability of LLM and Rule-Based Annotation ..." (2026, preprint) — DOI: 10.2139/ssrn.7217178 — Apparent agreement of 74.7–84.5% alongside a kappa of 0.004–0.027, explicitly highlighting the coexistence of two interpretations
• "A Review of a Hybrid Evaluation Framework for Clinical AI" (2026, preprint/review) — DOI: 10.21203/rs.3.rs-10047888/v1 — Review of 66 studies. Human-to-human kappa of 0.40–0.80, "approximating expert-level noise"
• "Human evaluators vs. LLM-as-a-Judge in global health" (2026, peer-reviewed) — DOI: 10.1038/s41746-026-02992-w — 5 judges versus 6 clinicians. Only 4 out of 11 items match comparably, adding a jury adds 1 more
• (Preprint edition of the above study, 2025, preprint) — DOI: 10.1101/2025.10.27.25338910 — The sentence arguing for a fundamental reconsideration of the concept of "correctness" is found here
• "Split and Merge: Aligning Position Biases in LLM-based Evaluators" (2023, preprint) — arXiv:2310.01432 — 11,520 answer pairs. 47.46% relative improvement in consistency, reaching 98%, with pre-correction values omitted from the abstract
• "Effect of evaluation prompt strategies ... in critical care" (2026, peer-reviewed) — DOI: 10.5114/ait/221943 — 3 types of prompts. Intraclass correlation of 0.94 versus 0.82, variance of 0.1 versus 4.7
• "Judge Reliability Harness: Stress Testing the Reliability of LLM Judges" (2026, preprint) — arXiv:2603.05399 — Swayed by formatting, paraphrasing, verbosity, and label flipping, with no steadfast judge
• "Generalizability Theory for LLM-as-Evaluator Reliability" (2026, preprint) — DOI: 10.2139/ssrn.6432598 — 64,800 observations. Question difficulty accounts for 74.6–93.0%, while temperature contribution is practically 0
• "Flattery, Fluff, and Fog: Idiosyncratic Biases in Preference Models" (2025, preprint) — arXiv:2506.05339 — Over 60% preference for inflated sides, human labels at -0.12 versus scoring model labels at +0.36
• "Is LLM-as-a-Judge Robust? Universal Adversarial Attacks ..." (2024, preprint) — arXiv:2402.14016 — Inflating scores with short attack phrases, where absolute scores are more vulnerable than pairwise comparisons
• "A Four-Axis Trustworthiness Benchmark for LLM-as-Judge ..." (2026, preprint) — arXiv:2608.14329 — Dropping from 0.74 to 0.27 under keyword stuffing, with a kappa of 0.16 against judges of other families
• "Self-Preference Bias in LLM-as-a-Judge" (2024, preprint) — arXiv:2410.21819 — Higher scores given to familiar text than humans would, regardless of whether it was self-generated
• "Do LLM Evaluators Prefer Themselves for a Reason?" (2025, preprint) — arXiv:2504.03846 — Largely justified, with the harmful fraction of self-errors more pronounced in stronger models
• "Self-Preference Bias in Rubric-Based Evaluation of Large Language Models" (2026, preprint) — arXiv:2604.06996 — Over a 50% increase in the likelihood of marking a failed item as a pass when it is its own output
• "Self-Preference Is Weak or Absent in Verifiable ... Revision" (2026, preprint) — arXiv:2606.20093 — Rejection rate difference of -5.1%p (95% CI [-12.9, +2.7]), making it impossible to rule out differences under 13%p
• "Self- and Other-Labels Induce Bidirectional Bias in LLM Judges" (2026, preprint) — arXiv:2608.18091 — 3 disappear and 1 reverses under blinding, showing scores shift bidirectionally based solely on name tags
• "Can LLMs Judge Legal Accuracy? ... in a Low-Resource Language" (2026, preprint) — DOI: 10.21203/rs.3.rs-10304807/v1 — 52 judges. Overall quality fails to predict the false approval rate, being stricter on AI answers
• "Evaluating AI-Based Automated Essay Scoring Through SDT" (2026, peer-reviewed) — DOI: 10.1177/01466216261471171 — 1,726 essays. Human discriminative power is about twice as high, while models exhibit central tendency and score compression
• "Evaluating Rater Effects of LLMs in Automated Essay Scoring" (2026, peer-reviewed) — DOI: 10.1111/emip.70018 — 10 models and multi-faceted Rasch. Explicitly states that due to a small sample size, ranking is not the goal
• "Measuring Self-Rating Bias in LLM-Generated Survey Data" (2026, preprint) — DOI: 10.2139/ssrn.6208718 — Error variance compressed 4-fold during self-rating (0.21 versus 0.87), with alternatives lagging in accuracy
• "Inside the Unfair Judge: A Mechanistic Account of LLM-as-Judge Bias" (2026, preprint) — arXiv:2607.11871 — Biases manifest along low-dimensional directions by type, enabling bidirectional manipulation
• "Breaking the Mirror: Activation-Based Mitigation of Self-Preference" (2025, preprint) — arXiv:2509.03647 — Up to a 97% reduction in unjustified self-preference, though unstable above justified self-preference
• "Fixing, Breaking, or Faking It? ... and the Limits of LLM-as-Judge" (2026, peer-reviewed) — DOI: 10.3390/jcp6050153 — 922 patches. Inter-judge $\kappa=0.75$, but $\kappa\le0.26$ against execution. Judges are suitable only for ranking
• "The Geometry of LLM-as-Judge: Why Consensus Is Not Human Alignment" (2026, preprint) — arXiv:2606.03043 — 42 judges. Reaching 58–66% of human agreement, with ensembles converging onto axes shared by the judges
• "When the Judge is Wrong: ... Against Graph-Verified Ground Truth" (2026, preprint) — DOI: 10.2139/ssrn.6482162 — 258 questions. Judges provided only with raw text show 54.3% agreement and a 33.1% false approval rate
• "NetHarden: Auditable Attack-Graph Hardening with LLM Evaluation" (2026, preprint) — DOI: 10.2139/ssrn.6910008 — Rank correlation of 0.875, release threshold agreement of $\kappa=0$, and fabricated identifiers scoring 0 on programmatic checks
• "Judge-dependent safety gains ... in clinical LLMs" (2026, preprint) — arXiv:2607.18086 — Overconfident answers drop from 49.3% to 24.7%, with alternative judge families showing half the effect and deployment readiness remaining unproven
• "Beyond E2E Scripts: ... Without Letting the LLM Be the Oracle" (2026, preprint/single case study) — DOI: 10.2139/ssrn.7173619 — Designed so the model cannot write observations, pass/fails, or deployment decisions
• "A Coin Flip for Safety: LLM Judges Fail to ... Measure Robustness" (2026, preprint) — arXiv:2603.06594 — 6,642 labels. Performance drops to random chance when distribution shifts overlap, while attacks artificially inflate success rates via loopholes
• "Multi model deliberation improves the stability ... by LLMs" (2026, preprint) — DOI: 10.21203/rs.3.rs-10367743/v1 — Coefficient of variation drops from 7.29% to 2.78%, though alignment remains task-dependent and systematically upward after deliberation
• "Anchor Is the Key: Toward Accessible Automated Essay Scoring ..." (2025, preprint) — DOI: 10.35542/osf.io/cbhgz_v1 — Providing graded example answers brings human agreement close to inter-human agreement levels
• "A Closer Look into Automatic Evaluation Using Large Language Models" (2023, preprint) — arXiv:2310.05657 — Outputting only numbers is suboptimal, whereas prompting for reasons increases correlation
• "Trust or Escalate: LLM Judges with Provable Guarantees ..." (2024, preprint) — arXiv:2407.18370 — Selective evaluation that escalates to a higher-tier judge when confidence is low, trading off coverage to maintain an 80% baseline