How AI Gets Things Wrong — The Test-Taker Who Stumbles Outside the Exam Hall

When a Hundred-Point Answer Crumbles Before an Unfamiliar Photograph

Around the time COVID-19 was spreading, several AIs emerged that could screen for the infection using a single chest X-ray. Some even reported results showing they missed not a single infected person, reaching 100%. These were figures suggesting they performed far better than medical specialists (PMID 34660166).

Yet when we look closely at where these AIs relied most heavily when deciding their answers, a strange scene appears. The spot they relied on most was outside the lungs. It is implausible for areas outside the lungs to dictate the answer when detecting a lung infection. When tested again with photos from elsewhere that were not used during creation, their performance dropped drastically (PMID 34660166).

Artificial intelligence is a general term for technology that seeks to make computers accomplish tasks that humans used to do by using intelligence. It is called AI for short. The work of making AI becomes easy to see if we compare it to a test taker studying. The collected examples are the workbook. AI studies with this workbook, and humans pull out some parts of the workbook in advance to give as final test questions. This is a device to screen out test takers who merely memorized the entire workbook, which "AI Models" in another part of this series covers in detail.

However, the set-aside test questions also came from the same workbook. Therefore, this test measures proficiency only on problems resembling the workbook. The "Outside the Test Venue" of this article is somewhere different from that workbook, namely the place where AI is actually used. The photos from elsewhere where the X-ray AI crumbled are precisely outside the test venue. This article follows what happens when a test taker who even passed the exam steps outside the test venue.

Errors branch out into several kinds. There are errors that produce plausible yet wrong words, errors that repeat biased results, errors that collapse in situations different from what was learned, and errors when someone deliberately tries to trick the system. Added to these is another problem where test scores are inflated beyond actual capability.

AI fails in various ways, and the way to check each varies. The test taker analogy merely carries over the structure of tests and grading, and this article does not judge whether AI understands anything either.

A Machine That Finds How to Answer from Examples

Suppose you are making a program that recognizes a cat in a photo. You could write down rules like "if it has pointy ears and whiskers, it's a cat," but the exceptions are endless. Today's AI walks a different path. Machine learning is a method where, instead of humans writing down rules one by one, computers figure out for themselves how to process inputs from sample data or experience.

A model is a computational device that takes an input and produces an output, and the numbers inside it can be adjusted to get closer to the desired answer. Training is the process of slightly tweaking the numbers inside the model to fit sample data or experience, and is also called learning. If you train it by showing countless cat photos and the label "cat," the model figures out for itself how to process photos to match those examples. The way it figures that out is contained in the numbers inside the model, and does not take the form of a list of rules that humans read.

Many of the errors that appear in this article start here. The way a model processes inputs comes mostly from examples, which also determine the model's structure and training method. Therefore, what is contained in the examples can seep into that method. It breaks down in front of situations not found in the examples, things skewed and packed into the examples are repeated as tilted results, and random clues that accidentally overlap with the correct answer in the examples can become the basis for the answer. How memorizing and properly learning are separated is covered in another piece in this series, "AI Models". This article looks at the next place, where the learned method goes wrong.

Answering Even When It Does Not Know the Problem

There is a task to read data contained in a table and turn its contents into writing. When this job was given to open model sets that anyone can download and use (Llama 2, Mistral, Zephyr), the writing was smooth and coherent. Yet whether judged by human evaluators or automated grading, in over 80% of the outputs there was at least one place where the meaning was wrong (DOI: 10.18653/v1/2024.acl-long.651, 2024).

Hallucination is when an AI creates content that sounds plausible but is factually incorrect or contradicts the given material. Hallucinations branch into several types. Those that conflict with the text given by the user, those that contradict what the model said earlier, and those that conflict with facts known to the world (DOI: 10.1162/coli.a.16). Failures reported on the clinical side also include fabricated citations (PMID 42251377). A smooth sentence is not a sign that the content is correct.

The model behind the chatbot makes writing by looking at the preceding words and picking the words that come next by probability to string them together. That method is covered in "How Generative AI Answers", another part of this series. Here we look at only two explanations for where hallucinations come from.

The first explanation lies on the side of the training data and the question. A model trained on vast online writing is fluent in language. At the same time, it can jump to conclusions about information biased toward the writing it learned. It may also misread a question with ambiguous meaning, or spit out information so that it only superficially fits the question (DOI: 10.48550/arxiv.2401.01313, a preprint that has not undergone peer review).

The second explanation resembles a test. Like a test-taker facing a difficult exam question, the explanation is that the model behind the chatbot sometimes guesses when it is unsure. Instead of saying it does not know, it puts out a plausible but incorrect sentence. This explanation was presented by a 2025 paper, which is a preprint that has not undergone peer review (arXiv:2509.04664v1).

This preprint separates why hallucinations arise and why they remain. First is why they arise. The model is pre-trained on vast writing, and if it fails to distinguish incorrect sentences from facts, it ends up outputting incorrect sentences as if they were facts. The authors view hallucination as a wrong answer coming from this judgment separating true and false, and see it as arising naturally in a statistical sense during the pre-training process (arXiv:2509.04664v1).

Next is why they remain. Most evaluations do not award points to answers stating that the model is unsure, such as "I don't know." Then guessing even when unsure raises the score. The authors argue that because of this grading method, hallucinations do not disappear and remain (arXiv:2509.04664v1).

Techniques to reduce hallucination are organized into more than 32 types (DOI: 10.48550/arxiv.2401.01313). Gathering 44 studies evaluating medical AI, the approach of first finding related documents and placing them right in front of the model reduced hallucinations by 30 to 50%. It reduced them, but did not eliminate them. The approach of inserting a human verification step in the middle reduced them by up to 95%, but carries concerns that it is difficult to scale up (PMID 42251377).

The authors of the previous preprint offer a prescription targeting the side of why they remain. Instead of making more new hallucination tests, they propose fixing the grading methods of existing tests that drive the leaderboards ranking AI scores (arXiv:2509.04664v1). Hallucinations are reduced by various methods, but according to the aforementioned preprint, they remain even in the most advanced systems (arXiv:2509.04664v1).

A One-Sided Workbook

When 304 students at a German university were asked, they rated decisions inside the university as fairer in both procedure and outcome when made by an algorithm than when made by a human (DOI: 10.1145/3351095.3372867). Contrary to the students' feelings, the criticisms compiled by the same paper point in the opposite direction. Rather than becoming objective and fair decision-makers, algorithms can repeat the biases already present in their training data.

Bias is the property where AI repeats or magnifies one-sided results concerning a specific group or situation, and it often stems from skewed training data, though it also arises from what is treated as the correct answer or how the model is evaluated and used. Bias enters through multiple pathways. Commonly cited factors include data that fails to evenly represent groups, incorrect labels, and the way humans and AI interact. Flawed proxy indicators are another example. This is when an easily measurable alternative value is used as the correct answer instead of what one actually wants to know, and that value happens to be skewed (PMID 42466917).

Looking specifically at the data side, societal data frequently fails to adequately include minority groups. The reasons for this are manifold, ranging from past discrimination to biases in how data is collected and selected. This is where the saying "garbage in, garbage out" comes from (DOI: 10.1145/3588433). If the collection method itself is skewed, gathering more data using the same method will only capture the same bias. Simply increasing the amount of data does not automatically erase bias.

Let us consider a hypothetical example. Suppose a company trains a resume screening model using hiring records from the past ten years. If those records show that a certain group was hired less often, the model may also learn that pattern as a "characteristic of a successful application." This is equivalent to a student studying exclusively from a biased workbook and learning that skewness itself as the correct answer.

Bias can also hide behind average scores. In the hypothetical hiring model mentioned earlier, the group that has been hired less often has fewer training examples, meaning the model may make mistakes on that group's applications more frequently. Furthermore, if the past decisions used as correct answers were themselves biased, that bias will not even be revealed by an accuracy score measured against those answers.

Real-world cases also exist. An AI used in pediatric emergency departments to sort patients by urgency showed high overall accuracy, but its performance varied across subgroups of children. However, the studies gathering this evidence are also mixed with publication bias, where positive results are more likely to be published (PMID 42137483). This bias stems from which research findings get published, entirely separately from the AI's own bias. That is why numbers like "high accuracy" must be read with caution. Even if the average is high, the model may fail more frequently in certain groups, and that gap is hard to see until you look at the data broken down by group (PMID 42466917).

Various mitigation methods have been proposed. Approaches spanning the data processing stage, the training stage, and the post-processing stage have been reviewed across 341 papers (DOI: 10.1145/3631326).

However, there are multiple definitions of what constitutes fairness in the first place (DOI: 10.1145/3457607). A review paper on medical AI concludes that fairness cannot be guaranteed by a single metric or a single validation, and that systems must be continuously monitored while in use (PMID 42466917).

Problems Outside the Workbook

A program that filters spam mail may fail to recognize spam that looks different from the spam it learned (DOI: 10.7551/mitpress/9780262170055.001.0001). If the appearance of spam changes, the kind of mail that was filtered out until yesterday might enter the inbox today.

Data has a structure of how often which inputs appear with which correct answers, and this is called a distribution. Distribution shift is when the distribution from which data appears when a model learns and the distribution of data it encounters when actually used become different from each other; this can change how often certain inputs appear, or change the relationship between inputs and correct answers. In the case of a spam filter, changes in the appearance of spam correspond to the former, while changes in what is considered spam correspond to the latter.

Also called dataset shift, it appears in most real-world applications. Its causes range from biases introduced by experimental design, to circumstances where the conditions at the time of use cannot be made identical in advance during learning (DOI: 10.7551/mitpress/9780262170055.001.0001). This means it can occur simply from data collected elsewhere, even if the world does not change. The sharp drop in performance of the X-ray AI from the very beginning also happened in front of photos from elsewhere.

A test-taker who has only solved problems inside a workbook falters when encountering problems outside the test center. Traditional machine learning methods often fail to handle such changes well, and these changes can sharply degrade performance. Methods to reduce the gap between learning and use have also been researched. One very simple method that matches the statistical characteristics of the two sets of data yielded excellent results on standard test data used as benchmarks (DOI: 10.1609/aaai.v30i1.10306).

Models do not always let us know the fact that they are standing in an unfamiliar situation. If the value representing how confident a model is in its own answer is well-calibrated, we can know when to trust the model's answer. However, in front of data with shifted distributions, common methods that adjust this confidence value afterward fell short. Some methods that comprehensively consider the answers of multiple models held up surprisingly well (DOI: 10.48550/arxiv.1906.02530, preprint version published on arXiv).

When a model collapses in front of distribution shift, there is a reason frequently discussed behind it. Shortcut learning is when a model learns easy decision rules that work well on familiar tests but fail to transfer to more demanding conditions or real-world sites. Here, rules do not mean a list that humans read, but rather refer to the way the model actually separates answers. Shortcuts do not show up on ordinary tests, but reveal themselves at points where conditions change (arXiv:2004.07780v5, preprint version published on arXiv).

A test-taker who learns the trick of choosing the correct answer by the length of the choices or the shape of the number rather than reading the meaning of the problem is an example. A similar phenomenon is known in education and comparative psychology, suggesting that shortcut learning may be a property common to learning systems, whether living organisms or machines (arXiv:2004.07780v5).

Let us return to the X-ray AI from the very beginning. The model that gave its answer by relying on the outside of the lung, the model whose performance dropped sharply on photos from elsewhere, overlaps with the appearance of shortcut learning. Authors who gathered research up to 2021 concluded that those models had frequently fallen into shortcut learning and were less suitable for use in clinical settings (PMID 34660166). There is no guarantee that a method that worked on a test will also work in the field.

The Same Test Paper to Human Eyes

If you show a single photo to a model that recognizes images while adding a change so small that human eyes cannot catch it, you can make the model classify that photo as something completely different. That change is different from noise sprinkled at random. It is a change found to make the model's wrong answer as large as possible. The same change has even fooled other models trained on different parts of the same data (arXiv:1312.6199v4, a preprint published on arXiv in 2013).

An adversarial example is an input that looks almost identical to human eyes, but has been deliberately chosen and tweaked just a little bit to make the model get it wrong. In front of such an input, the model may output a wrong answer with high confidence (DOI: 10.48550/arxiv.1412.6572, a preprint published on arXiv). It is similar to stamping a test paper with a stain so faint that it is invisible to human eyes, confusing the test-taker.

What divides distribution shift and adversarial examples is what has changed. Distribution shift is when the data as a whole changes, and can happen without anyone aiming for it. An adversarial example is when someone handpicks and slightly alters individual inputs to make a specific model fail. Intent alone does not separate the two. Just like spam where tactics change to dodge filters, there are also distribution shifts that the other side intentionally causes.

Research to stop them has also advanced. The method of training by assuming the worst-case attack significantly increased resistance to several types of attacks (DOI: 10.48550/arxiv.1706.06083, a preprint published on arXiv).

However, other defenses were easily breached. In studies from 2017, ten proposals claiming they would spot adversarial examples were all defeated by new methods of attack (DOI: 10.1145/3128572.3140444). There was even a defense that claimed to lower the attack success rate from 95% to 0.5%. Yet in front of three new attacks, the model using that defense and the model not using it were breached identically (DOI: 10.1109/sp.2017.49).

Discoveries are also introduced suggesting that attacks like this may be an inherent weakness of deep learning models, meaning the models that are the mainstream of today. On the other hand, the authors who introduced the method of training by assuming the worst-case are more optimistic. They view the power to withstand predefined types of attacks as a stepping stone toward a model that withstands any attack (DOI: 10.48550/arxiv.1706.06083). Where someone tries to trick you on purpose, ordinary accuracy does not guarantee safety.

The Test-Taker Who Saw the Exam in Advance

There is a standard test that measures AI's math-solving skills using elementary school-level arithmetic problems. When 1,000 new problems tailored to match this test in human solve rates, number of reasoning steps, and answer sizes were newly created and given to previous models to solve, their accuracy dropped by up to 8% (DOI: 10.48550/arxiv.2405.00332, preprint published on arXiv in 2024).

A benchmark is a collection of test questions designed to make multiple AI models solve the same problems and compare their scores, and they are usually public. The arithmetic test mentioned earlier is also a benchmark. The claim that the model behind a chatbot is exceptional is usually backed by scores on such public benchmarks (DOI: 10.18653/v1/2024.findings-acl.716). The problem is that public test papers can leak into training data.

Benchmark contamination is when test questions or very similar texts are mixed into a model's training data in advance. It is also called data contamination. A contaminated model can receive a higher score on that test than its actual capability, which can lead to published scientific conclusions that are wrong. The scale of the problem is not easy to measure (DOI: 10.18653/v1/2023.findings-emnlp.722, position paper). This is because training data is extremely large and often not public (DOI: 10.18653/v1/2024.findings-acl.716).

Even so, there are measurements. When examining more than 15 models across six widely used multiple-choice tests, the degree of contamination ranged from 1% to 45% depending on the test. Contamination also inflated scores. In two of the contaminated tests, accuracy was inflated by up to 14% and 7% respectively, while in another test it only rose slightly. Contamination grew rapidly between 2020 and 2023, and larger models benefited more from contamination (DOI: 10.18653/v1/2024.findings-emnlp.30).

There is more to the story of the arithmetic test mentioned earlier. The drop in accuracy on the new problems varied by model. Models with a higher likelihood of generating existing test questions verbatim tended to show a larger score gap between the two tests, and several model families showed traces of being tailored to the existing test across almost all sizes. This means some models may have partially memorized the existing test.

Conversely, many models, including the most advanced ones, showed few such traces. Even on new problems guaranteed not to be in the training data, all models generally showed the ability to solve problems they were seeing for the first time (DOI: 10.48550/arxiv.2405.00332). Therefore, the drop at the beginning appeared large in only some models, and it does not mean that all models' scores came from memorization. A benchmark score is simply a score on that test, and capability on problems seen for the first time must be measured separately with new problems.

Add One More Grader

Let us say it is Monday morning, and after handing over last week's sales table, you assigned a chatbot to draft a meeting document. The draft is smooth. Even back in 2024 when three open models were tested, smooth prose translated from tables often harbored meaning errors (DOI: 10.18653/v1/2024.acl-long.651). Whether that exact proportion holds for the chatbot you are using today is unknown. Therefore, the places to check in this scene are the numbers and sources in the draft, and checking means comparing it line by line against the table.

If a company brings in such a model to evaluate applicants or employees, the adopting side has work to do. That work includes breaking down results by group, re-testing with the company's own data, and keeping watch while the model is in use. These are methods recommended to institutions by a review on the fairness of medical AI, principles that can be carried over and applied to other settings where people are evaluated (PMID 42466917). If a model boasts a high benchmark score, whether there are results measured again with new problems is the place to check (DOI: 10.48550/arxiv.2405.00332).

These checking places are sometimes actually empty. In 2023, 65% of hospitals that responded to a survey by the American Hospital Association used predictive models to forecast future risks and the like. Among those hospitals using predictive models, 61% checked their accuracy using data from their own hospital or affiliated health system. Only 44% checked for bias (DOI: 10.1377/hlthaff.2024.00842). This means that more than half of the hospitals using the model did not report checking for bias with their own data.

There are tilts on the human side as well. The leaning of placing excessive trust in AI's answers and the phenomenon of becoming numb due to frequent false alarms are cited as risks in settings where AI and humans work together (PMID 42466917).

Designs that incorporated checking showed an effect. In studies of medical systems using text-generating AI, inserting a human verification step inside the system reduced hallucinations, with the largest reduction reaching 95%. However, concerns followed that it was difficult to scale up, and approaches combining multiple methods generally outperformed those using just one. The authors conclude that technological devices and human safety measures must be used together (PMID 42251377).

There is also the competence captured in scores. In elementary arithmetic, models did not suffer major score collapses even on problems they were seeing for the first time (DOI: 10.48550/arxiv.2405.00332). If you know the ways AI gets things wrong, you also know where to check. For a test-taker who has stepped out of the exam room, every way of getting things wrong requires a separate grading spot, and both devices and humans enter those spots.

Where to Look Next

• A deeper look into why hallucinations happen: "Why LLMs 'Lie'"
• How much hallucinations are reduced when related documents are attached: "Does Retrieval Cure Hallucination?"
• What actually went up when benchmark scores went up: "The Score Went Up. But What Went Up?"
• The bias that arises when even grading is left to AI: "The Grader Is Also a Language Model"
• When a sentence hidden in the text read by the model tricks it: "When Someone Else's Sentence Becomes Your Instruction — Prompt Injection"
• The debate surrounding whether the model understands something: "Does AI Really 'Think'?"
• The difference between what is memorized and what is learned: "AI Models", another part of this series
• The method of choosing the next word by probability to generate an answer: "How Generative AI Answers", another part of this series

Source Papers

Any preprints published on arXiv are marked as such (whether they were later published in conferences or journals was not verified in this list). Two papers with only a first edition (v1) are indicated as preprints not subjected to peer review.

• "Current limitations to identify covid-19 using artificial intelligence with chest x-ray imaging (part ii). The shortcut learning problem." (2021, review) — PMID 34660166 — Chest X-ray COVID-19 classification models have reported sensitivities up to 100%, but the region contributing most to the classification lay outside the lungs and performance dropped significantly on external data. Existing models often fall into shortcut learning, making them less suitable for clinical use.
• "Shortcut Learning in Deep Neural Networks" (2020, preprint and perspective paper published on arXiv) — arXiv:2004.07780v5 — Shortcuts are decision rules that work well on standard benchmarks but fail to transfer to harder conditions like real-world settings. Similar phenomena exist in comparative psychology and education, suggesting this may be a common trait of learning systems.
• "Beyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data-to-Text Generation" (2024) — DOI: 10.18653/v1/2024.acl-long.651 — Open models (Llama 2, Mistral, Zephyr) write fluent text, but over 80% of their outputs contained one or more semantic errors (human evaluation and GPT-4-based metrics).
• "Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models" (2025, survey) — DOI: 10.1162/coli.a.16 — Organizes hallucinations into misalignment with user input, contradiction with prior context, and inconsistency with world knowledge.
• "Mitigating hallucinations in healthcare AI: a systematic review of evidence-based strategies." (2026, systematic review) — PMID 42251377 — Analyzed 44 papers. Retrieval augmentation reduced hallucinations by 30 to 50%, and human intervention reduced them by up to 95%, but raised scalability concerns. Combined approaches (retrieval plus verification) were generally better than single methods. Fabricated citations and similar issues are examples of clinical failures. Combining technical and human safeguards is necessary.
• "A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models" (2024, not peer-reviewed preprint) — DOI: 10.48550/arxiv.2401.01313 — Jumping to conclusions from training data biases, misreading ambiguous questions, and superficial matching can lead to hallucinations. Over 32 mitigation techniques are categorized.
• "Why Language Models Hallucinate" (2025, not peer-reviewed preprint) — arXiv:2509.04664v1 — Argues that hallucinations occur statistically like binary classification errors when models fail to distinguish wrong sentences from facts during pretraining, and persist because evaluation rewards guessing over admitting uncertainty. They persist even in state-of-the-art systems. Proposes fixing the scoring methods of existing benchmarks.
• "Implications of AI (un-)fairness in higher education admissions" (2020) — DOI: 10.1145/3351095.3372867 — In a survey of 304 students at a German university, algorithmic decisions were rated higher than human decisions in both procedural and distributive fairness. Summarizes criticisms that algorithms can reproduce biases in training data.
• "Representation Bias in Data: A Survey on Identification and Resolution Techniques" (2023, survey) — DOI: 10.1145/3588433 — Social data often fails to adequately represent minority groups, due to causes ranging from historical discrimination to selection and sampling bias. "Bias in, bias out."
• "Artificial Intelligence-Driven Triage in Pediatric Emergency Departments: Accuracy, Bias, and Impact on Clinical Outcomes: A Narrative Review." (2026, narrative review) — PMID 42137483 — Despite high overall accuracy, performance varies across pediatric subgroups, and the risk of amplifying bias was not sufficiently addressed.
• "Bias Mitigation for Machine Learning Classifiers: A Comprehensive Survey" (2023, survey) — DOI: 10.1145/3631326 — Categorizes 341 bias mitigation papers into preprocessing, in-processing, and post-processing.
• "A Survey on Bias and Fairness in Machine Learning" (2021, survey) — DOI: 10.1145/3457607 — Bias examples in real-world applications, sources of bias, and a taxonomy of fairness definitions.
• "Bias and Fairness Across the Healthcare AI Lifecycle: A Clinician-Oriented Review." (2026, narrative review) — PMID 42466917 — Shortcut learning, dataset shift, automation bias, and alert fatigue are mechanisms of unfairness. Recommends subgroup evaluations and similar measures. Fairness is not guaranteed by a single metric or one-time validation, and continuous monitoring is required.
• "Dataset Shift in Machine Learning" (2008, edited volume) — DOI: 10.7551/mitpress/9780262170055.001.0001 — Dataset shift, where the distributions of training and testing differ, occurs in most real-world applications. Example: A spam filter may fail to recognize spam whose format differs from what it learned.
• "Return of Frustratingly Easy Domain Adaptation" (2016) — DOI: 10.1609/aaai.v30i1.10306 — Traditional machine learning often fails to handle shifts in training and testing input distributions, and such domain shifts drastically degrade performance. A simple method (CORAL) reduces shift and yields superior results on standard benchmarks.
• "Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift" (2019, preprint edition) — DOI: 10.48550/arxiv.1906.02530 — Traditional post-hoc calibration fell short under dataset shift, while some methods spanning multiple models were robust.
• "Intriguing properties of neural networks" (2013, preprint edition) — arXiv:1312.6199v4 — Imperceptible changes found by maximizing prediction error can cause images to be misclassified, and the same changes fool other networks trained on different subsets.
• "Explaining and Harnessing Adversarial Examples" (2014, preprint edition) — DOI: 10.48550/arxiv.1412.6572 — Models give confident wrong answers to inputs with small, intentionally worst-case perturbations.
• "Towards Deep Learning Models Resistant to Adversarial Attacks" (2017, preprint edition) — DOI: 10.48550/arxiv.1706.06083 — Robust optimization substantially increases resistance to a wide range of attacks. Frames the recent discovery that adversarial attacks may be an inherent weakness of deep learning models as a problem, and views robustness against well-defined attacks as a stepping stone toward fully resistant models.
• "Adversarial Examples Are Not Easily Detected" (2017) — DOI: 10.1145/3128572.3140444 — All 10 proposed detection methods were neutralized by new loss functions.
• "Towards Evaluating the Robustness of Neural Networks" (2017) — DOI: 10.1109/sp.2017.49 — Defensive distillation, which reportedly lowered attack success rates from 95% to 0.5%, failed to significantly increase robustness against three new attacks (breaking through 100% on both distilled and non-distilled models).
• "A Careful Examination of Large Language Model Performance on Grade School Arithmetic" (2024, preprint edition) — DOI: 10.48550/arxiv.2405.00332 — Up to an 8% drop in accuracy on GSM8k and newly matched problems GSM1k, with traces of systematic overfitting in some model families. However, frontier models show fewer traces and all models generally generalized to new problems.
• "Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models" (2024) — DOI: 10.18653/v1/2024.findings-acl.716 — LLM performance claims are supported by public benchmarks, but contamination is difficult to detect due to massive and opaque training data.
• "NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark" (2023, position paper) — DOI: 10.18653/v1/2023.findings-emnlp.722 — Contamination causes performance overestimation and can lead to incorrect scientific conclusions, and its magnitude is not easy to measure.
• "An Open-Source Data Contamination Report for Large Language Models" (2024) — DOI: 10.18653/v1/2024.findings-emnlp.30 — Across 15+ models and 6 multiple-choice benchmarks, contamination was 1-45%, inflating C-Eval and HellaSwag accuracy by up to 14% and 7%, with a minor effect on MMLU. Surged from 2020 to 2023, with larger models benefiting more.
• "Current Use And Evaluation Of Artificial Intelligence And Predictive Models In US Hospitals" (2025) — DOI: 10.1377/hlthaff.2024.00842 — In 2023, 65% of US hospitals used predictive models, among which 61% evaluated accuracy using their own data and 44% evaluated bias.