Not the Model I Signed Off On — When Yesterday's Verified Behavior Changes Today

Sent the Same Sentence Twice

We sent the same sentence twice. The rate at which the exact same answer came back both times was 22.1%.

This number came from 4,104 controlled trials run with eight models across five commercial APIs. An API is a window through which a program calls a model living on a company's server from across the internet. The conditions were exactly what the provider documentation wrote down as "fixing the answer like this." First, we turned the temperature down to 0. In the words of the primer "How Generative AI Answers — A Sentence Built One Piece at a Time", temperature is a setting that determines how boldly the language model will pick even low-probability candidates when choosing the next token (a piece of text). If you set it to 0, it picks only the candidate with the highest probability every time. On top of this, we pinned the seed value to be the same. A seed value is the starting number for a calculation that mimics a lottery, and giving the same seed ensures the same lottery sequence comes out. The primer wrote that turning off the lottery like this makes nearly the same answer come out for the same request. According to the documentation, the answer should be the same every time.

The ones that were the same were about one in five times. On the other hand, for models installed and run directly on one's own computer, meaning local execution, 95.6% were the same. The abstract does not disclose whether the two sides are the same model. That is a gap of more than fourfold. This wobbling remained even in conversations sent and received multiple times, or in the method of looking up and attaching documents. One of the models tested did not produce a completely identical answer even once during 50 runs (DOI: 10.21203/rs.3.rs-9096283/v1, preprint). The 22.1% earlier is a bundled value for eight models, and this 0 times is the value for one of them. The abstract does not compare the two. The property where the answer changes when you put the same input in twice with the same settings is called non-determinism.

The strange part is what comes next. Those answers where all 50 were different were almost identical in meaning. The score measuring how similar the meaning was exceeded 0.97. The answer we just checked and the answer that came next have different words, but read the same. Therefore, to the user's eyes, a signal that it is wobbling does not appear on the screen. The same research team also conducted an experiment narrowing down the cause. The pointed-to cause was not the fact itself that it "runs on a server above the clouds," but the complexity of the operational infrastructure loaded onto that server. However, this experiment is also a preprint, meaning a manuscript released before undergoing peer review. Among the research gathered by this article, none measured this gap again at the same scale. A significant portion of the research appearing in this article is preprints, so we attach a grade to each citation.

This article takes over the question handed down by "The Score Went Up. But What Went Up?". That article asked whether the same value comes out if the same model is measured again with the same benchmark, and reproducibility when a model of the same name changes over time was handed over to this article. Reproducibility is the property where the same result comes out when the same procedure is walked through again as is. The earlier experiment is a report that even if you fix the settings as the provider documentation wrote down, this property wobbles. This article starts from there and goes all the way to the place where the model itself changes.

The sibling pieces follow where the work entrusted to language models goes wrong. "When Someone Else's Sentence Becomes Your Instruction", "AI Coding Productivity", "Does Retrieval Cure Hallucination?", "Multiplication Won't Tell You", and "The Grader Is Also a Language Model" are those places. What this article looks at is the premise of entrusting itself. In the saying that you entrust something, there is one silent premise. That the counterpart you tested and chose continues to be that counterpart.

Why Two Are Different — Addition Order and Someone Else's Request

If you add a very small number to a large number first, the small side gets swallowed by rounding and disappears. If you gather and add the small numbers together first, they survive. That is why when a computer adds numbers with decimals, the order changes the result. In mathematics it is the same value, but in a machine it is not. Every time a language model picks a token, it does this kind of addition hundreds of millions of times. If the addition order changes, the scores of two candidates can flip somewhere below the decimal point. Once they flip, the subsequent sentences split down entirely different paths.

The reason addition order changes lies in the way servers work. A server—meaning the computer on the company side that runs the model and sends back the answer—does not process requests one by one. It calculates the requests that arrived at that moment as a single batch. When the batch size changes, the order in which calculations are split and combined also changes. Ultimately, the answer you receive is tied, little by little, to how many requests from other people came in together at that time. The property that decimal addition is sensitive to order and the fact that batch size changes that order have been pointed out as the system-side roots of non-determinism. Turning off batching fixes the answer, but the number of requests processed in the same amount of time drops drastically. If you redesign the underlying layer to calculate regardless of batch size, a cost is always attached even to requests that do not need fixing. Therefore, the authors propose generating the answer through the fast path, verifying it with a fixed order, and rolling back only the mismatches (arXiv:2601.17768, preprint).

Up to this point, this is fluctuation that occurs even when the model stays the same. The server software that carries the model is also continuously updated. As optimizations are applied rapidly, failures where quality quietly worsens while not a single error indication pops up can get mixed in. There is research that built a system to automatically find the causes of such failures. They created a test set using failure cases gathered from actual serving software, and the rate of guessing the cause on the first try was 80%, while the rate within five tries was 88%. However, the abstract merely records the comparison target as "the previous state-of-the-art system" without disclosing the figures for it (arXiv:2606.04594, preprint).

And the Model Itself Changes

The value measured in March 2023 was different from the one measured in June. This is the result of re-measuring a model serviced under the same name with the same problems at a three-month interval. On a problem of identifying prime numbers, GPT-4 got 84% right in March and 51% in June. The name and the problems remained the same.

Two caveats need to be attached to this study. First, the results went in both directions. GPT-3.5 did better in June than in March on that exact prime number problem. On problems requiring multiple steps to find an answer, GPT-4 did better in June while GPT-3.5 dropped. On sensitive questions and opinion survey questions, GPT-4 was less willing to answer in June. In code generation, both showed an increase in formatting errors in June. The authors pointed to GPT-4's reduced ability to follow user instructions as one common factor behind the various changes. Second, the models appearing here are from the 2023 generation. Since the thesis of this article is that "the board changes," the same yardstick applies to this evidence. This result is an observation of that board at that point in time, not a diagnosis of any model currently in service. This article uses model names only where which board it is affects the argument, following the abstract as written, and omits the names in sections comparing performance, writing only "a certain API." This is because the topic is not product comparison, but rather how the same name changes over time. The authors' conclusion was also closer to a call to keep monitoring because the behavior of the "same" service can change significantly in a short period, rather than a judgment on a specific product (arXiv:2307.09009, preprint).

This problem existed even before language models. Commercial prediction APIs were also changing without a word. When commercial prediction APIs from major companies were re-measured twice with the same data across 2020 and 2021, clear changes were confirmed in 12 out of 36 combinations. Among them was data where predictions became significantly worse over time. To detect changes, you have to repeatedly call the API, and every single one of those calls costs money. Therefore, the authors devised a separate method to reach the same estimation with up to 90% fewer samples than randomly drawing and measuring (arXiv:2107.14203, preprint).

If things can change, it seems like we could just attach precise name tags, and such name tags actually exist. A model snapshot is a name tag that attaches even the date behind the product name, nailing down which version hardened on which day. It is written in the format of not just "that company's that model," but "that model's August 6 edition of that year." The remaining question is how much this name tag is actually used.

By the Time a Paper Comes Out, That Model Is Already an Old Version

There is a survey analyzing 48 papers filtered from 179 oncology language model papers published in PubMed up to December 31, 2024. In roughly one out of every four cases, the exact model that was the core of the study had already been retired or a newer version had come out by the time the paper was published. This was the case even though the time taken from submission to release had a median of 3.7 months (interquartile range 2.5 to 5.9 months), which is fairly short for academic publishing. However, this survey is a conference abstract rather than a formal paper. It is thinner evidence than a preprint, and among the studies gathered in this article, there was no other survey to replace this tally.

Other numbers from the same survey show the background behind this 25%. While almost all of them (97.9%) stated the model family, only 27% stated a version stamped with a date (a snapshot). Only 22.9% disclosed core settings like the temperature, and among those, 45.4% set the temperature higher than 0. They measured it with the knob left open so that the answer changes every time. Original code was made public in 18.7% of the papers that used a programming language, and 35.4% were simply results asked by hand in a chat interface. The checklist put forward by the authors includes items like these. It instructs to specify date-stamped versions, record settings tied to that version rather than "latest", use APIs instead of chat interfaces, set the temperature to 0, run multiple times to check for variation, and be careful when targeting models that will soon be retired (DOI: 10.1158/1557-3265.aimachine-b021, 학회 발표 초록).

These numbers point to a problem one step ahead of reproducibility. In roughly one out of four papers, the target to attempt reproduction with had already been retired or become an old version. Even if you write down the steps meticulously, you cannot retrace them if the counterpart those steps called no longer exists.

The phenomenon where reported numbers cannot be reproduced appears outside of medicine as well. However, the cause in this case was not that the model disappeared, but that the models still in existence were confused with different versions or configured differently. There is a study that delved into why reported numbers do not match across papers in benchmarks for fixing code bugs—in other words, collections of test questions that make multiple models solve the same problem and compare their scores. It tried to recreate 35 results reported in the literature using 12 models, and 12 succeeded. The causes were mundane. Confusing a version trained one more time with examples pairing instructions and answers—that is, a version that has undergone instruction tuning ("How Guessing the Next Word Becomes a Conversation")—with the base model before it, giving different prompt formats or generation lengths, and crudely reducing the model numbers to 4 bits to run it lightly changed the results. When instead of picking only the candidate with the highest probability, they let it draw according to probability (sampling), the variance grew (DOI: 10.3390/software4030017). When a model changes, it is not just the model that changes. The instructions polished to fit that model become outdated together.

Even Prompts Go Out of Date

A prompt is the text you put in front of a model to get an answer ("How Guessing the Next Word Becomes a Conversation"). There is a study that re-tested five prompt-writing techniques across six models. The researchers paired older and newer versions from the same families to make three pairs, and generated 19,620 pieces of code using 218 Python functions, grading them on whether they actually ran. The results did not all point in the same direction. In the newer GPT versions, the benefits of instructions that impose structure shrank or turned negative. These are instructions like few-shot prompting, which attaches a few examples with correct answers, or chain-of-thought prompting, which asks the model to write out intermediate steps as it answers ("Does Prompting Actually Work? — The Science of Asking AI Well"). The authors interpreted this as those scaffolds having been absorbed into the model itself. On the other hand, the Qwen family continued to see substantial benefits from few-shot prompting and contrastive examples (chain-of-thought showing correct and incorrect solutions side by side), while in the Mistral family, the benefit of one technique held steady while the other two shrank. The authors' conclusion stands on this limitation: the ways in which prompts go out of date differ by model family, meaning prompt strategies must be re-tuned to the specific family and generation and cannot simply be carried over and reused (arXiv:2608.24641, preprint).

There are even cases where the sign flips entirely. The research team that created a prompt method heavily packed with rules tested that method across three generations of models from the same company. On 1,317 elementary math problems, the middle-generation model performed better with that method than with standard chain-of-thought (97% versus 93%). But in the newest generation tested, it flipped. The same method scored 94.00%, while chain-of-thought scored 96.36%. The authors called this a transition where help turns into hindrance. Constraints that used to prevent clumsy common-sense missteps become shackles that force a better model to follow instructions too literally. This result is unfavorable to the authors because it reports that their own method loses in the new generation (arXiv:2510.22251, preprint).

Things wobble even without changing the technique as a whole. That holds true even if you change only the formatting while leaving the meaning intact. As seen in experiments in "Does Prompting Actually Work?", the researchers measured accuracy while varying only surface elements like delimiters or line breaks in a few-shot prompting setup. On one open-source model from the 2023 generation, accuracy varied by up to 76 percentage points. This sensitivity remained even after scaling up the model, adding more examples, or applying instruction tuning. And the tendency for formatting that worked well on one model to also work well on other models was weak. This means there is no guarantee that the optimal formatting chosen for Model A will be optimal for Model B either. The authors question the validity of the practice of locking in a single piece of formatting arbitrarily and then comparing models (arXiv:2310.11324, preprint).

One property reveals itself here. Prompt techniques are attached to the model rather than the task. What I polished is less "how to solve this task" and more "how to pull this specific model of this generation toward this task." When the counterpart changes, part of that effort becomes a liability.

The Average Stays the Same, But Individual Cases Flip

When a new version comes out, the total score usually goes up. That is why people upgrade, and it is usually the right call. The problem lies in the fact that the total score going up and the specific case you were using continuing to work are two different things (The Score Went Up. But What Went Up?).

The phenomenon where a new version's overall score goes up, but individual cases that the old version got right end up wrong, is called regression. To measure regression, you have to pair items up by question rather than looking at the total score. You count how many items the old version got right but the new version got wrong. When this ruler was applied to various tasks on natural language processing testbeds, this flipping was found to be spread across tasks (arXiv:2105.03048, preprint), and the same regression was measured in tasks that extract sentence structures whole (arXiv:2202.02976, preprint).

The same goes for language models. When the underlying base model is updated, the task-specific components built on top of it start getting things wrong that they previously got right, even though not a single step of the training procedure was changed. The authors who analyzed this phenomenon believe this goes beyond the problem of a single performance number. It shakes the user's mental map of "this model can do this and cannot do that." The authors report that they reduced the flipping by up to 40% by training compatibility components separately, but the abstract does not reveal the rate before the reduction (arXiv:2407.09435, preprint).

Regression stays out of sight because the total score masks it. Examples from outside language models show this clearly. In systems where a decision immediately leads to action—such as semiconductor quality control or engine failure prediction—the model type, training data, and settings were all kept the same, and only the method of cleaning inputs was changed to a different method with the same amount of information. The rate at which the final decision changed ranged from under 0.2% to over 30% depending on the data. Among them were cases where predictions flipped even though the prediction accuracy remained almost identical. The authors propose managing this rate as a separate metric (DOI: 10.36227/techrxiv.177220377.73247298/v1, preprint).

Upgrading to a new version feels like it would reduce failures, but failures can also shift into different shapes. There is a study that lined up models from the same family that differed only in training volume, size, and whether they underwent instruction tuning, and observed what they do when their confidence drops. The axis this study measured was not version replacement, but those three factors. Therefore, it applies to version replacement only under the premise that these axes move together when versions change. The order was consistent. Less refined models repeat the same words, getting slightly better yields sentences with broken meanings, and getting the best results in plausible fabrications. Methods that sample randomly according to probabilities reduce obvious symptoms like repetition, but increase hard-to-detect hallucinations (arXiv:2407.06071, preprint). It is difficult to measure improved spots and failures that shifted to less visible sides with the same scale.

If You Measure It, It Disagrees

There is a study that had 2023-generation commercial chatbots run the triage procedure for patients at a disaster site. Five disaster medicine specialists created nine prompts, and paired each prompt with 391 simulated patient cases, asking ten times each. This comes to a total of 35,190 times. Among them, 99.7% yielded answers that fit the format. The problem was the variation among those answers.

When breaking down the overall variation, the share caused by repeating the exact same prompt as-is was 14%, while the share caused by changing the prompt was 4.1%. This means that the difference from sending the same instruction twice was greater than the difference between the nine directives written differently by the specialists. The overall accuracy was 63.9%, and ranged from 46.7% to 71.8% depending on the prompt. Cases classified as more urgent than reality accounted for 32.9%, and those classified as less urgent accounted for 3.1%. The authors' conclusion was that the tested version was insufficient for this use case, and that such tools must be rigorously evaluated by techniques that divide variation by source before being brought into the clinic (DOI: 10.2196/preprints.55648, preprint).

The property of being invisible to the user's eyes repeats here as well. There is an experiment where 13 food photos were fed into four commercial vision APIs over 500 times each to have them estimate carbohydrate amounts. The instructions were the same, the temperature was lowered down to 0.01, and the total was 26,904 times. The median of how scattered the estimated values for the same photo were compared to the average (the coefficient of variation) ranged from 2.4% to 11.0% depending on the API. Translated into insulin doses, this is uncertainty between 0.9 units and 4.7 units. In five photos with clear reference values, while one API never once produced an answer that would lead to an overdose exceeding 2 units, the other APIs produced such answers 12% to 37% of the time. The risks pointed out by the authors are two: a systematic bias toward over-calling, and random wobbling within the same photo. The closing remark in the conclusion was that both are invisible to the end user. The authors recommend multi-query aggregation, uncertainty notification, and user verification before using the values (DOI: 10.2139/ssrn.6577780, preprint. The tested models and versions were those in service at the time of that study).

When there is wobbling, you cannot clearly see changes either. There is a study that assigned the task of measuring the maximum diameter of lesions in CT images to three language models and diagnostic radiologists. Agreement was measured using an index where values closer to 1 mean better matches. The self-agreement of a radiologist measured again after a week or more was 0.98. The models turned out as 0.78, 0.47, and 0.25 when asked twice with an interval of a week or more. What to look at in these numbers is the identity rather than the size. When two answers disagree, this design cannot distinguish whether it is because the original wobbles every time or because the model changed in the meantime. The conclusion of this study was that language models do not yet reach the measurements of specialists. At the same time, it added that the good agreement and reproducibility shown by one model demonstrate rapid progress in image interpretation capabilities (DOI: 10.1007/s11604-025-01884-5).

The same thing happens in tasks that do not require creativity. There is a study that attempted in three ways the task of extracting immunogenicity figures for 50 antibody drugs from regulatory documents and moving them into a table. The approach of attaching search and the approach of having the model read the PDF as a whole could not be used due to errors of pasting another drug's figures into this drug's spot and fabrication. The approach of first changing the document into a neatly organized summary caught those errors and extracted core figures accurately. However, even when setting values were fixed and the exact same neatly organized input was fed, the shape of the table changed slightly every day, and context information occasionally dropped out entirely. The authors' conclusion was that while this method greatly accelerates initial data collection, meticulous validation must be attached because reproducibility is not guaranteed (DOI: 10.70534/qm-2026-0003).

But They Don't Always Waver

For the task of grading 51 reflection essays using a six-step rubric, the results were different. The essays consisted of 15 actually written by medical students and 36 fabricated for research purposes. The research team ran 29 combinations of prompts and models. They toggled whether or not to give the model a persona to adopt, whether or not to include examples, whether or not to have it think step-by-step, changed the temperature, and swapped between four models. Agreement with human graders was high in 28 out of 29 conditions, and agreement when re-running under the same conditions was high in 28 out of 29 conditions. Contrary to expectations, conditions that changed step-by-step thinking, temperature, and personas produced no significant difference in accuracy compared to the baseline prompt. However, the abstract reports that the same abstract was more accurate in the condition with examples attached, and that accuracy varied depending on how many steps the grading criteria were spelled out into (DOI: 10.64898/2026.03.20.26348918, preprint).

There is also a study that had three models answer mock cases for a breast tumor board, which is a meeting where multiple doctors gather to decide on treatment. Two of the three are different versions of the same product. This study reports that differences among models were not significant across five evaluation domains. However, the scale was small with 90 evaluations across five cases, and inter-rater agreement among graders remained at a moderate to good level. English responses received higher scores than Turkish responses, but the abstract does not attach testing results to this difference. The only thing the abstract notes as significant was that Turkish readability varied depending on the model (p=0.015) (DOI: 10.2139/ssrn.7151327, preprint).

Therefore, the claim of this article is not that "models are different every time." Some tasks waver significantly, while others barely waver at all. The problem is that you can only know which one it is by testing it, and how you test it changes that answer.

There are cases where how you test changes the answer. This is a study that gave symptom scenarios to a fictional ordinary person persona and had them choose their next action. In a pilot experiment where two sets of prompts were run side-by-side in the same batch, all 40 pairs matched completely. There was not a single discrepancy. However, in a separate audit where 24 pairs were separated and run apart, the rate of discrepancy fluctuated from 0% to 45.8% with every repetition. The authors' interpretation is that a design where pairs are run together all at once can severely underestimate the differences when prompts are changed. In a subsequent controlled experiment, lowering the temperature to 0 increased stability across repetitions in three out of four conditions. But the discrepancy when prompts were changed remained high whether the temperature was set to 0 or left at the default (DOI: 10.32388/be0zbc.3, preprint — published on an open review platform).

And there is one more thing. Reproducibility and accuracy are different properties. There is a study that had one commercial generative AI and eight dentists perform the task of classifying 100 dental radiographs as a screening indicator for osteoporosis. The agreement index $\kappa$ is a complete match at 1, and near 0 for chance-level. When asked again with an interval of more than two weeks, the AI agreed with itself almost perfectly ($\kappa=0.987$), and was about six times faster. However, agreement with the dentists was a $\kappa$ of less than 0.2, described in the abstract as "slight agreement or lower." Cases where normal and severe were swapped exceeded 10%. Over the same period, the dentists showed a learning effect where agreement with each other improved in the second round. The authors recommend a hybrid model where AI is used as a standardized feedback tool and humans make the final diagnosis (DOI: 10.20944/preprints202605.0011.v1, preprint). Not wavering does not mean being correct. It can remain fixed while being wrong.

What Can Be Fixed

There are three layers of things we can hold on to.

The first layer is nailing down name tags. Version pinning means choosing a specific version stamped with a date instead of using the "latest" one, and binding your work to use only that version. This is the first item on the checklist we saw earlier. However, what this handle buys you is time. Among the 25% of cases we looked at earlier, the only thing that neutralizes version pinning is retirement. If all that happened was the release of a newer version, the pinned version is still alive. Since the abstract reports both cases combined, we can only say that the rate at which version pinning was neutralized by the time the paper came out does not exceed 25%. This survey did not measure the rate after more time passed. Version pinning buys time, not permanence.

The second layer goes further. It pushes the shaky parts outside the model. A benchmark where a language model-based conversational service was made to create the random allocation sequence for a clinical trial shows this contrast cleanly. Running 1,200 times with Python code containing a pinned seed yielded a 100% exact match reproducibility. When the same investigators asked the model only in words without code or a designated seed for 320 runs, allocation integrity, format, and completion rates were all 100%, but exact match reproducibility was 0%. The authors insist that this result should be interpreted narrowly. Achieving reproducibility with conversational tools requires executable code, a fixed seed, logs, and human verification, and such tools must not replace a validated allocation system (DOI: 10.3390/a19070551).

A weaker version of the same idea is forcing the shape of the output. In a sarcoma tumor board simulation, answers were made to fill in a framework embedded with 21 decision codes, and the temperature was set to 0. When 51 cases were run three times each, 93.9% of the decision codes were identical across all three runs. The percentage of cases where every single code matched was 33.3%, and by clinical domain, it ranged from 88.9% to 98.0%. There were no instances of hallucinated structures across all 154 runs. However, the abstract does not disclose the reproducibility rate before the template was applied. Thus, it cannot be distinguished from the value that would have come from a temperature of 0 alone. The interpretation added by the authors is also important. The least solidified decision codes (74.5%) were slots with clinically interchangeable options. This means that wobbling is not entirely an error (DOI: 10.21203/rs.3.rs-10035573/v1, preprint, single-center pilot study). One design dealing with legal documents established this policy as a principle. Uncertainty is funneled exclusively into interpreting the intent of the question and the final narrative, while hierarchy, timing, and causal lookups of legal provisions are handled by deterministic operations that always produce the same result for the same input. However, the authors themselves note that this is a structural specification, not an empirical benchmark (arXiv:2510.06002, preprint).

The third layer is out of my hands. Backward compatibility is the property where a new version keeps things that worked in the old version working. Techniques to secure this already form a lineage. There is a method of mixing the parameters of old and new versions (arXiv:2301.10546, preprint), and a method of training features extracted by the new model so they can be compared directly with those of the old model, eliminating the need for recalculation (arXiv:2003.11942, preprint). There is also a method of attaching a conversion layer so that user teams do not need to retrain even when the creator team updates the version (arXiv:2206.03040, preprint). In 2026, a method emerged that suppresses not just predictions, but even the explanations that change after an update (arXiv:2608.08674, preprint). All of these techniques tinker with parameters and the training process. In short, backward compatibility is a handle held by the seller. People who use closed models—where parameters are not made public and the model is accessed only via an API—cannot apply any of these techniques themselves and can only demand backward compatibility.

What Cannot Be Fixed, and What Comes Next

On the evaluation side, a proposal has emerged to treat evaluation scores as claims with expiration dates (as also seen in "The Score Went Up. What Went Up?"). The suggestion is to record three things together with every score: what kind of evaluation produced it (such as human evaluation ranking as stronger evidence than automated metrics), what scope it applies to, and when it expires. This is because results spoil as test questions leak into training data and distributions shift. The example used to support this proposal is incisive: when 54 models from a public leaderboard were re-evaluated across ten scenarios, comparing the top five picked by average score against the top five picked by the weakest signal showed that the two lists did not overlap at all (arXiv:2607.26191, preprint, position paper).

Even after doing all that, something remains. The eight-week trace record of a single personal assistant agent operating in the wild demonstrates this. An agent is a language model that does not stop at generating words, but also wields tools to read the web, open files, and execute commands ("Multiplication Won't Tell You"). This record attached a root-cause analysis to every 22 incidents. Roughly 70% of silent breakdowns—where error signals did not reach a human—were discovered by humans looking at the screen, rather than by tests or audits. Tracing back 15 incidents and auditing them showed that the rate of pre-emptive prevention was 0%, while the rate of preventing a once-experienced breakdown from recurring was 87%. The authors' summary is this: audits are not predictive devices, but regression-blocking devices. The place where incidents survived the longest was the seams between components, where no tests run (arXiv:2606.14589, preprint. This is an eight-week record on a single production system).

Taking a step back, it can be summarized as follows: in systems that change on their own, a gap opens up between writing things down transparently and being able to retrace them. No matter how faithfully procedures are disclosed, if the counterpart those procedures call upon changes, reproduction fails. A review pointing out this gap proposes positioning reproducibility not as a gateway that ends once passed, but as a property that must be remeasured for as long as the service lives (DOI: 10.20944/preprints202606.2079.v1, preprint, narrative review).

Among the studies gathered in this article, none confirmed what changed and how inside commercial models using provider-side records. The fact that changes occurred was discovered by dividing the arena to remeasure from the outside, or by comparing points in time when the arena was discarded and replaced. There are also reports that performance and stability are different axes. When variations of instructions carrying the same meaning were fed into 14 code generation models across three public families, being good and being consistent generally moved independently (arXiv:2509.13680, preprint). At least within this range, models with higher scores were not necessarily more uniform models.

Therefore, the premise of delegation must be rewritten. The guarantee that the counterpart I tested and chose remains that same counterpart is divided into the share I can produce and the share I must demand of the counterpart. Shifting shaky tasks into seeded code, having things filled into fixed templates, and version locking bound to dated releases are my share. Beyond that, when the parameters changed and whether the old behavior is still alive are questions only the seller can answer. While those answers are missing, the sentence "validated yesterday" is not a stamp applied once and for all, but a receipt attached with an expiration date.

Source Papers

• "Same Prompt, Different Answer: … in LLM APIs" (2026, preprint) — DOI: 10.21203/rs.3.rs-9096283/v1 — Temperature 0 and fixed seed tested 4,104 times. API reproducibility 22.1%, local 95.6%.
• "LLM-42: Enabling Determinism in LLM Inference…" (2026, preprint) — arXiv:2601.17768 — The root of non-determinism is floating-point non-associativity and dynamic batching.
• "Ekka: Automated Diagnosis of Silent Errors…" (2026, preprint) — arXiv:2606.04594 — Frequent changes in serving software lead to quality degradation without error signals. Diagnosis 80%/88% (comparison figures not disclosed).
• "How is ChatGPT's behavior changing over time?" (2023, preprint) — arXiv:2307.09009 — Comparing March and June versions. Primality testing for GPT-4 84%→51%, while GPT-3.5 increased. Multi-hop showed the opposite.
• "Did the Model Change? … Assessing Machine Learning API Shifts" (2021, preprint) — arXiv:2107.14203 — Out of 36 commercial prediction APIs, 12 changed, with some significantly worsening.
• "Abstract B021: Current oncological LLM research…" (2025, conference abstract) — DOI: 10.1158/1557-3265.aimachine-b021 — 25% of 48 papers used discontinued or older models by publication. Snapshot reporting 27%.
• "Investigating Reproducibility Challenges in LLM Bugfixing…" (2025) — DOI: 10.3390/software4030017 — Only 12 out of 35 reports in the literature were reproduced. Causes include version confusion, formatting, quantization, and sampling.
• "Aging of Prompt Engineering Techniques Across LLM Versions" (2026, preprint) — arXiv:2608.24641 — 3 generation pairs and 19,620 runs. Newer GPT versions showed reduced or negative structural prompt gains, while Qwen maintained them.
• "…The Prompting Inversion" (2025, preprint) — arXiv:2510.22251 — 1,317 GSM8K questions. Rule-based prompts reversed at 97% (94.00% vs. 96.36%).
• "…Sensitivity to Spurious Features in Prompt Design" (2023, preprint) — arXiv:2310.11324 — A formatting change alone caused a difference of up to 76 points. Effective formats showed weak correlation across models.
• "Regression Bugs Are In Your Model!" (2021, preprint) — arXiv:2105.03048 — Reversals where a new version gets questions wrong that an older version got right are widespread.
• "…Model Update Regression in Structured Prediction" (2022, preprint) — arXiv:2202.02976 — Reassessed the same regression on syntax and semantic analysis tasks and compared mitigation techniques.
• "MUSCLE: A Model Update Strategy for Compatible LLM Evolution" (2024, preprint) — arXiv:2407.09435 — When updating the base model, downstream adapters flip even with the exact same training procedure. 40% reduction (baseline not reported).
• "Quantifying Representation-Induced Decision Drift…" (2026, preprint) — DOI: 10.36227/techrxiv.177220377.73247298/v1 — Changing only the preprocessing flips decisions by over 0.2% to 30% (including cases where accuracy remains virtually unchanged).
• "From Loops to Oops: Fallback Behaviors… Under Uncertainty" (2024, preprint) — arXiv:2407.06071 — As improvement progresses, failures shift from repeated loops to collapsing sentences to fabrication.
• "…ChatGPT to Perform Disaster Triage… START Protocol" (2023, preprint) — DOI: 10.2196/preprints.55648 — 35,190 runs. Repeating the exact prompt accounted for 14% of the variance, and replacement accounted for 4.1%. Accuracy 63.9%.
• "Reproducibility And Accuracy of LLM Vision APIs…" (2026, preprint) — DOI: 10.2139/ssrn.6577780 — 26,904 runs, temperature 0.01. Median coefficient of variation within images 2.4–11.0%. Invisible to users.
• "…LLM measurements of liver metastases" (2025) — DOI: 10.1007/s11604-025-01884-5 — Two measurements a week apart. Humans scored 0.98, models 0.78, 0.47, and 0.25.
• "LLM-Supported Data Collation: Addressing Accuracy and Reproducibility" (2026) — DOI: 10.70534/qm-2026-0003 — Even with fixed settings and inputs, phrasing changes daily and context is occasionally omitted.
• "LLM scoring of medical student reflection essays" (2026, preprint) — DOI: 10.64898/2026.03.20.26348918 — Accuracy and reproducibility ICC exceeded 0.80 in 28 out of 29 conditions. No significant difference across temperature or persona.
• "Cross-Model and Cross-Version Evaluation of LLMs…" (2026, preprint) — DOI: 10.2139/ssrn.7151327 — 5 cases. Differences between models were not significant, while significant differences arose from the response language.
• "Simulating Lay Health-Seeking Behavior with LLM Personas…" (2026, preprint — published on open review platform) — DOI: 10.32388/be0zbc.3 — All 40 cases matched when measured in pairs, but diverged by 0–45.8% when separated.
• "Generative AI vs. Dentists: Reliability and Reproducibility…" (2026, preprint) — DOI: 10.20944/preprints202605.0011.v1 — Self-agreement κ=0.987, agreement with dentists κ<0.2.
• "LLM-Linked Chatbot Platforms for… Clinical Trial Randomization" (2026) — DOI: 10.3390/a19070551 — Code execution with fixed seed achieved 100% reproducibility over 1,200 runs, whereas text-only instructions achieved 0% over 320 runs.
• "Schema-enforced LLM framework for sarcoma tumor board…" (2026, preprint) — DOI: 10.21203/rs.3.rs-10035573/v1 — Schema enforcement combined with temperature 0 achieved 93.9% reproducibility in deterministic code. The lowest-scoring items were clinically interchangeable slots.
• "Deterministic Legal Agents: A Canonical Primitive API…" (2025, preprint) — arXiv:2510.06002 — Isolates uncertainty into intent interpretation and final drafting. The authors explicitly state this is a structural specification rather than empirical validation.
• "Backward Compatibility During Data Updates…" (2023, preprint) — arXiv:2301.10546 — Mixes the weights of older and newer versions to reduce reversals.
• "Towards Backward-Compatible Representation Learning" (2020, preprint) — arXiv:2003.11942 — Trains new features to be directly comparable with old features, eliminating the need for recalculation.
• "Learning Backward Compatible Embeddings" (2022, preprint) — arXiv:2206.03040 — Transformations that allow consuming teams to avoid retraining even when the producing team upgrades versions.
• "Backward Compatibility in Tree-Based Explanations…" (2026, preprint) — arXiv:2608.08674 — Suppresses changes after updates even when what changes is the explanation rather than the prediction.
• "Evaluation Scores Are Perishable Knowledge Claims" (2026, preprint, position paper) — arXiv:2607.26191 — Proposal to attach methods, scope, and expiration dates to scores. The top 5 on leaderboards do not overlap depending on the aggregation method.
• "When Errors Become Narratives: … in a Production LLM Agent Runtime" (2026, preprint) — arXiv:2606.14589 — 1 production instance, 8 weeks, 22 incidents. Roughly 70% were discovered by humans. 0% prevented proactively, 87% prevented from recurring.
• "Beyond AI Transparency: … Governance Framework…" (2026, preprint, descriptive review) — DOI: 10.20944/preprints202606.2079.v1 — Transparent reporting alone does not guarantee the reproducibility of adaptive systems.
• "Prompt Stability in Code LLMs: Measuring Sensitivity…" (2025, preprint) — arXiv:2509.13680 — Performance and stability largely moved independently across 14 models from 3 families.