Multiplication Won't Tell You — Where Agents Break on Long Tasks

Using 9.9 Million Tokens to Score 15.2%

The window where you type commands in text to operate a computer is called a terminal. There is a benchmark that seats AI in front of it and makes it do tasks that take a long time. In the words of "How AI Gets Things Wrong — The Test-Taker Who Stumbles Outside the Exam Hall", a benchmark is a collection of test questions designed to make several AI models solve the same problems so their scores can be compared. The tasks in this benchmark number 46, including reproducing experiments, modifying software, interactive games, and scientific calculations, and partial credit is awarded for subtasks broken down finely from each task.

15 of the top-tier models at the time were attached to this setup. These models are language models, meaning models that take text and output the text that should follow it ("How Generative AI Answers — A Sentence Built One Piece at a Time"). Language models handle text by cutting it into units called tokens, and a token is sometimes a single word and sometimes a piece of a word. A single task required an average of 9.9 million tokens. The number of execution rounds was about 231, and the time taken was 85.3 minutes.

The scores were low. Even the best-performing model scored 15.2% under the criterion where getting a partial score of 0.95 or higher counts as passing, and 10.9% under the criterion where a perfect score is required to pass. The overall averages for all models were 4.3% and 1.7%, respectively (arXiv:2607.08964, preprint).

The starting point of this paper is that AI has reached a point where it can finish short, clearly defined tasks on its own. However, in these tasks that take anywhere from several minutes to several hours to execute, the scores were at this level. Therefore, the sentence this article wants to test is this:

"If you take a single step well, a long task will eventually work out."

First, let's set the terms. An agent is a language model that doesn't stop at generating words, but also holds tools to read the web, open files, and execute commands. When it calls a tool once, the result comes back, and the agent looks at that result to decide what to do next. This article calls a single cycle of deciding once, moving once, and receiving a result a step.

A long-horizon task is a job that doesn't end with a single answer, but requires dozens or hundreds of steps where the result of an earlier step becomes the condition for a later step. The terminal test mentioned earlier is such a job. While the 231 rounds continue, the results obtained earlier determine what needs to be done next.

Before reading, let's make the grading clear. A preprint is a research manuscript made public before undergoing peer review by fellow researchers. Among the 23 items this article cites, 21 are preprints, and only two appeared in peer-reviewed venues. Three out of the four most substantial pillars supporting the conclusion of this article also rest on preprints. Preprints are manuscripts uploaded to open repositories like arXiv, SSRN, and Research Square. Mixed among them are one paper that only describes a design without running experiments yet, and one paper that puts forth a claim and asks the academic community for responses. The grading is noted together in parentheses at every spot supporting a conclusion. Because this field changes quickly, all the numbers appearing here are values observed by that paper at that point in time.

Multiplication is a Prediction, Not an Observation

Let us suppose that a single step has a 95% chance of succeeding. If a task requires taking twenty steps in a row to finish, the overall success rate is 0.95 multiplied twenty times, which is roughly 36%. Even if we raise the accuracy of a single step to 99%, one hundred steps give 37%.

This calculation is most commonly used when explaining why long tasks are difficult. This calculation is arithmetic, not measurement. For multiplication to hold true, the steps must be independent of one another. Independence means that the probability of the third step failing has nothing to do with how the second step went. The numbers above are merely values that emerge when you lay down that premise. Among the studies gathered in this article, none have actually measured and confirmed this curve in a real agent. So there is only one question to ask. When you actually measure it, does this shape appear?

The first measurements we look at show a shape different from multiplication.

We start with results measured in multi-step reasoning. Here, reasoning means drawing a conclusion by stepping through given facts, as seen in 「Does AI Really 'Think'? — A Question Researchers Are Still Fighting Over」. There is a study that mechanically generated six branches of problems—arithmetic chains, symbolic algebra, graph traversal, logical deduction, constraint satisfaction, and compositional reasoning—while adjusting their difficulty, and applied seven top-tier models to 12,000 problems. Accuracy dropped in an S-shaped curve, and performance collapsed the moment it crossed a certain complexity threshold. An S-shaped curve is a shape that holds up almost entirely at first and then drops steeply in a certain section. The location of that threshold differed for each problem branch, and it was steepest in problems that required combining multiple pieces (DOI: 10.21203/rs.3.rs-8251575/v1, preprint).

Three limitations are attached to this result. First, this study measured multi-step reasoning without tools, rather than an agent holding tools. Second, the seven models applied were of the GPT-4o and Claude 3.5 Sonnet generation. The issue that "the primary cause of failure can migrate across model generations," which we will see later, can also apply to this evidence. Third, the horizontal axis of this study is not the number of steps, but a complexity index that bundles chain length, number of branches, and memory load, so it was not compared on the same axis as the multiplication curve. Even so, the gentle curve drawn by multiplication and the cliff behind the threshold are different pictures.

A drop was also measured in an agent holding tools. This is a test that selected 68 data analysis tasks from analysis records (notebooks) actually uploaded to the data analysis competition site Kaggle. The tasks consist of 2,225 turns spanning six fields, where a turn is a single exchange with the agent. The tasks were designed so that the distance from any given turn to the prior results it relies on averaged 11.3 turns. When five top-tier models of that time were applied, the best-performing model had an average accuracy of 48.45%, and dropped by nearly 47 points while going from the earlier turns to the later turns. Between 52% and 69% of failures were long-horizon view errors, meaning errors caused by failing to sustain a long flow (arXiv:2605.30434, preprint). This means that the longer a task gets, the more mistakes are made later on.

Giving It More Steps Does Not Make It Better

The authors of the same data analysis test wrote down one more thing. Giving an agent more steps did not necessarily raise its performance. The authors believe the bottleneck lies in maintaining a correct analysis state rather than the number of steps it can use, in other words, its interaction budget (arXiv:2605.30434, preprint).

The same observation came out in search agents as well. This is a study that dissected six search agents finding answers through repeated searches step by step, comparing them against relevance judgments made by humans for each document. How many searches were performed and how good the answer was were only weakly linked. The accuracy of the answer matched better with the quality of the evidence reeled in up to that point than with the number of searches or the amount of text piled up in front of the model, in other words, the context. Useful evidence often appeared early in the exploration. Even so, the agents kept searching, and steps that reeled in nothing remained as a long tail (arXiv:2608.01913, preprint). This study divided failures into two types: cases where the necessary evidence was ultimately not found, and cases where it was found but not used properly. The mere fact that the latter exists serves as a counterexample to the idea that "if we just search more, it will work."

There are also exploration tasks where tool calls reach the hundreds. Exploration tasks where hidden rules must be figured out independently were set up in three environments, and in the heaviest setting, a single run takes over an average of 200,000 tokens and over 400 tool calls. Even in the standard setting, it exceeds an average of 35,000 tokens and 60 calls. Here, the agents consistently underperformed, and human participants scored higher. The authors write that simple scaling did not work in this task, but the abstract does not reveal what was scaled up, so it cannot be used as evidence regarding the step budget. The authors grouped eight error types extracted from the collected runs into two categories of causes: one is the phenomenon of getting trapped inside the context, and the other is a lack of functional fundamentals (arXiv:2509.21766, preprint).

What the data analysis test and the search study—which were given more steps—broke is the idea that "if we just run it longer and more, it will work." The multiplication picture is not broken by this. The multiplication model is a story about the number of steps required to finish a job. What these two studies dealt with is the number of steps allowed to the agent, in other words, the budget. Even if multiplication is correct, there is no reason for the success rate to rise just because the budget is increased. Therefore, these studies did not directly measure whether the steps are independent, and this article has not found a study that squarely measured the independence assumption. The name given by the exploration task study to the cause of errors, "the phenomenon of getting trapped inside the context," naturally brings to mind a pathway where earlier steps seep into later steps. However, since the abstract does not define this name, this is merely an interpretation read from the name. We look at one pathway where earlier steps spill over into later steps in the next section.

As the Log Grows Longer, Judgment Blurs

With every step, an agent piles up what it did and what it received in one place. In the terminal test from earlier, all 231 rounds pile up wholesale. There is a limit to that space. The context window is the limit on the amount of text a model can place in front of itself and look at at one time, and anything pushed out of it ceases to exist for the model. This accumulated log is the most direct pathway through which the result of a previous step carries over to the next.

That is why the window fills up quickly in long tasks. When that happens, task requirements, facts discovered in the environment, previous attempts, diagnoses, and unfinished sub-goals may end up buried inside the window or pushed outside of it, failing to influence decisions at the moment they are needed. The authors call this the loss of behavioral state. However, this description was written by a paper introducing its own memory method while introducing the problem (arXiv:2607.08716, preprint).

The saying "performance drops as the context gets longer" is widely accepted. Yet among the studies gathered for this piece, the paper that puts that topic front and center is one that has not yet run experiments. This paper hypothesizes that the larger a codebase—meaning all the code that makes up a piece of software—the longer the context becomes, and that this lowers the accuracy of tool calls. It stops at proposing the design of an experiment to test that hypothesis. The authors themselves write that there is yet no published research directly testing that combined coupling hypothesis. However, its introduction organizes prior research into three branches: that accuracy drops based on context length alone, regardless of retrieval failures; that the point where performance actually begins to drop is well below the limit announced by the company; and that the longer the accumulated context, the lower the task-resolution rate (DOI: 10.2139/ssrn.7344299, preprint / design proposal). These three branches were not measured by that paper, but read and organized by it from prior research. Mixing the two would mean citing a non-existent experiment as if it existed.

Measurements in this direction do exist. However, most take the form of "our solution worked." Research proposing to compress context states that an endlessly growing context muddies reasoning with irrelevant information. It reports that success rates increased compared to existing compression methods, by up to 46% in smaller models (arXiv:2510.00615, preprint). Other research proposes swapping out the continuous appending of logs for an execution state that summarizes how far things have come. This research writes that a purely append-only log increases response latency and pollutes the context to cause failure, and reports that accuracy went up (arXiv:2608.26263, preprint). Research that attaches a module to select and look back at memories raised scores by 8.3 percentage points and 6.8 percentage points across two benchmarks (arXiv:2607.08716, preprint). There is also a manuscript pointing out that the practice of web agents reading tens of thousands of tokens of webpage structure wholesale at every step eats away at reasoning long before the task is finished. This side is a raising of the problem, not a measurement (arXiv:2606.06708, preprint / position paper).

These gains are entirely the results of papers presenting their own methods. A result showing that things improved after a fix does not separately isolate and demonstrate that context length was the cause. The size of a problem is naturally painted at its largest by the side offering the solution.

But They Did Not Collapse All on Their Own Because They Were Long

A multi-agent system is an arrangement where multiple agents talk back and forth and work together. There is a study that had three agents converse from 200 turns to 500 turns, observing only their outward behavior without looking inside the model. The failure names the researchers attached are these: collapse, where meaningful speech suddenly stops; propagation, where one agent's breakdown drags another down with it; compensatory expansion, where the rest fill the empty space with extra talk; and hollow loquacity, where substance drops out while length is maintained.

The results run in the opposite direction of the previous section. Collapse was not a degradation that happened on its own, but a triggered failure that occurred when a disturbance was introduced. Out of 12 baseline runs executed without disturbances, 0 collapsed. Among 6 branch runs where a disturbance was applied by swapping the traits given to the agents (trait mutation), 5 collapsed, and they even broke down at scheduled turns (DOI: 10.2139/ssrn.6420858, preprint).

This result does not support "If it is long, it rots on its own." What this result points to is that length is a condition and the trigger is separate. It is a third picture, different from both the cliff behind the threshold seen in multi-step reasoning and the gentle attenuation drawn by multiplication. However, because it was observed in a setup where multiple agents converse, it cannot be directly transferred to a situation where a single agent works with tools.

The same study contains an even more chilling observation. Even the collapsed agents continued to pass identity-checking questions—questions asking whether they remember the identity given to them—with high fidelity. The authors call this a structural decoupling between the ability to remember identity and actual conversational behavior. This means that the checks commonly used failed to catch the collapse.

Where It Broke — Failure Attribution

If a single run that ends in failure is 145 steps long, how do you know which step was the culprit? Several studies gathered in this article ask the question that comes after "What percentage succeeded?", which is "Where did it break?"

Two terms are needed here. A trajectory is a record of the steps an agent took from start to finish, left in order, including what it thought, which tools it called, and what it received in return. Failure attribution is the task of looking at a failed trajectory and pointing out at which step things went decisively wrong.

Failure attribution is needed for more than just post-hoc debugging. The same problem appears when training long-horizon agents using reinforcement learning. In the words of "AI Models — A Calculation That Learns From Examples Instead of Rules", reinforcement learning is a method that gives scores to actions taken and adjusts behavior in the direction of increasing those scores. If it fails at the end, even the early steps that were correct get penalized together, and because successful trajectories are so rare, the signals to learn from almost disappear. However, this description was written by a paper introducing the problem while presenting its own method, and the solution that paper proposes goes in a different direction. Instead of pinpointing the culprit step, it bundled together devices that split the trajectory at intermediate milestones and finely distributed the credit for how far it progressed toward those milestones. With this method, the success rate on long ALFWorld tasks became 92.9%, nearly twice the 53.5% of the compared existing method (arXiv:2605.06078, preprint). Pinpointing which step is the culprit and teaching without pinpointing are different problems.

Failure attribution itself does not yet work well. There is a benchmark that collected 1,140 trajectories that actually failed across five domains, without artificially injecting errors. Each trajectory comes with a human-independently-graded correct answer, which is the earliest step where things went decisively wrong (the root-cause step). The median trajectory length is 145 steps. The accuracy at which the strongest existing method here correctly guessed the root-cause step was 13.2% (arXiv:2608.15242, preprint). The method proposed by the same paper got 51.1% right for the responsible role—meaning which of several roles went wrong—and 24.1% right for the step. Because the authors solved their own benchmark with their own method, this should be read together with the 13.2% of the existing method used as a comparison standard (baseline).

Research published in peer-reviewed conference proceedings points out the same difficulty. This paper diagnoses that existing methods using language models for failure attribution in multi-agent systems struggle due to long logs and a lack of specialized knowledge. So, mimicking how humans debug, it was split into two stages: first narrowing down the scope of the failure, and then pointing out the step. Step-level accuracy improved by up to 24.27% compared to existing methods (DOI: 10.1609/aaai.v40i39.40594, peer-reviewed proceedings). However, the abstract does not reveal the absolute level of the existing methods. It does not state what score it went from and to, nor whether 24.27 is percentage points or a relative improvement.

The Model Judge Could Not Tell the Types of Causes Apart

It is hard for a person to read through a 145-step record. So, studies that leave failure attribution to language models are on the rise. They feed the trajectories and ask, "Which step is the culprit?" As discussed in The Score Went Up. But What Went Up?, a language model tasked with grading like this is called a model judge. The heaviest result in this article is a study that tested this judgment, and this study passed peer review and was published in an academic journal.

This study split the causes of failure into two to design the experiment. Localized contamination is a failure created by a bad input that entered in a single step, and distributed drift is a failure caused by drifting little by little across multiple steps. The two pictures this article has compared all along—whether it is a single accident or an accumulation—directly became the problems for machine judgment. The research team changed the level of detail showing the trajectory across two agent tasks into four steps, and asked ten judge models while showing them trajectories with controlled flaws.

Most judges pointed to the step that produced the result, regardless of the cause. Thus, the accuracies for getting each of the two causes right end up being practically the same value read twice, and the difference between the two (the margin) stays near 0. A prompt is the text put in front of a model to get an answer (How Guessing the Next Word Becomes a Conversation — The Minimum Foundation for Reading Language Models). Even under the condition that showed the trajectory in the most detail, no judge exceeded a margin of 0.16 with the standard prompt. Even when changing the size of the flaw across a 2,500-fold range, the margin remained the same. In realistic tasks, four out of six judges went backward. When the cause was distributed, they instead pointed to a specific single step as the culprit more often (DOI: 10.3390/make8090275, peer-reviewed journal page).

The same judges did point out flaws under easy conditions. Under a condition where comparing just a single value was enough to show that the trajectory was wrong, they got the flawed step right 50 out of 50 times. Under a condition requiring arithmetic, it was 1 out of 50 times, and under a condition where there was no contradiction at all in the trajectory, it was 0 out of 50 times. The authors note that this limitation does not stem from a lack of capability. A deterministic rule—that is, a pre-determined calculation rule—not a model, separated the two causes in the trajectories of the main experiment. Once the prompt was tweaked once, the margin rose up to 0.34.

There are also reports that model judges align well with humans. In a diagnostic benchmark that collected over 3,100 trajectories across four domains to attribute failures, the agreement rate between the judge and human annotations was 0.84, which was higher than the agreement rate among human annotators themselves, which was 0.61. Annotations are judgments attached by humans to trajectories. However, the authors themselves call this benchmark a methodological first step at an early stage (arXiv:2604.11978, preprint). The two results do not directly collide. The former study asked about the ability to distinguish types of causes, while this study asked whether the attached judgments match those of humans. This contrast is simply this article placing side by side the research questions revealed by each of the two abstracts, and is not something written by the authors of either side. A report that a judge matches humans well does not mean the judge can distinguish anything.

The authors of the earlier study that tested the two causes narrow the scope of their conclusion themselves. They state that this is a warning about reliability within the evaluated models and prompts, and the constructed tasks. Even within that scope, something remains. When a model judge answers, "It broke at this stage," a substantial portion of that answer may be a reflex pointing to the stage where the result popped out. How far to trust model judges leads into The Grader Is Also a Language Model — How Far Can Its Verdict Be Trusted?.

Can We Know in Advance?

There is a study that used an agent's confidence as a failure signal in a deep investigation task. It has the agent state how confident it is, or measures how confidently it outputs the next word. AUROC is a scale showing how well a signal separates failed runs from successful runs, where guessing randomly is 0.5 and perfection is 1.0. By the end of the trajectory, the stated confidence separated failures quite well. The average AUROC is 0.85. But halfway through the trajectory, none of the examined signals exceeded an average of 0.60 (arXiv:2608.29685, preprint).

The question that follows finding the broken spot afterward is this. Can we know it will break in advance?

The authors also point to the reason. It is that the agent frequently switches exploration directions while in progress. Even if a signal that it is currently in a dead end appears honestly, if the agent soon moves to a different path, the connection between that signal and the final result is severed. Therefore, the authors' recommendation is unexpected. Rather than intervening in the middle of a trajectory, they suggest looking at the confidence after it finishes to decide whether to rerun it, and in their experiments, that worked better.

There is an even more troublesome result. This is a study that pinned checkpoint verification points into a security task requiring multiple steps, and separated failures that occurred before reaching the point of using the ability to be tested from failures that occurred after reaching it. In tasks where the state observed earlier must be used again later, a significant portion of failures occurred even before seeing that state. The research team tried inserting guide phrases that clarify the procedure in an experiment with 92 seeds—meaning random starting conditions—with the design locked down in advance. Then, the rate of observing the state rose from 65.5% in the control condition without guides to 95.4%. Up to this point, it is a clean diagnosis.

Yet when the exact same design was repeated on the next generation of the same model family, the effect came out in reverse. Whether the state was observed no longer reliably predicted task completion either. The authors' conclusion is this. The primary cause of failure can shift as it crosses model generations (arXiv:2608.20563, preprint).

This means a diagnosis that was correct in Gemini 2.5 Flash did not hold true in the same family's next generation, 3.7 Flash. These are results from a branch of tasks where observed states must be reused.

Some Failures Come From Outside the Model — Unresponsive Timeouts

Suppose an agent places an order using a tool, but no response arrives and a timeout occurs. From this signal alone, it is impossible to know whether nothing was executed at all, or whether the execution went through but only the response was lost. In the former case, resending is correct, but in the latter case, resending causes the exact same thing to happen twice. Choosing to stop is wrong in the exact opposite way. Not all failures happen inside the model.

Idempotence is a property where sending the same request twice yields the exact same result as sending it once. If an order is idempotent, clicking twice still results in a single order.

A study that tested this situation head-on paired two hidden states—namely, an unexecuted state and an executed state—behind identically appearing timeouts in each experimental unit. Consequently, the upper bound of success when recovering without additional information becomes exactly 0.5. It is like flipping a coin. Across 81 software work frameworks, the results of the main experiment running 756 pairs on a single model of a service are as follows. The condition that only provided advice via prompts was 0.4979, the condition that allowed querying the state to check was 0.8066, and the condition that laid down a reliable idempotent contract ensuring the service side processes the same request only once was 1.0000 (DOI: 10.21203/rs.3.rs-10730245/v1, preprint).

The initial figure of 0.4979 is pinned right to the coin-flip ceiling. Well-coached advice practically accomplished nothing. Opening a window to ask about the state jumped the figure to 0.8066, and making retries safe by design made failures vanish. The authors conclude that providing recovery means on the interface side can shift performance from an informationless ceiling to reliable state recovery. They also attach a caveat to the same sentence. Providing information does not guarantee that it will be used correctly.

The scope is also narrowed directly by the authors. How often such things happen in actual operations, what happens inside the model, and how providers differ from one another are outside the target of estimation. Under some experimental conditions, output format violations were also at a non-negligible level. Yet the direction is clear. This experiment left the model as is and only changed the interface, yet failures decreased. A large share of failures in this experiment came from the outside world failing to inform what had happened. Failures remaining even after giving information belong to the side using that information.

Some Chains Stopped Not Because They Failed, But Because Someone Said "Done"

Among the ways a long chain breaks, the most human-like one exists.

Finite element analysis is an engineering calculation method that breaks a structure down into tiny pieces to calculate forces and deformations. There is a study that ran 1,120 controlled experiments with seven agent combinations, set against four tasks of writing code to perform this calculation. The first two results found are "no effect." Adding more reviewing agents, or changing the team composition, failed to provide consistent improvement. An agent built to evaluate critically did not actually do that job.

The primary failure mechanism pointed out by the authors is early agreement—phrases that sound like work is finished, such as "looks good" or "execution successful," which cuts off iterative improvement prematurely. Once we strip away only the misleading agreements and leave the necessary guidance, the success rate jumped from 10.7% to 77.5% in one task, and from 0.7% to 45.0% in two other tasks (DOI: 10.2139/ssrn.6532560, preprint). However, this is an intervention by the authors themselves. The model used in the experiment was single and already a previous generation by the time the manuscript came out, and the tasks are also of one branch. It is safer to take away only the mechanism rather than the magnitude of the numbers.

There is also a benchmark that separately measured whether agents back away by saying they do not know. In a benchmark where regulatory tasks such as loan screening and insurance claim adjudication were designed as long-horizon tasks, a study ran six memory structures. Memory structures are the methods that determine how long records are kept and retrieved. All six structures made judgments in every single case, and none of the structures used the option to back away by saying they do not know. In the same paper, there is also a scene where the authors lose their own prediction. The authors had pre-registered that the summarization method would fail at factual retrieval, but the data turned out largely the opposite. A plain summary with instructions to preserve the facts was actually a strong baseline across multiple axes (arXiv:2604.19457, preprint).

What a Single Number Hides

Finally, we look at what is actually being measured itself. The authors of that loan screening benchmark start from this sentence. Today's evaluations report task success as a single number, and that single number flattens different patterns of failure together (arXiv:2604.19457, preprint).

The program framework that actually runs a benchmark is called an evaluation harness. A survey paper that organizes these harnesses by type notes that an evaluation harness determines what the task is, what the model is allowed to do, and how evidence is graded. Among them, harnesses that measure tool use separately measure the decision not to call a tool and recovery from tool failure (DOI: 10.20944/preprints202609.0236.v1, preprint/survey paper).

An article synthesizing methods for measuring failure diagnosis and recovery capabilities diagnoses that existing benchmarks mainly measure task success under normal conditions, and show almost nothing about why it failed or whether it can recover. As what is needed, it lists controlled fault injection, fine-grained diagnosis, and evaluation protocols that separate passive success where things simply ran smoothly from active recovery where the model fixed itself. The same article notes that this work has already begun. Results have shown that trajectory-based diagnosis is vastly better than quick fixes that merely adjust prompts, and perturbation taxonomies that categorize faults by type as well as metrics that measure recovery separately from task completion have also been proposed. As a remaining issue, it points to the validity of the benchmark itself. Hidden flaws discovered in widely used benchmarks cause model errors and benchmark flaws to become intertwined (DOI: 10.2139/ssrn.7335138, preprint/synthesis).

This passage meets the question raised by "The Score Went Up. But What Went Up?". If what wavered there was the meaning of "got it right", here the meaning of "finished" wavers.

What Can and Cannot Be Said Now

To summarize, here are the following seven points. Of these, six rely on preprints, and only ⑤ has been published in a peer-reviewed venue.

① Multiplication is a prediction, not an observation. The calculation that 95% at one step becomes 36% at twenty steps is arithmetic for when steps are independent. In multi-step reasoning with increased complexity, a collapse behind a threshold was observed rather than a gradual decay. However, this measured complex complexity rather than step count on the horizontal axis, and it is not the result of an agent wielding tools (DOI: 10.21203/rs.3.rs-8251575/v1).

② Increasing steps did not turn out to be a solution. Giving more steps did not necessarily raise performance (arXiv:2605.30434), and search effort and answer quality were only weakly coupled (arXiv:2608.01913). However, this is a result for the allowed budget, and it did not measure the independence of steps.

③ Nor is length immediately collapse. In experiments of 200 to 500 turns, collapse occurred 0 times out of 12 baselines. Collapse happened when disturbances that introduce variation into properties were added (5 out of 6 variation branches), and the authors read this as a triggered failure rather than spontaneous degradation (DOI: 10.2139/ssrn.6420858).

④ Finding the broken spot is not yet done well. In trajectories with a median of 145 steps, the rate at which the strongest existing method accurately pinpointed the root-cause step was 13.2% (arXiv:2608.15242).

⑤ Within the tested range, model judges did not reliably distinguish between types of causes. With standard prompts, the margin separating localized contamination from distributed drift was near zero, and the judges converged on a single tendency to point to the step that produced the result. Tweaking the prompts raised the margin slightly (DOI: 10.3390/make8090275, peer-reviewed venue).

⑥ In deep investigation tasks, signals in the middle of the trajectory barely heralded failure. At the halfway point, no signal exceeded an average AUROC of 0.60 (arXiv:2608.29685), and there are cases where the diagnosis itself flipped across model generations (arXiv:2608.20563).

⑦ Certain failures vanished upon changing the interface. Recovery, which remained stuck at a coin-flip ceiling with advice alone, became 0.8066 with state checks and 1.0000 with by-design safe idempotent contracts (DOI: 10.21203/rs.3.rs-10730245/v1).

What cannot be said with certainty is also clear. This article found no studies that broadly measured what shape of function the failure rate of long tasks draws. The drops cited here each come from different tasks, different models, and different scales, so they cannot be placed on a single curve. Nor do we know whether the steps are truly independent of one another, or how often failures are cumulative. This is because model judges, commonly entrusted with sorting that out, failed to properly distinguish the two causes within the tested range.

What remains is a slightly different sentence. Finding the spot where a chain snapped afterward is different from knowing in advance where it will snap. What the research gathered in this article can do is part of the former. Regarding the latter, the answer produced by the research gathered in this article is that it is still difficult in the tested tasks.

Source Papers

21 of the 23 items are un-peer-reviewed preprints. The two items published in peer-reviewed venues are indicated as such.

• "Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading" (2026, preprint) — arXiv:2607.08964 — 46 tasks, 15 models. Average 9.9 million tokens, 231 turns, and 85.3 minutes per task; top models achieve 15.2%/10.9%, overall average 4.3%/1.7%.
• "SCALER: A Procedurally Generated, Leakage-Resistant Benchmark for Evaluating Multi-Step Reasoning in Large Language Models" (2025, preprint) — DOI: 10.21203/rs.3.rs-8251575/v1 — GPT-4o and Claude 3.5 Sonnet generations, 7 models, 12,000 questions. S-curve attenuation and collapse thresholds; multi-step reasoning rather than tool-wielding agents.
• "LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis" (2026, preprint) — arXiv:2605.30434 — 68 tasks, 2,225 turns. Top score 48.45%, a 47-point drop between adjacent turns, 52–69% of failures caused by long-horizon visibility errors; adding steps does not necessarily improve performance.
• "Diagnosing Search Behavior and Failure Modes in Long-Horizon Search Agents" (2026, preprint) — arXiv:2608.01913 — Weak alignment between search effort and answer quality, long tails of fruitless steps, decomposition of search failures versus utilization failures.
• "UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon Scenarios" (2025, preprint) — arXiv:2509.21766 — Scale of 200,000 tokens and 400 tool calls. Lower scores than humans, simple scaling failure, eight error types.
• "Remember When It Matters: Proactive Memory Agent for Long-Horizon Agents" (2026, preprint) — arXiv:2607.08716 — Loss of behavioral state: requirements, environmental facts, previous attempts, diagnoses, and unresolved subgoals become buried or pushed out.
• "Context Rot and MCP Tool Attrition in Production-Scale Codebases" (2026, preprint / design proposal) — DOI: 10.2139/ssrn.7344299 — Design proposal without experiments. Three preceding lines summarized in the introduction (effect of context length alone, effective window lower than published limits, cumulative context and resolution rate).
• "ACON: Optimizing Context Compression for Long-horizon LLM Agents" (2025, preprint) — arXiv:2510.00615 — Costs generated by growing contexts, reasoning degradation caused by irrelevant information.
• "SKILL.state: Scalable Long-Horizon Agent Skills" (2026, preprint) — arXiv:2608.26263 — Append-only logging leads to delayed deterioration and context pollution failures.
• "Signal-Driven Observation for Long-Horizon Web Agents" (2026, preprint / position paper) — arXiv:2606.06708 — The argument that the practice of swallowing tens of thousands of tokens of screen structure at every step erodes reasoning long before the task ends.
• "Behavioral Drift in Multi-Agent LLM Systems: Emergent Failure Modes, Cascade Dynamics, and Measurement Challenges" (2026, preprint) — DOI: 10.2139/ssrn.6420858 — 200–500 turns. 0 collapses out of 12 baseline runs, 5 collapses out of 6 mutation branch runs; collapsed agents still pass identity checks.
• "LongRCA Bench: Diagnosing Responsible Roles and Root Causes in Long-Horizon Agent Failures" (2026, preprint) — arXiv:2608.15242 — 1,140 failure trajectories without injection, median 145 steps, strongest baseline 13.2%. Author method: 51.1% for roles, 24.1% for steps.
• "Scope Delineation Before Localization: A Two-Stage Framework for Enhancing Failure Attribution in Multi-Agent Systems" (2026, peer-reviewed venue) — DOI: 10.1609/aaai.v40i39.40594 — Diagnosis of long logs and lack of domain expertise, two-stage approach of scoping first and then pointing, up to 24.27% improvement at the step level.
• "One Reflex, Two Answers: Model Judges Weakly Discriminate Localized from Distributed Causes in Agent Failure Attribution" (2026, peer-reviewed venue) — DOI: 10.3390/make8090275 — Ten model judges, four publication levels. Margin below 0.16 on standard prompts, 0.34 after prompt intervention, invariant even when scanning 2,500-fold defects, 4 out of 6 reversals on real-world tasks, comparison 50/50 versus arithmetic 1/50 versus consistent trajectory 0/50.
• "The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break" (2026, preprint) — arXiv:2604.11978 — Over 3,100 trajectories across four areas. Inter-annotator agreement 0.61, model-judge-to-human agreement 0.84. The authors note this is an early methodological step.
• "Last Step Matters: Early Uncertainty Cannot Predict Failure in Long-Horizon Agents" (2026, preprint) — arXiv:2608.29685 — Average AUROC of confidence at completion time is 0.85, whereas no signal at the 50% mark is below 0.60. The cause is path-switching. The recommendation is final-confidence-based restarting.
• "Beyond End-to-End Success: Diagnosing Failures in Long-Horizon Security LLM Agents" (2026, preprint) — arXiv:2608.20563 — In 92 pre-specified seed experiments, guidance improved state observation from 65.5% to 95.4%. Opposite effect in next-generation models, shift in dominant failure causes across generations.
• "Did It Happen? Counterfactual Evaluation of LLM Agent Recovery from Ambiguous Tool Outcomes" (2026, preprint) — DOI: 10.21203/rs.3.rs-10730245/v1 — Uninformed recovery upper bound 0.5. Advice 0.4979 / state query 0.8066 / idempotent contract 1.0000. Significant strict-output violations in some cells; operational incidence rates, internal mechanisms, and provider comparisons are outside estimates.
• "Optimizing Multi-Agent LLM Collaboration via Communication-Flow Control" (2026, preprint) — DOI: 10.2139/ssrn.6532560 — 1,120 controlled experiments. Increasing reviewers or changing composition yields no gains; premature agreement is the dominant failure mechanism. Post-intervention: 10.7% to 77.5%, 0.7% to 45.0%.
• "Four-Axis Decision Alignment for Long-Horizon Enterprise AI Agents" (2026, preprint) — arXiv:2604.19457 — A single success scalar flattens failure modes. All six memory structures make judgments in every case, and the authors' pre-registered predictions were largely contradicted by the data.
• "Benchmark, Agent, and Tool-Use Harnesses for Large Language Models: A Taxonomy of Evaluation Infrastructure" (2026, preprint / survey) — DOI: 10.20944/preprints202609.0236.v1 — Evaluation setups determine tasks, permitted behaviors, and grading. Tool-use items include "deciding not to call" and "recovering from tool failure".
• "Designing Benchmarks and Evaluation Protocols for Agent Failure Diagnosis and Repair Capabilities" (2026, preprint / synthesis) — DOI: 10.2139/ssrn.7335138 — Current benchmarks measure success only under normal conditions. Protocols separating controlled defect injection, fine-grained diagnosis, passive success, and active recovery. Benchmark flaws mix with model errors.
• "Milestone-Guided Policy Learning for Long-Horizon Language Agents" (2026, preprint) — arXiv:2605.06078 — Credit misattribution that punishes correct earlier steps due to endpoint failures, loss of learning signals caused by sparse successful trajectories.