Did It Happen? Counterfactual Evaluation of LLM Agent Recovery from Ambiguous Tool Outcomes

Abstract A timeout after a side-effecting tool call does not reveal whether the action failed before execution or executed successfully and only lost its response. Blindly retrying is correct in the former case and can duplicate the effect in the latter; stopping has the opposite failure pattern. We introduce a counterfactual benchmark in which every experimental unit pairs these two hidden commit states behind an identical agent-visible timeout.

The design gives recovery without an information-bearing affordance an exact balanced exactly-once success ceiling of 0.5: status makes the hidden state observable in principle, while a stable-idempotency control makes retry safe by construction. The benchmark contains 81 software-engineering templates across create-once, ordered-transition, and reserve-then-link state topologies. In a 756-pair main Alibaba Cloud qwen-plus serving cell, intent-to-evaluate state success was 0.4979 with prompt advice, 0.8066 with status reconciliation, and 1.0000 with a stable idempotency contract.

The predeclared contrast was 0.4053, with a fixed-suite repeat interval of 0.3992 to 0.4115. Two predeclared directional replications, one separately designated aggregator-served exploratory cell, and two post-main institutional secondary cells preserved the positive state-only direction, although strict-output violations were material in some cells. The results show that interface-level recovery affordances can move a benchmark from the no-information ceiling to reliable state recovery, while information alone does not guarantee correct use.

The study estimates these effects under controlled ambiguous outcomes; production incidence, internal mechanisms, provider comparisons, and hardware effects are outside its estimand.