One Reflex, Two Answers: Model Judges Weakly Discriminate Localized from Distributed Causes in Agent Failure Attribution

When an autonomous language-model agent fails, a growing body of work hands the diagnosis to a large language model (LLM) that reads the trace and names the responsible step. We ask whether this attribution separates two causes, a localized corruption from one bad input and a distributed degradation across many steps. Using controlled fault injection across two agent tasks, four trace-completeness levels and ten judges, we find discrimination weak wherever the causes are demonstrably identifiable from the trace.

Most judges apply one propensity, naming the step that produced the result, so their two per-class accuracies become readouts of one number whose margin stays near zero. Two frontier models produce results consistent with opposite ends of that axis, one asserting an inconsistency the trace does not contain, the other abstaining. On the realistic task four of six judges invert, naming a concrete step more often on the distributed cause than on the localized one.

The limit is not incapacity. Where contradicting the trace takes only a comparison the same judge localizes the fault in fifty of fifty runs, against one of fifty where it would take arithmetic and zero of fifty where the trace is consistent, and a 2500-fold sweep of fault magnitude leaves the margin unchanged. A deterministic non-model rule separates the causes on the main-panel traces, where no judge under the standard prompt exceeds 0.16 at full detail; one prompt intervention lifts the margin to 0.34.

We read this as a reliability caution for LLM-as-judge attribution across the models, prompts, and constructed tasks evaluated here.