Benchmark, Agent, and Tool-Use Harnesses for Large Language Models: A Taxonomy of Evaluation Infrastructure

Large language model evaluation has moved from static benchmark scripts to executable harnesses:systems that bind datasets, prompts, model adapters, inference backends, tools, environments, sandboxes,scorers, traces, and leaderboards into a measurement contract. This survey explains why that layer nowdetermines what a benchmark score means. We distinguish three harness families.

Benchmark harnessesmake model comparison possible by fixing task registries, prompt templates, inference adapters, answerparsers, metrics, and aggregation rules. Agent harnesses turn evaluation into an environment experimentin which repositories, browsers, operating systems, mobile apps, users, databases, reset procedures, andexecutable oracles become part of the measured system. Tool-use harnesses isolate the interface problem: retrieving tool documentation, selecting APIs, formatting arguments, deciding when not to call atool, carrying state across turns, and recovering from tool failures.

We map the infrastructure shared bythese families, show how their scoring contracts differ, and explain why scaffold effects, oracle design,environment drift, schema sensitivity, and trace policy can change the scientific meaning of a result. Thepaper argues that harnesses are no longer peripheral implementation details. They define what the taskis, what the model is allowed to do, how evidence is scored, and whether a result can be audited or cited.We provide a layered taxonomy, a literature stem map, family-specific failure analyses, guidance for in-terpreting leaderboard scores, and a reporting agenda for model developers, benchmark maintainers, andresearchers building agentic systems.