Beyond E2E Scripts: Using LLM-Proposed Scenarios Without Letting the LLM Be the Oracle

Large language models can enlarge the scenario space explored by end-to-end tests, but they are unsafe as test oracles: they can invent impossible states, misread application semantics, and overstate weak evidence. Bayesilisk separates scenario proposal from verification. App-specific connectors publish a bounded search surface consisting of routes, actions, identifiers, state facts, invariants, and executable probe hooks.

The Bayesilisk core expands that surface into candidate scenarios, rejects malformed candidates by deterministic contract validation, prioritizes valid probes with Grassmann-style attention and a Bayesian score, and verifies only evidence returned by connector execution. The same boundary supports coding-agent assistance: agents may draft connector context, action mappings, and repair briefs, but cannot author observations, pass/fail results, or issue readiness. The reusable search surface is a library of category-theory motifs: abstract tokens are objects, connector actions are morphisms, and each connector is a functor to its concrete application; a deterministic scanner derives the surface from an API description, and a stateless loop controller drives motif binding, verification, and repair to convergence while the oracle stays deterministic.

This paper gives a compact formal model for that loop and reports a Cal.com case study at a fixed May 2026 revision. From connector-declared route facts and a typed Abstract Bayesilisk Action Graph (ABAG), Bayesilisk generated seven probes. All of them returned deterministic expected-vs-observed violations.

One upstream report has a targeted fix pull request that identifies the root cause and adds a regression Playwright test; an independent human reviewer approved that fix.