Most documentation describes the chain as something you read after a run. This page is the other direction: when you write a check, you are choosing a link.
Which links you can write a check for
Four of the six. Every predicate you author is filed at one link, and the count per link is uneven:
Connection is measured by the runner. Discovery also supports catalog assertions. The runner records initialize and tools/list outcomes and preserves their setup attribution. Six optional discovery assertions inspect complete raw per-server catalogs before deduplication. Missing or partial catalog capture reports an evaluator error, not a pass. Scored gating catalog failures can fail Discovery; advisory failures remain visible evaluator results without failing the stage. A description-based deprecation check is advisory only.
Connection remains measured from setup evidence whether or not a suite authors assertions. Discovery reports both runner evidence and configured catalog checks.
The stage is where the evidence is filed
A check’s link says where its evidence is recorded. It does not assert that a failure originated there.noToolErrors is the clearest case. It files at Response, because a tool error is the server’s answer. Until analyzer version 11 it filed at User value, and the same defect was counted twice: the analyzer already failed Response on an observed tool error, while the predicate row failed User value. Which link a reader saw as the first break depended on which row they looked at first.
Two consequences worth keeping in mind:
- A failure at one link is frequently caused upstream of it. Selection failing because two tools have near-identical descriptions is a Discovery problem wearing a Selection label.
- Moving a check to a different link changes where historical failures are attributed, so it is a versioned analyzer change rather than an edit anyone can make locally. The three
widget*kinds are current candidates to move to Response.
Graders that are not predicates
Three graders file at a link without being checks you write in a list:
A gating
toolCalledWith is promoted into the matcher’s expectations and graded there. An advisory one stays a predicate row, because promoting it would create an expectation that can fail the trial, which an advisory check must never do.
What a suite is not measuring
Coverage is the question the six links exist to answer, and it is easy to write a plausible-looking suite that leaves most of the chain untouched. A case withtoolCalledWith, noToolErrors and responseContains measures Selection, Response and User value. It says nothing about Tool call — whether the arguments the model sent were valid against the tool’s own schema — and has no authored Discovery assertions. Connection still reports runner evidence.
To cover Tool call, add argumentsMatchToolSchema. Read Connection in the run’s chain. To inspect catalog metadata, configure Discovery assertions and read their evaluator rows alongside the runner’s discovery measurement.
Two habits keep coverage honest:
- Read the chain on a passing run, not only a failing one. Six links reading
notMeasuredis not the same as six links passing, and only one of those is worth shipping on. - Treat a link with no check as unmeasured rather than fine.
notMeasuredis an absence, and absence is not a pass.
Observations never decide a link
Five kinds are marked Observation in the reference tables:noEndingQuestion, noRepeatedIdenticalCall, noDeprecatedToolCalled, toolErrorNamesInput and fullPageHasContinuation.
Each is a heuristic that can be right about what it saw and wrong about what it means. A poll loop and a wasteful retry are the same shape; a full page is not proof that more results exist; “Rate limited. Retry in 30 seconds.” names no input key and is a good error message. So they carry one rule everywhere: role: "advisory" is required, a required one is refused when written, and they are recorded beside a verdict without changing it.
They still file at a link, which is what places them in the right group when you are reading what a suite measures. They do not make that link pass or fail.
Where to go next
- Predicate gate reference — every kind, its fields, and the link it measures
- Running evals — authoring a suite and reading its results
- Validators — the tool-call matcher that files at Selection