Ten agent eval protocols say when a run stops. Fewer say whether the result is settled.

The replay holds the agent's actions fixed and varies only the accounting ^[1]. That is what makes it a mechanism demonstration rather than a benchmark result: if the actions did not change, the divergence between the two labels cannot be charged to the model behaving differently ^[17]. Same trajectory, two labels, and the one that gets published depends on when the evaluator looked.

The second half of that finding is the part an infrastructure team can act on. The carryover into the next run appeared when service state persisted and did not appear after isolation or verified reset [c5b], which places the defect in the harness configuration rather than in the system under test ^[18].

The two conditions also do not come as a pair ^[6]. Polling until the write lands settles whether the write succeeded, and leaves the written state sitting where the next run can read it. A fresh container prevents that read, and tells you nothing about whether the operation the score refers to ever finished ^[6]. Each of the two obvious fixes buys exactly one condition.

Which is roughly why the protocol review found what it found. The endpoint is defined by the harness: the evaluator stops asking for actions after the model reports completion, or on a success condition, a timeout, or an action limit ^[9][10]. All ten protocols document that boundary and what gets scored [c6a], because all of it is in code the evaluator owns. The status of a pending write, a leftover credential, or an account created mid-run sits in a service the evaluator does not own ^[9]. So counting a stopped run as one trial ^[4] is an assertion about the environment, offered in place of evidence about it ^[15].

The two field reports the authors cite map cleanly onto the two conditions. The UK AI Security Institute's concurrent samples reusing accounts and artifacts left by other samples is separation failing ^[11]. The OpenAI and Hugging Face accounts of evaluation activity that used external services and continued past the original runtime is persistence past the endpoint, the case where the scored outcome was still moving ^[12]. The proposed open-effects record does not repair either one. It lists what may still be live after the endpoint, its status, and whether it could change the score or reach another run ^[16]. That is a disclosure format, and its whole value is that it gives a harness somewhere to write down what it does not know.