Artificial Analysis retries a provider safety error ten times before scoring the attempt zero

Artificial Analysis scores a fallback attempt normally ^[7], and it says the fallback model may be less capable than the first choice ^[8]. So for any agent that routes around a refusal, the capability score is a blend. Some tasks were completed by the model you picked. The rest were completed by whatever the harness reached for next.

The retry rules are asymmetric. A provider safety error gets up to ten retries, and the attempt scores zero only if every retry is blocked ^[5]. A model refusal that ends an attempt scores zero right away, with no retry ^[6]. One zero can therefore sit behind as many as eleven calls to the provider ^[22]. The published rate counts retained, scored attempts and drops superseded retries ^[9], so it is a per-attempt figure. Your own error dashboard counts every call. The two numbers will not match.

A blocked rate of 2% could have cost an agent no more than two index points, Artificial Analysis says, because every blocked attempt scores zero ^[11]. The bound is proportional, so a 5% blocked rate caps the loss at five points ^[19]. The index runs 303 tasks across three evaluations ^[12]. If those three are equal in size, 2% is about six refused attempts ^[20].

For the published rate to predict what you will see, your task mix has to resemble the index's ^[13]. The overall rate gives each of the three component benchmarks equal weight ^[10]. A refusal in the smallest evaluation therefore moves the published figure more than a refusal in the largest. Under equal sizes, one refusal is worth about a third of a percentage point ^[21].

Micah Hill-Smith, now CEO, and George Cameron, now chief product officer, co-founded Artificial Analysis ^[17]. Hill-Smith has described the origin: while building a legal-research tool, he found different models worked better at different steps, with no dependable way to compare quality, speed and price ^[18]. Refusal reporting extends that comparison from model quality to provider policy.

The metric does not determine whether a provider's policy is appropriate, Artificial Analysis says; it measures the practical effect of that policy within a defined set of tasks ^[15]. The company also says buyers still have to decide whether a lower refusal rate reflects greater usefulness, looser safeguards or a harness that handles provider restrictions more effectively ^[16]. Two of those three readings describe engineering a buyer can test. The third describes how much safeguard the vendor relaxed, and the refusal rate on its own does not tell them apart.