On Scale AI's visual-inference benchmark, the best model trails humans by 39.5 points

The gap is 39.5 points at the top of the table and 62.2 points at the median ^[24][25]. Both numbers depend on how a task is scored.

Each answer is free text. An automated language-model judge grades it against criteria written for that task ^[6]. Free-form answers remove multiple-choice elimination as a strategy, and they put a model in the grading path. Scale checked the second problem sensibly: five models were regraded with judges from three vendors, and the rankings held with more than 95% agreement ^[8]. The task set was also filtered hard. Scale says 3,466 tasks were authored and 522 survived three independent review rounds ^[7], a keep rate of about 15% ^[26].

The headline metric is pass@1, defined on Scale's leaderboard as the average across three attempts per task ^[10]. In effect it is expected accuracy on one try. The human baseline comes from just 20 participants answering free-form ^[11].

A builder gets more out of the failure analysis than the leaderboard. Elorian's site argues that current vision-language models translate images into language before reasoning, and that this step is fragile for spatial and relational tasks ^[20]. Scale's attribution of nearly all failures to perception or latent inference fits that account ^[14]. If the mistake happens when the image becomes a description, a longer chain of thought reasons over a wrong description. Scale reports that models averaged about 4,000 reasoning tokens per task, and that greater effort reduced GPT-6-astra's score on some subdomains ^[17]. The evidence covers one model on part of the test. Runtimewire's account does not include an effort sweep across all 25 models, so the case against buying more reasoning is shown for the leader and only suggested for the rest.

The tool runs point the same way. Closer inspection cut errors such as missed visual cues, but errors that read a two-dimensional overlap as three-dimensional alignment went from 102 to 101 ^[19]. A zoom gives the model more pixels. It does not tell the model which object sits in front of the other. The tool-use score also covers about 74% of the full task set ^[27], so it is not comparable with the full-benchmark result ^[18].

For 53.6% to predict a product's error rate, the product's inputs would have to resemble HSS prompts, such as whether a vehicle can fit in a parking space or why someone slowed down after running ^[5]. The model would also have to be new to the media. Scale notes that some source media comes from the public web, so prior exposure in training cannot be ruled out ^[12]. On questions about beliefs or intentions, the score measures agreement with annotator judgment ^[13]. Runtimewire's write-up says the headline score is not a measure of how often a model will fail in everyday use ^[23].

Elorian co-founder and CEO Andrew Dai helped build the benchmark with Scale ^[21]. Elorian says it has raised $55 million to build AI around visual reasoning ^[22]. The benchmark finds the weakness Elorian was founded to fix, a coincidence I would hold against it more without the cross-vendor grading check.

I think caution on spatial and social inference is justified by this evidence. I would not ship a feature that judges spatial fit or a person's intent from images, without human review, on any of the 25 models tested. I would not quote 53.6% as that feature's error rate either.