PerceptionBench puts a number on the step your pipeline treats as free

Moonshot AI has published PerceptionBench, a benchmark that measures whether a multimodal model can read an image at all, with reasoning and outside knowledge deliberately removed from every question ^[1][4]. Of the 16 frontier models tested, none reached 60 percent ^[2][9], which puts a number on the stage most production pipelines treat as free.

How it was built matters more than the ranking. The authors analysed 42 open-source benchmarks and found their error profiles barely overlapped, so each was sampling a different corner of perception and no small group of them covered the whole ^[5]. Rather than defining categories first, they collected real model errors and traced each back to the earliest step that failed, producing ten skill domains: Visual Relation, Counting, Attributes, Depth & 3D, Localization, Comparison, Fine-grained Recognition, Context Integration, OCR and Hallucination ^[6][3]. From an internal pool of more than 17,000 verified questions they released 3,000, under a fifth of the set ^[7][23]; 60 percent derived from attributed model errors and 40 percent from reformulated augmented images, roughly 1,800 and 1,200 items ^[7][24]. The tasks are mundane: where a symbol sits on a clock face, how many flowers are inside a red box, which of two pencil cups is gray-pink rather than solid pink with a cartoon ^[8].

The scores, as reported by the-decoder: GPT-5.6 Sol 59.7 percent, Kimi K3 58.5, Claude Fable 5 57.2, Gemini 3.1 Pro 56.2, GPT-5.5 55.8 ^[9]. Open-weight models trail, with Qwen3.5-397B-A17B at 47.5 and GLM-4.6V at 32.5 ^[10]. The top five sit within 3.9 points of each other ^[20], while the spread from first to GLM-4.6V is 27.2 points ^[19].

For selection purposes the aggregate is nearly useless, and the authors say so: models with almost identical totals diverge sharply by category ^[11]. Hallucination is the weakest domain on average ^[12]. GPT-5.6 Sol, first overall, scores 26.9 percent there, 32.8 points below its own aggregate ^[12][21], while Gemini 3.5 Flash, weaker overall, scores 50.6, or 23.7 points better on that skill ^[12][22]. That sub-test checks whether a model invents objects that are not present when the correct answer is zero ^[13]. If you run an extraction pipeline where absence is a valid answer, that is your failure mode, and the headline leaderboard hides it.

The operationally useful claim is the diagnostic one. Moonshot's authors argue that many failures logged as reasoning errors occurred earlier, at the image-reading step, so when a multi-step task collapses the first step has often already gone wrong ^[14]. Longer thinking budgets, chain-of-thought scaffolding and self-consistency voting all compound a misread premise rather than catching it. PerceptionBench's method is the transferable part: decompose questions into perception-only sub-questions until you can name the visual ability that failed ^[15]. The dataset and evaluation code are on GitHub ^[16].

One caveat on provenance: Moonshot AI built the benchmark, and its own Kimi K3 places second on it ^[25]. K3 remains a few points behind Claude Fable 5 and GPT-5.6 Sol on general benchmarks and well behind on offensive cybersecurity and complex maths, but ranks level with them on perception ^[17].

Watch whether vendors start publishing per-category perception numbers instead of a single aggregate, and whether the same team's earlier WorldVQA split between recognition and reasoning gets folded into this taxonomy ^[18].