Despite its 2,969-fact corpus, banking contributes least to Sierra's agent-building benchmark score

Reconstruct the aggregate before quoting it. Banking supplies 35 of the 53 construction tasks ^[8]. Claude Opus 5 scored 5.9% on banking, against 72.8% on retail, 55.9% on airline and 48.2% on telecom ^[7]. Weight those by task count, assuming the other 18 tasks split evenly at six per domain: 35/53 x 5.9 gives 3.9 points, and 6/53 x (72.8 + 55.9 + 48.2) gives 20.0 points, which sums to 23.9 ^[2]. That is the reported headline score ^[6]. The even split is an assumption on my part, but the fit is tight and the shape holds either way: roughly two thirds of the number is a banking score ^[1].

Banking is where the information sits. Its corpus holds 2,969 individual policy facts, and a single task can depend on as many as 580 of them ^[8], about a fifth of the corpus bearing on one conversation ^[3]. Sierra's process analysis says the developer agents stopped researching the business too soon and asked too few questions when information was missing ^[13]. Anyone who has onboarded a contractor will recognise the pattern. Those two failures bite hardest in the domain with the most facts to find, which is also the domain carrying most of the score.

A new benchmark is supposed to produce low scores at first: a test the frontier routinely aces stops discriminating between systems ^[14].

The 82.2% reference bar deserves more care than the arithmetic invites. Sierra's paper says the reference agents were hand-built by a benchmark author working with a frontier model and with access to the ground-truth requirements the autonomous developer agents had to discover for themselves ^[11]. The accompanying blog post by Sierra researchers Ben Shi and Keshav Dhandhania describes this as a model paired with "an engineer with deep context" ^[10], while Sierra calls the figure an oracle reference, a measure of best-case performance under full information, not a measure of average human performance ^[12]. The 58.3-point gap ^[4] therefore prices access to the spec. It bundles together what human review contributes and what a correct requirements document contributes, without separating the two.

Whether 23.9% transfers to your build depends on where your requirements live. The benchmark withholds the spec and leaves it latent in documents, transcripts, an API and a codebase ^[4]. If your requirements are already written, reviewed and signed off, the axis this benchmark scores hardest is absent from your workload. Grading matters too: a task passes only when the built agent makes the correct changes in the business's underlying systems, not when its answer reads well ^[5]. The developer agents also worked under model and cost constraints ^[4], so a team willing to spend more per conversation sits outside the measurement.

The comparison worth carrying into an architecture review is 5.9% on banking when the requirements had to be discovered ^[7] against 82.2% on the same benchmark when the builder already had them ^[9][11]. That is an argument about specification quality, and specification is currently the human's seat in the loop, alongside choosing architectures and testing the result ^[15].