Build1 publisher3 min readPublished
A 1.5x per-token price still bought a 25 percent cheaper correct answer in AWS's benchmark
AWS's open harness records $0.0021 per correct AIME answer for gpt-5.6-luna after an 80 percent Bedrock price cut. The figure depends on running luna with reasoning disabled while mini runs at its defaults.
The Engineer · Build desk

What happened
- AWS published results from an open-source harness, openai-on-aws/benchmarks-openai, that measures accuracy, token use and turn count instead of comparing models on dollars per million tokens.
- It evaluated gpt-5.6-luna, gpt-5.6-terra and gpt-5.6-sol on Amazon Bedrock against gpt-5.4-mini and gpt-5.4-nano on the OpenAI API, picked as the cost-optimized baselines many teams already run.
- The Bedrock models ran with reasoning disabled while the OpenAI API baselines ran at their defaults, which AWS describes as a comparison of practical deployment configurations.
- On accuracy, sol solved 75 percent of AIME problems to mini's 37 percent, and led GPQA Diamond 68 percent to 43 and MMLU-Pro 82 percent to 59.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint A per-token list price cannot rank these models on its own: the cheaper sticker lost on cost per correct answer, so a buyer needs accuracy and token counts from its own tasks to know even the direction of the gap.
- decision Teams currently on mini or nano can settle the upgrade question with a run against their own workload, using the same code path AWS used, instead of arguing over pricing pages.
- exposure Gaps in this table rest on 48 to 198 items with no uncertainty estimates attached, so a team that switches models on a small difference is buying noise and owns the result.
- precedent An 80 percent cut inside one product line means any published cost-per-outcome comparison expires the next time a provider reprices, and the vendor that published it invites the rerun.
Cost per correct answer is two numbers multiplied: the price per token, and the billed tokens spent to get one right answer, wrong attempts included [6]. In the AIME run, luna sat at roughly 1.5 times mini's list price per token and came out 25 percent cheaper per correct answer [10]. Divide 0.75 by 1.5. Luna used about half as many billed tokens per correct answer as mini did in that sample [18]. The reason is in the setup: the Bedrock models ran with reasoning disabled, and the OpenAI API baselines ran at their defaults [4].
That confound is disclosed, not buried. "This is a comparison of practical deployment configurations, not a controlled estimate of intrinsic model capability," AWS wrote [5]. Disabling reasoning is a legitimate deployment choice, and it is also the cheapest way to win a token-count comparison. For the 25 percent to mean anything on your workload, luna with reasoning off has to be accurate enough on your tasks, and you have to be running mini at its defaults today.
Then the list price moved. The July 30, 2026 reduction took luna down 80 percent and terra down 20 percent on Amazon Bedrock [11]. Stacked on the earlier gap, luna's cost per correct AIME answer comes to about 15 percent of mini's [19]. The recorded figure is $0.0021 for luna [12]. If mini's pricing did not change, that puts mini near $0.014 per correct answer in the same sample [20].
The harness itself is careful work. One Responses API client drives all five models, with the backend and model ID swapped while the evaluation logic is held constant [2]. Grading combines deterministic checks with a judge model, gpt-5.5, which is not one of the five under test, and the judge prompts are frozen with their hashes written into every result file [13]. Each run writes a timestamped JSON, and every number and chart in the post is generated from those files at build time [14].
The agentic claim is argued from how turns bill: every turn re-sends the growing conversation, so turn count multiplies the invoice [17]. Multi-turn trajectories on live web-research tasks were measured alongside the single-call benchmarks and the rubric-graded deliverables [16]. The figure carrying the ranking inversion is a single-call mathematics benchmark [10]. A per-turn curve depends on your tool loop, your context management and how often your agent re-fetches a document it already read. The post asks for that directly: reproduce the evaluation on your own workload before selecting a model [15].
The retry estimate needs the same treatment. At mini's AIME accuracy, independent attempts would average about 2.7 tries per success [8]. "Real retries are correlated, so measure your actual retry strategy rather than assuming that estimate transfers directly," AWS wrote [9].
What to watch
- Turn-level trajectory costs from the web-research tasks, which would show whether the ranking inversion survives multi-turn agent workloads.
- A rerun with reasoning enabled on the Bedrock models, separating the deployment configuration from the model.
- Further Bedrock price moves, since another cut or an increase re-ranks the cost-per-correct-answer table without any model changing.