BuildNot yet confirmed elsewhere1 publisher2 min readPublished
Grok 4.7 on Bedrock misses 3 of 18 agent runs at low effort and none at xhigh
AWS ran Grok 4.7 on Bedrock through 72 agent runs and found xhigh effort right 18 of 18 times, against 15 of 18 at low. The post says low used about an eighth of the output tokens, so effort can be tuned per request against the cost of a wrong answer.
The Engineer · Build desk
Three figures: if the post's figure is measured against xhigh, a correct answer at low effort takes about 15 percent of xhigh's output tokens, which is unconfirmed; low effort scored about 83 percent correct; with 18 runs per level, one run moves a level's score about 5.6 points.
| Figure | Value | Note | Claim |
|---|---|---|---|
| Output tokens per correct answer at low, vs xhigh | 15% about | unconfirmed; if measured against xhigh | 20 |
| Correct at low effort | 83% about | 17 | |
| Score moved by one run | 5.6 points, about | 19 |
What happened
- The three tasks were an outage diagnosis, a maintenance schedule under interacting constraints, and a budgeted migration choice, each with one answer verified by brute force.
- Across every effort level, the agent answered 67 of the 72 runs correctly.
- Grok 4.7 is reached through a cross-Region inference profile on the bedrock-runtime endpoint, and requests name one of two profiles instead of a bare model ID.
- On the Converse API, reasoning effort goes in additionalModelRequestFields, and inferenceConfig takes maxTokens only.
- The IAM identity needs bedrock:InvokeModel on both the inference profile and the account's default project, and the default-project permission is required in addition.
Why it matters
- decision Teams running long-horizon agents on Bedrock get a measured case for routing by stakes: low where a wrong answer is cheap to catch, xhigh where it is not.
- cost If xhigh is the baseline, each correct answer at low takes about 15 percent of xhigh's output tokens. That gap is what a team gives up 3 misses in 18 runs to get.
- constraint With 18 runs per level, a single run moves a level's score by 5.6 points, so the gap between 83 and 100 percent rests on three runs.
The 72 runs are 3 tasks times 6 repeats times 4 effort levels [13]. Each level therefore got 18 runs, which fits the xhigh and low counts the post reports [6][7]. Only those two levels are named in the post's summary. Five runs missed in total [14]. Low accounts for three [15] and xhigh for none [6], so the two unnamed levels together missed two [16].
Low scored 83 percent to xhigh's 100 [17]. The post's summary figures are totals, so the split by task is not visible. If all three low misses landed on one task, that task went 3 of 6 at low [18]. Spread across tasks, the same three misses would look milder.
AWS says low used about an eighth of the output tokens [7]. The summary sentence does not name its baseline. It sits beside the xhigh result, and we read it that way. AWS reports token counts instead of prices because prices change and counts are what a team's own costs derive from [8]. For a result meant to be rerun, that is the right unit.
SpaceXAI says the model is trained to work longer on harder tasks and to know when to check its own work [2]. The test scores final answers by effort level. It does not isolate self-checking.
For the numbers to transfer, your tasks need the properties of these three. They require search rather than lookup, and each has one answer that a brute-force check can verify [3]. We'd expect the gap between levels to narrow on lookup-shaped work. The post says it also shows how to measure effort on your own tasks and how to monitor it in production [12]. In our view the expensive part of that is the grader: a scheduling puzzle has a brute-force checker, and much of what production agents produce does not.
What to watch
- Per-task and per-level results for the two unnamed effort levels, which would show whether the three low-effort misses cluster on one task.
- An independent rerun on tasks without a brute-force checker, where correctness has to be graded some other way.
- Grok 4.7 token prices on the Bedrock pricing page, which would turn the output-token ratio into a cost per correct answer.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence50
- Adoption
- Insufficient
- Hype gap+10
- Incentives55
- Confidence60
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [2]
According to SpaceXAI, Grok 4.7 is trained to work longer on more difficult tasks and knows when to check its own work.
- [3]
AWS evaluated the agent on three tasks that require search rather than lookup: diagnosing a two-service outage from its alerts, scheduling maintenance under interacting constraints, and choosing an optimal set of migrations under a budget. Each task has exactly one correct answer, verified by brute force.
- [4]
AWS ran all three tasks at every reasoning effort level, six times each, for 72 agent runs.
- [6]
At xhigh reasoning effort the agent answered all 18 runs correctly.
- [7]
At low reasoning effort the agent answered 15 of 18 runs correctly while using about an eighth of the output tokens.
- [8]
AWS reports token counts rather than currency because token prices change and the counts are what a team's own costs are derived from, so a measurement in tokens stays valid.
- [9]
Grok 4.7 runs on the bedrock-runtime endpoint and is accessed through a cross-Region inference profile; two profiles are available, and requests name one of the inference profiles rather than a bare model ID.
- [10]
On the Converse API, in inferenceConfig set maxTokens only, and put model-specific parameters such as reasoning effort in additionalModelRequestFields.
- [11]
The IAM identity needs bedrock:InvokeModel on both the inference profile (arn:aws:bedrock:{region}::inference-profile/us.xai.grok-4.7 or the global. equivalent) and the account's default project (arn:aws:bedrock:{region}:{account-id}:project/default); the default project permission is required in addition to the inference profile.
- [12]
The post says it shows how to measure the trade-off on your own tasks and how to monitor it in production.
- [13]
The test covered four reasoning effort levels, with 18 runs per level.
- [14]
Five of the 72 runs were answered incorrectly.
- [15]
Three of the 18 runs at low effort were answered incorrectly.
- [16]
The two unnamed effort levels together missed two runs.
- [17]
Low effort scored about 83 percent correct.
- [18]
If all three low-effort misses fell on one task, that task scored 3 of 6 correct at low effort.
- [19]
With 18 runs per level, one run moves a level's score by about 5.6 percentage points.
- [20]
If the eighth in the post is measured against xhigh, a correct answer at low takes about 15 percent of xhigh's output tokens.
Sources
1 independent publisher whose own reporting we read for this story.
- aws.amazon.comBuild long-horizon agents with Grok 4.7 on Amazon Bedrock
1 article · October 13, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.