Product1 publisher2 min readPublished
Fireworks' own DeepSWE numbers put four coding models inside the noise band
The vendor selling the cheapest model in the comparison reports a 0.7-point quality spread across four frontier models against run-to-run variation of 1.4 to 3.2 points. That leaves price per task, $0.43 against an implied $6.45 for GPT-6 Astra.
The Product Desk · Product desk
What happened
- Fireworks published benchmark results for DeepSeek-V4.1-Flash claiming GPT-6 Astra-level coding accuracy on DeepSWE at one fifteenth of the price per task.
- The post reports that all four models tested land within 0.7 points on DeepSWE pass@1, while Fireworks' own run-to-run variation on the benchmark is 1.4 to 3.2 points.
- Fireworks puts DeepSeek-V4.1-Flash at $0.43 per task, which it says is 5.5x below Gemini 3.8 Flash, 15x below GPT-6 Astra and 28x below Claude Opus 5.
Compiled by The Product DeskSomething wrong?How this is made
Why it matters
- decision A team comparing coding models cannot rank this group on the published quality numbers, because the spread between them is smaller than the noise in the measurement, so the choice falls to price per task and to whatever else the team can measure itself.
- constraint The 15x saving depends on an agent loop that re-reads the same context until 99.6 percent of input tokens are cache hits, so a workload with fresh context on every call will not see the same discount.
- capability Fireworks says $0.43 a task buys 15 autonomous coding attempts for the price of one Astra call. Designs that try three times and keep the patch that passes tests become affordable at that rate.
- contradiction Fireworks sells the model as a new pareto frontier across quality, speed and cost while its own variance figures say the quality axis is a tie, and the cost multiple drops from 15x to 12x when the benchmark changes.
Anyone picking a model for a nightly bug sweep can skip most of the Fireworks post. "On quality, these four models are the same class," the company wrote of its DeepSWE results [4]. The 0.7-point spread it measured sits between two and four and a half times inside the 1.4-to-3.2-point run-to-run variation from the same harness [1].
The run itself is worth a closer look. Fireworks reports 36.9M input tokens per task against 211K output tokens, a ratio of 174 to 1 [8]. Of those input tokens, 99.6 percent were cache hits, and cache-hit charges came to 60 percent of the bill [9]. Everything the model wrote, plus the 0.4 percent of input that missed cache, accounts for the other 40 percent [3]. At $0.43 per task across 36.9M input tokens, the job bills out at roughly 1.2 cents per million input tokens [4].
That price comes out of the cache behaviour. The model is a 552B-parameter mixture of experts with an encoder-decoder split, 8B active parameters for input and 16B for output [7], and Fireworks says its KV cache consumes a quarter of the HBM and an eighth of the SSD storage of DeepSeek V4 [10].
Change the shape of the workload and the multiple moves. On Terminal-Bench 2.1 the gap is one point on quality and 12x on cost per task, down from 15x on DeepSWE, because DeepSeek-V4.1-Flash uses roughly four times more output tokens than Astra there [11]. All of these figures come from one vendor's harness on its own platform, and the post lists speed among the tradeoffs it claims without publishing per-task latency [1][13].
A team can run its own eval twice against the same model with the same prompts and measure the spread. If the gap between your candidates is smaller than your own seed-to-seed variance, you are choosing on price whether you say so or not. The other question is narrower: does a bad patch get caught by tests before a human reads it? Cheap retries pay only where the retry is automatic and the verdict is machine-checkable, because three attempts at $0.43 still cost less than one at the price Fireworks' 15x implies [5][2]. On a sweep of 1,000 tasks a night, that is $430 against $6,450 [5].
What to watch
- An independent DeepSWE rerun across multiple seeds would show whether the 0.7-point spread holds outside Fireworks' harness.
- Per-task wall-clock numbers for the same four models, given that DeepSeek-V4.1-Flash writes roughly four times more output on Terminal-Bench.
- The Humanity's Last Exam results Fireworks says it ran as a third benchmark, which would test whether the same-class finding survives off coding tasks.