Build1 publisher3 min readPublished
Blacksmith enforces its org-only rule by leaving the job queued forever
An unaffiliated 10-trial benchmark clocked Blacksmith's runners at 3.17x and 3.77x faster than ubuntu-latest, with no overlap in the ranges. Reaching that baseline first meant moving the repository off a personal account.
The Engineer · Build desk

What happened
- An unaffiliated 10-trial comparison of the same job on ubuntu-latest and blacksmith-4vcpu-ubuntu-2404 came out 3.17x and 3.77x faster on Blacksmith, with the two ranges never overlapping.
- The benchmark could not start on the author's personal repository, because Blacksmith refuses to run there and the job sat in queued with no error and no explanation.
- Moving the repo to a fresh organization and installing Blacksmith's GitHub App there unqueued the job within seconds.
- Blacksmith's own Migration Wizard pull request set the runner tag to blacksmith-4vcpu-ubuntu-2204, an image its infrastructure did not have, so the job queued forever a second time.
- The author says the harness is one small, CPU-light Node script run 10 times, not a real build.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Trying Blacksmith on a side project means creating an organization and moving the repo first, so the free 3,000 minutes a month and card-free signup do not reach anyone whose CI lives on a personal account.
- exposure Installing the app puts a pull request on the default branch without being asked, so teams that gate automated commits have a governance question to answer at install time, not later.
- decision Anyone quoting the 3.17x has to check their pipeline looks like the test: a small CPU-bound process run repeatedly, with installs, image builds and slow suites explicitly outside what was measured.
Blacksmith's quickstart mentions "organization" 33 times and "personal repo" twice, and the second of those two mentions is where it says the product is unavailable for personal repositories [9]. So the rule is documented; the enforcement is a job the scheduler will never place, sitting in `queued`, which is also what waiting for capacity looks like.
The next failure mode was a cancellation. On a later run both jobs started, and the Blacksmith job was killed after about five minutes with no steps executed, which the author read as an account-verification hiccup rather than anything about the runner image [14]. "None of these were show-stoppers, but if you're evaluating Blacksmith on a deadline, budget for at least one queued-forever mystery and one silently-cancelled run before you get a clean baseline," the author wrote [15].
The timing is better built than most runner comparisons. Each trial was measured with `date +%s%N` around the `node` invocation instead of bash's `time` builtin or the wall-clock job duration GitHub reports, and one untimed warmup run was discarded before the loop started [16][2]. The workflow file and the raw run logs are public, at `.github/workflows/blacksmith-benchmark.yml` and `.github/scripts/run-trials.sh` in github.com/Shed-engineering/gzip-threshold-json-api [19].
The result that survives hardest is the separation. On client.js, the slowest of Blacksmith's ten trials came in at 166ms and the fastest of the github-hosted trials at 368ms [17]. Take the comparison most favourable to the shared runner and Blacksmith is still 2.2x ahead [1].
The narrower spread is a weaker finding than it looks. Blacksmith's ranges were 51ms and 14ms wide against github-hosted's 100ms and 51ms [18]. Absolute width falls with runtime, so a runner three times faster starts ahead on that measure even when its variability is identical. If the 51ms width is the client.js pair, it is 31 percent of that job's slowest trial, against 27 percent for the 100ms width on trials no faster than 368ms [2].
For 3.17x to transfer, the workload has to be what this one is: a small CPU-bound process invoked over and over. Blacksmith runs Firecracker microVMs on bare metal with gaming CPUs, and `runs-on: blacksmith-...` is the entire migration [5]. Dependency installs spend their time on network and disk, where a faster core moves the total less. The author says a workflow dominated by dependency installs, Docker layer builds or a slow test suite could land at a very different ratio, better or worse [20].
The company raised a $45M Series B led by Peak XV Partners at a $550M valuation in March 2026, on top of a $3.5M seed from GV and Y Combinator [6]. It claims more than 6,000 companies running CI on it, has one independent dev.to write-up about cutting CI costs, and otherwise appears in generic runner comparison listicles next to Depot and WarpBuild, with no Show HN or Launch HN [7]. The free tier is 3,000 minutes a month with no credit card required at signup [11].
What to watch
- Whether Blacksmith returns an error on a personal-repo install instead of holding the job in queued.
- Whether a 22.04 image appears on Blacksmith's infrastructure, or the Migration Wizard stops writing the blacksmith-4vcpu-ubuntu-2204 tag.
- A comparable 10-trial run on an install-heavy or Docker-heavy workflow, the case this harness does not cover.