BuildNot yet confirmed elsewhere1 publisher3 min readPublished
Firecracker won't run on a Mac, so Encore rebuilt the Linux image toolchain to match
Encore's crackling drives Firecracker on Linux and Apple's hypervisor on macOS. Getting one set of images to boot on both meant re-implementing much of what Docker and a Linux host do for free.
The Engineer · Build desk
What happened
- Encore has run every one of its backend builds inside a Firecracker microVM since mid-2022.
- Firecracker drives KVM and needs a Linux host with /dev/kvm, which no Mac has, and most Encore engineers develop on Macs.
- For four years, changing the build system meant doing it on a machine other than your own.
Why it matters
- cost Each guest-side edit was billed as three trips to a remote box: ship the layers, squash them there, restart the container. The engineer paid that toll per iteration, not per release.
- exposure The development path wrapped Firecracker in a privileged container holding /dev/kvm and /dev/net/tun, so the boundary around the microVM on the dev host was not the boundary production relies on.
- constraint One shared host means port allocation lives in a gitignored per-engineer file, so the environment cannot be reproduced by cloning the repository and engineers have to negotiate to avoid collisions.
- precedent Parity with a hypervisor that will not travel now looks like a second maintained backend rather than a patch, which is the bill any other Firecracker shop on Macs should expect to be handed.
The expensive half was never the Go binaries. Those cross-compiled with `GOOS=linux GOARCH=amd64`, rsynced to the build host, and Encore decided whether a restart was needed by counting the transferred files [9]. Images were the problem, because Firecracker boots a block device while Docker produces layers, and Encore says it could not find an existing tool that turned one into the other [10]. So the conversion got written in house: `docker save`, explode the tarball locally, rsync the layers and manifest across, then pipe a shell function into a login shell on the far end and run it there [11].
What that function does is a list of jobs a container runtime normally handles. It re-extracted every layer in manifest order and deleted the `.wh..wh..opq` whiteout markers with `find`, because `tar` will not apply them [12]. It wrote a hardcoded `/etc/resolv.conf`, since the VM had no DNS otherwise [13]. It lifted the image's environment variables out of the Docker config with `jq` [14], then called `mksquashfs` to produce something Firecracker could boot [15]. There was a cache keyed on the Docker image id to skip the whole path on a match [16], which is least useful in the case that matters: the conversion ran whenever anything in the image changed, which was most of the time if you were working on the guest side [17].
Underneath that, the dev host ran Firecracker inside a Docker container, which meant `--privileged` plus `/dev/kvm` and `/dev/net/tun` passed through, because the process inside was going to make tap devices and boot VMs of its own [19]. Firecracker wants those taps on a host bridge, and a container has no host bridge, so a script created a `docker0` bridge and made `eth0` its member before the build service came up [20].
The two spans Encore gives leave no gap between them, so for the entire life of the production hypervisor, nobody worked on the build system on the machine they were typing on [23]. Each engineer instead got a personal user on one shared box in a datacentre, reached over Tailscale [7]. The shell was eventually rewritten in Go, but the pipeline and the host stayed the same [21], which is the tell: this was not a stopgap anybody expected to remove, it was infrastructure.
The fact that carries the story is the declined port. According to Encore, a working proof of concept on Apple's Virtualization.framework existed and the maintainers turned it down [24], so the work of making a Mac boot the same artefact landed on the people who needed it. Encore's answer is a single microVM API over two hypervisors, and the entry fee was rebuilding much of the Linux image toolchain to run on macOS [1]. Firecracker's minimalism is the point of it: only what a Linux kernel needs [3]. The cost of that minimalism does not disappear when a shop runs Macs. It moves into the shop's own repo, where it now has to be maintained twice over.
What to watch
- Whether crackling is released as a standalone component with a stable API, or stays welded into Encore's build service.
- Any movement from Firecracker's maintainers on macOS or a non-KVM backend, which would strand the downstream work.
- Whether the shared datacentre build box gets retired now that laptops can boot the same images, and what happens to the per-engineer port config.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence56
- Adoption30
- Hype gap+12
- Incentives68
- Confidence60
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Encore built crackling, a single microVM API that drives Firecracker on Linux and Apple's hypervisor on macOS; booting the same images on both required rebuilding much of the Linux image toolchain to run on macOS.
- [2]
Encore builds and deploys backend applications, and since mid-2022 every one of those builds has run inside a Firecracker microVM.
- [3]
Firecracker strips the emulated hardware down to what a Linux kernel needs, giving each build the isolation of a virtual machine with startup close to the cost of a container.
- [4]
Firecracker drives KVM, so it needs a Linux host with /dev/kvm, which no Mac has, and most engineers at Encore develop on a Mac.
- [5]
For four years, working on Encore's build system meant working on it somewhere other than the engineer's own machine.
- [6]
Onboarding was a one-time script that SSHed into the shared build machine as root, pulled the engineer's public key from their GitHub keys URL, created a user, added them to the kvm and docker groups, copied VM images into ~/images and hard-linked the firecracker binary into ~/binaries, since every user needed it under their own tree on the one box.
- [7]
Each engineer ended up with a personal environment in a datacentre, reachable over Tailscale, sitting next to everybody else's.
- [8]
A second script read the engineer's username and port out of CUE config in a gitignored per-engineer file, because everyone shared that host and had to agree not to collide.
- [9]
Binaries were cross-compiled with GOOS=linux GOARCH=amd64, rsynced across, and the number of transferred files was counted to work out whether anything needed restarting.
- [10]
Images were the hard half because Firecracker boots a block device and Docker produces layers; Encore could not find an existing tool that converted Docker layers into a block device Firecracker could boot, so it built the conversion itself, half on the laptop and half over SSH.
- [11]
The pipeline ran docker save, extracted the layers and manifest.json from the tar locally, rsynced them to the build host, then piped a shell function (squash_layers) into a login shell over SSH to run it on the far end.
- [12]
squash_layers re-extracted every layer in manifest order and deleted the .wh..wh..opq whiteout markers with find, because tar will not apply them.
- [13]
The conversion wrote a hardcoded /etc/resolv.conf because the VM had no DNS otherwise.
- [14]
The conversion pulled the image's environment variables out of the Docker config with jq.
- [15]
The conversion finished by calling mksquashfs to produce something Firecracker could boot.
- [16]
The cache was keyed on the Docker image id so the whole conversion path could be skipped when it matched.
- [17]
The conversion ran whenever anything in the image had changed, which was most of the time if the engineer was working on the guest side.
- [18]
Restarting took a third SSH, to kill the engineer's container and start its replacement with docker run.
- [19]
Firecracker ran inside a Docker container on the dev host, so the container was run --privileged with /dev/kvm and /dev/net/tun passed through, because the process inside was going to create tap devices and boot virtual machines of its own.
- [20]
Firecracker expects tap devices to attach to a host bridge and a Docker container has none, so a shell script created a docker0 bridge and set eth0 as its member before the buildsvc build service came up.
- [21]
The bash was later rewritten in Go, but the pipeline and the host did not change.
- [22]
A single guest-side change crossed the network at least three times: an rsync of the layers to the build host, an SSH to squash them there, and a third SSH to restart the container.
- [23]
The four-year period of remote-only work on the build system covers essentially the whole life of the mid-2022 Firecracker deployment, so no Encore engineer edited the build system locally while Firecracker was in production, until crackling.
- [24]
According to Encore, Firecracker's maintainers turned down a working proof of concept built on Apple's Virtualization.framework and said they do not plan to support macOS any time soon.
ReportedInsufficientSource: Encore blog post2 sources— create a free account to open themView cited source
Sources
1 independent publisher whose own reporting we read for this story.
- We rebuilt the Linux microVM stack on Apple Silicon
encore.dev
1 article · August 22, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.