Build1 publisher3 min readPublished
containerd 2.2's mount manager runs slower than the shell steps it bundles
containerd 2.2's mount manager took 29.6 to 47.2 ms per Activate call in a dev.to benchmark, against 23.5 to 25.9 ms for truncate, mkfs, losetup and mount. Most of the gap is design overhead, and a panic plus an orphaned loop device from the same runs are the better reason for shim authors to wait.
The Engineer · Build desk

What happened
- Teardown through Deactivate plus umount took 32 to 45 ms, against 11 to 13 ms for umount and losetup -d run by hand.
- A first manual baseline built with dd took 358 ms to 1.6 s and made the manager look ten to fifty times faster.
- The manager has no ctr subcommand and lives in a Go package meant to be embedded by a snapshotter or a runtime shim.
- The test ran in Docker Engine 29.3.1, whose containerd reports v2.2.2, with the manager package pinned to that version in go.mod.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost On the test host a create-and-teardown cycle through the manager costs roughly 27 to 53 ms more than the shell path, and nodes that churn many loop-backed containers pay it on every one.
- exposure A shim that embeds the manager can end up holding attached loop devices its own bookkeeping cannot locate, so reconciling against losetup falls to the embedder.
- decision Any team judging the manager on speed needs a truncate-based baseline, because a dd baseline turns a small loss into an apparent tenfold win.
- constraint Operators cannot try the manager from ctr; evaluating it means writing Go against the package and pinning its version.
Activate runs the same binaries as the manual path [8]. The mkfs transformer in `core/mount/manager/mkfs.go` creates the image with `os.OpenFile` and `f.Truncate(size)`, then calls the real `mkfs.ext4` or `mkfs.xfs` through `exec.CommandContext` [13][8]. It has no custom fast-path formatter [8]. On top of those steps the manager writes to BoltDB, creates symlinks and keeps its own records, according to the post [9]. The design is sound. A transformer creates and formats the file, a handler attaches it as a loop device, and the result reaches the runtime as an ordinary mount [16]. The author wrote: "The manager's job is composability across snapshotters, not raw speed." [18]
The caller still has to do the last step [7]. Activate returns `info.Active`, the loop attach it handled, and `info.System`, the ext4 mount the caller issues itself with `mount(2)` [7]. The working 200MiB config chains two entries: a `mkfs/loop` mount carrying `X-containerd.mkfs.size=200MiB` and `X-containerd.mkfs.fs=ext4`, then a `format/ext4` mount whose Source is the template `{{ mount 0 }}` [15].
Comparing fastest to fastest and slowest to slowest, the manager adds 6.1 to 21.3 ms per activation [1]. Teardown takes roughly three times as long as the manual path [2]. Both figures come from three runs per path on one machine, each formatting a 200MiB ext4 image [2]. For them to transfer, BoltDB writes and symlink creation on your host would need to cost about what they cost on the author's. The one concurrency result is ten goroutines each activating a 50MiB image in 70.1 ms [14]. That works out to about 7 ms of wall time per activation at a quarter of the image size, so it does not line up with the serial numbers [4].
The dd detour is the most useful part of the post. dd wrote 200MB of real zero bytes before `mkfs.ext4` ran. `truncate -s 200M` makes a file of the same size as a sparse hole in under a millisecond, and `mkfs.ext4` writes only its own metadata [17]. The first baseline was mostly timing the disk [17]. Once the manual path used truncate, as the manager's transformer already does, the tenfold win disappeared and mildly reversed [13][19].
The latency is a design cost the author traced to bookkeeping [9]. The failures are what argue for caution. The post's headline places the panic on a one-mount mkfs chain [4], while the working example used two [15]. I'd expect a library to reject a short chain with an error. The BoltDB error reached the caller raw, and the loop device that stayed attached was one nothing could find again [3]. In my view an embedder should reconcile attached loop devices against its own state and not rely on Activate's error types. Each failure appeared once in the post's testing, so how often they recur is unknown [3].
What to watch
- A containerd fix that turns the one-mount mkfs chain panic into a returned error.
- Whether Deactivate or a later release can find and detach loop devices left behind by a failed Activate.
- A run on another host or image size showing whether the 6 to 21 ms activation overhead stays constant.