Build1 publisher2 min readPublished
Dirtying all of a 64 GB heap during a fork snapshot costs 16.8 million page faults
A dev.to walkthrough traces fork() from the cleared write bit in every PTE down to do_wp_page, where a refcount check decides whether a snapshot window costs a 4 KiB copy for each dirtied page or nothing at all.
The Engineer · Build desk
What happened
- A dev.to walkthrough describes fork() as copying the page table hierarchy and clearing the write bit in every entry, leaving parent and child pointing at the same physical frames.
- A store into a shared page raises a fault, and the kernel allocates a fresh 4 KiB frame, memcpy's 4096 bytes, updates the entry and sends an inter-processor interrupt to invalidate TLBs.
- The same handler has a cheap exit: when the frame's refcount is back to 1, it restores the write bit and returns to ring 3 without copying any data.
- Background persistence in Redis and similar caches forks to take point-in-time snapshots while the event loop keeps serving clients, on the stated assumption that fork costs near-constant time.
- The post's worked example is a 64 GB resident dataset under sustained writes, and it attributes trap-handling avalanche, TLB thrashing and memory inflation to the writes that follow the fork.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Sizing a host for a forking snapshot means budgeting for the dirtied share of the heap on top of the heap, with twice resident as the ceiling.
- cost The per-page trap and copy is charged to the parent's write path, so the snapshot's bill arrives as tail latency on live client requests, on whichever core took the fault.
- decision The choice this forces is when to fork: a window picked for a low store rate costs far fewer copies than a window picked for operator convenience.
- contradiction Mechanism and evidence sit at different levels here, because the call chain can be verified against the kernel tree while the 64 GB OOM cannot be verified without a fault count from your own host.
Take the post's 64 GB example and count the events. At 4 KiB per page, a 64 GB resident set is 16,777,216 pages [14]. If the parent dirties all of them while the child is still alive, the kernel runs roughly 16.8 million faults, each one an allocation, a 4096-byte memcpy, a PTE update and an inter-processor interrupt for the TLB shootdown [4]. The new frames come from system RAM, so the worst case needs a second 64 GB, 128 GB in total [15].
That worst case is not the normal case, and the diagram says why. do_wp_page has two exits [3]. When the frame's refcount is above 1, the kernel allocates and copies [4]. When the refcount is 1, it restores the write bit and returns to ring 3 with no data copied [5]. The number of copies is therefore the number of pages written while the frame is still shared, and pages the parent never stores to during the snapshot window cost nothing beyond the page table walk fork() already did.
The fault path in the diagram starts at a store instruction [3]. A snapshot child that only reads its inherited pages never triggers a copy from its own side, so the copies come from the parent's writes. That is also where the latency lands, because the faulting store sits in the event loop serving live clients [10].
For the 64 GB failure to transfer to your host you need two numbers: the pages the parent dirties per second during the window, and how long the window lasts. The post does not include any measurements [16]. Their product, compared against free RAM plus swap, is what decides whether the fault cascade exhausts swap and reaches the OOM killer [13].
There is a cost before any write happens. copy_process() calls dup_mm(), and dup_mmap() then iterates every vm_area_struct in the parent and walks the PGD, P4D, PUD, PMD and PTE levels, duplicating the directories while leaving the physical frames alone [7][8][9]. That work scales with the mapped address space, not with how much of it is hot. The "near-constant time with zero footprint" assumption the post names as the operational one [11] is already wrong at the syscall, before the first trap.
Credit where it is due: the write-up names the register and the symbols at every step, the CR0.WP violation, vector 14, do_page_fault, do_wp_page, and the CR3 load on context switch [3][9], so a reader can check it against the tree. The prose around the diagram reaches for "catastrophic eviction loops" [17]; the diagram is the part you can grep for.
What to watch
- Whether anyone publishes fault counts and window durations for a real 64 GB forking snapshot instead of a worked example.
- Whether the walkthrough's truncated per-PTE section gets completed, since that is where the fork-time cost is quantified.
- Whether a follow-up covers page sizes other than the 4 KiB PTE case the diagram assumes.