BuildNot yet confirmed elsewhere1 publisher3 min readPublished
A 58% flake in a shared disk cache was a real race, found only when Linux-only CI met Windows
Seven failures in twelve runs of one Vesuvius Challenge test exposed three PermissionError surfaces in a chunk cache, and one of them under-evicts on every operating system.
The Engineer · Build desk
What happened
- Running one vesuvius test twelve times on Windows 11 produced seven failures, every one a PermissionError: [WinError 5] Access is denied.
- The ScrollPrize/villa CI matrix tests Ubuntu only, with a workflow note asking for the list to be extended once macOS and Windows build scripts are confirmed.
- The component involved backs the package's documented volume_cache_dir setting and the --cache-dir flag on its inference CLI.
- After the change, the Windows suite moved from 47 passed and 2 failed to 52 passed and 0 failed.
Why it matters
- exposure By the author's reading, any Windows training run using a DataLoader with num_workers above zero sat in this exact concurrency pattern and could die mid-epoch, which is expensive when the epoch is...
- constraint The eviction accounting error means the byte ceiling operators size their disks against was never actually enforced, on Linux as much as on Windows, so cache directories could grow past budget...
- capability Forcing each PermissionError surface by monkeypatch rather than by racing gives the project a test it can run on its Ubuntu-only CI, where the underlying race is invisible.
- precedent With the upstream zarr issue unresolved, the working assumption for downstream packages is that they must absorb Windows rename semantics in their own wrappers rather than wait for the library.
Zarr commits each cache entry by writing a temporary file and calling os.replace [15]. On POSIX, rename(2) over a file another process holds open is legal; on Windows, MoveFileEx with MOVEFILE_REPLACE_EXISTING returns ERROR_ACCESS_DENIED [16]. Content addressing turns that from an occasional collision into the steady state, because four workers reading the same volume derive the same keys and try to commit the same entries [17]. Repair the write path and the reader's open gets denied by somebody else's in-flight commit instead [18].
The third surface is the one that travels. The package's own LRU sweep caught only FileNotFoundError when deleting old entries, Windows raises PermissionError for a file still in use, and the sweep debited the file's bytes from the size budget anyway [19]. A cache whose entire job is staying under a byte ceiling was therefore under-evicting while its accounting insisted it had freed space, and that arithmetic is wrong on Linux too [19].
The rates deserve more attention than the headline percentage. Seven failures in twelve runs is 58.3% [3][13], which means a single-run CI job comes back green roughly 42% of the time [14]. That is the figure that explains how this lived in a hot-path component [4] without anyone filing it. After the fix, twelve consecutive runs were clean [7]. Treating runs as independent at the original rate, twelve greens in a row would turn up about once in 37,000 attempts [28], which is what makes that a verification rather than a lucky afternoon.
The remedy declines to branch on platform at all [20]. Refused writes become no-ops logged at debug, unreadable entries report as a cache miss so the outer cache refetches from source [20], and the sweep now skips undeletable entries without crediting their bytes and evicts the next-oldest [21]. The bill for that is stated rather than hidden: a full disk produces slow refetches instead of a crash [22]. The suite arithmetic is also tidy, 47 passed plus 2 failed is 49 before, 52 after, so the three new deterministic tests are the whole delta [24].
Two provenance notes. This is one self-reported account written for a contest [26], and the fix is a pull request against villa [10], not a landed upstream repair. The upstream zarr issue has been open since October with three confirmations and no fix [6], so any downstream project touching that store owns the rename semantics itself for now.
There is also a reason nobody had run this suite off Linux: uv sync --extra all is unresolvable on Windows and macOS because the CUDA-only cucim-cu13 dependency ships manylinux wheels only [25]. The same trip turned up an LRU test that stamped files 1 to 3 nanoseconds apart, which NTFS at 100 ns resolution collapses to st_mtime_ns == 0, destroying the ordering the test asserts; ext4 resolves to 1 ns, so Linux CI never noticed [23].
What to watch
- Whether zarr-python#3522 gets a fix, which would let downstream projects drop their own store wrappers.
- Whether villa adds Windows and macOS CI runners, and whether the manylinux-only cucim-cu13 wheel keeps blocking that.
- Whether PR 1545 lands as written, including the silent degrade-to-refetch behaviour on refused cache writes.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence58
- Adoption22
- Hype gap+12
- Incentives55
- Confidence52
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
The author set up the vesuvius package on a Windows 11 machine, working with an AI coding assistant to run the test suite on a platform it had never been tested on; one test failed, then passed, then failed again.
- [2]
The test test_shared_cache_multiprocess_reads_are_not_torn spawns four processes that read one scroll volume through a shared on-disk chunk cache.
- [3]
Run twelve times on Windows, the test produced 7 failures out of 12, all PermissionError: [WinError 5] Access is denied.
- [4]
The cache is on the hot path for real usage: it is the component behind the package's documented volume_cache_dir config and the --cache-dir flag of its inference CLI.
- [5]
According to the writeup, any PyTorch DataLoader with num_workers > 0 puts multiple processes into exactly this concurrent pattern, so on Windows training runs would randomly die mid-epoch.
- [6]
The commit-path behaviour is upstream issue zarr-developers/zarr-python#3522, open since October with three confirmations and no fix.
- [7]
Measured result after the fix: 0 failures in 12 consecutive runs, down from 58% failures.
- [8]
The package's Windows test suite went from 47 passed / 2 failed to 52 passed / 0 failed.
- [9]
Three deterministic regression tests were added that force each PermissionError surface via monkeypatching instead of racing for it, each verified to fail against the unfixed code on any OS.
- [10]
The fix, tests and methodology are in pull request ScrollPrize/villa#1545, touching three files: the cache wrapper and eviction fix in vesuvius/src/vesuvius/ink_detection/volume_io.py, regression tests in vesuvius/tests/ink_detection/test_volume_io.py, and a platform marker in pyproject.toml.
- [11]
The Vesuvius Challenge's open-source monorepo ScrollPrize/villa contains the vesuvius Python package, which researchers use to stream multi-terabyte CT scan volumes and train ink-detection models.
- [12]
The villa project's CI only tests Ubuntu, and the workflow file says "Extend this list once the build scripts for macOS and Windows are confirmed".
- [13]
Seven failures in twelve runs is a 58.3% per-run failure rate.
- [14]
At the observed rate, a single run of the test passes about 42% of the time (5 of 12).
- [15]
The zarr library commits each cache entry with a write-temp-then-os.replace pattern.
- [16]
On POSIX, rename(2) over a file another process has open is legal; on Windows, MoveFileEx(MOVEFILE_REPLACE_EXISTING) returns ERROR_ACCESS_DENIED.
- [17]
Four workers populating the same content-addressed keys collide constantly.
- [18]
After the write path is fixed, a second surface appears: a concurrent commit can deny the reader's open.
- [19]
The package's LRU eviction sweep caught only FileNotFoundError when deleting old entries; Windows raises PermissionError for in-use files, and the sweep then subtracted the file's bytes from the size budget anyway, under-evicting a cache whose job is staying under a byte budget. The author calls this a genuine bug on every operating system.
- [20]
The fix wraps only the cache-side store: refused writes degrade to no-ops logged at debug, unreadable entries report as a miss so the outer cache refetches from source, and there are no platform-specific branches.
- [21]
The eviction sweep now skips undeletable entries without crediting their bytes and evicts the next-oldest instead.
- [22]
The documented trade-off is that a full disk becomes slow refetches rather than a crash.
- [23]
The LRU test stamped files with 1-3 nanosecond timestamps, which NTFS at 100 ns resolution collapses to st_mtime_ns == 0, silently destroying the ordering the test depends on; ext4 has 1 ns resolution, which is why Linux CI never noticed.
- [24]
The suite held 49 tests before the fix (47 passed plus 2 failed) and 52 after, so the three added regression tests account for the entire increase.
- [25]
The CUDA-only cucim-cu13 dependency ships manylinux wheels only, which made uv sync --extra all unresolvable on Windows and macOS.
- [26]
The writeup is a submission for DEV's Summer Bug Smash: Clear the Lineup, powered by Sentry.
- [27]
The author's framing: "A 58% flake isn't a flake. It's a bug with a coin flip attached."
- [28]
If the race were unchanged and runs independent, twelve consecutive clean runs would occur about once in 37,000 attempts.
Sources
1 independent publisher whose own reporting we read for this story.
- dev.toThe flaky test was right: a 58%-reproducible race in a scroll-reading pipeline's disk cache
1 article · August 21, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.