A cached COPY --from=builder shipped yesterday's artifact under today's tag

A developer writing on dev.to reports tagging an image from current source and finding that the compiled `openapi.so` inside it was a day old ^[1][2]. The failure did not raise an error at any step, which means the tag was accurate about the commit and wrong about the contents, and nothing in the pipeline said so ^[5].

According to the account, the builder stage was not at fault: the artifact it produced matched what the current source should compile to, byte for byte ^[3]. The final image got the wrong bytes because the runtime stage's `COPY --from=builder` took a layer cache hit and carried the previous build's output forward ^[4]. Rebuilding the same commit produced a different, correct image, which the author calls the worst available outcome, since a reproducible failure can be chased and a self-healing one guarantees the next occurrence is just as invisible ^[6][7].

The uncomfortable part is that Docker's documentation says this should not happen this way. For most instructions the cache key is the instruction text, and the docs are explicit that when processing a `RUN apt-get -y update` command the updated files in the container are not examined, only the command string ^[8]. `ADD` and `COPY`, and `RUN` with a bind mount, are the documented exception: the checksum comes from file metadata rather than the instruction string, with modification time deliberately excluded ^[9]. Source had changed, so the metadata should have moved with it and the cache should have missed ^[10]. The author says the mtime exclusion is the only thread available and declines to build an explanation on it ^[11].

What replaced the explanation was a check that stops trusting the tag: for each module, pull the compiled artifact out of the image, hash the source file it should have come from, compare against the compilation cache entry that source must have produced, and stop the build before anything is pushed if they disagree ^[12].

Weeks later, a release existed to change exactly one module, and the check printed `OK: <image> carries the current source (3 modules checked)` ^[13]. The three modules it checked were three other ones; the module the release was for was not among them ^[14]. The witness list is three names in a hand-maintained tuple against a package of 65 modules ^[15], which leaves 62 modules unwitnessed and coverage at about 4.6 percent ^[21][22].

The comment next to that tuple is not careless. It says the list is small on purpose, that these are the modules whose staleness would be both invisible and expensive, and that widening it would cost time for little gain because one stale layer stales the whole `COPY` ^[16]. That last clause is load-bearing. If staleness always arrives a layer at a time, three witnesses are as good as 65 and the sampling is sound; if staleness can ever land on one module while its neighbours stay current, three witnesses are three lottery tickets ^[17]. The check has never tested that assumption and prints the same word either way ^[18].

The author's remedy is fault injection in both directions and run twice: one run proves the checker can fire, the second shows what it is blind to ^[19]. The pasted example does exactly that, with a two-name witness tuple and three cases: unchanged, stale on the list, and stale off the list ^[20].

Two things to watch in your own pipelines. First, whether the whole-layer assumption holds when you deliberately stale a single module, because that is a cheap experiment and the entire sampling argument depends on the answer ^[17]. Second, whether your green output reports a count next to the word OK, and whether anyone reads the count ^[13][14].