Skip to content

Build1 publisher2 min readPublished

Reviving a 2015 crc32 crate pushed its 17 KiB of lookup tables into build.rs

An auditor of abandoned crates.io packages rewrote crc32 from the zlib source, added slicing-by-16 and Python bindings, and shipped the result as crc32-v2. Projects that list crc32 still compile the 2015 code.

The Engineer · Build desk

Illustration accompanying Reviving a 2015 crc32 crate pushed its 17 KiB of lookup tables into build.rs

What happened

  • The crc32 crate on crates.io was last published in 2015, with no changelog, no CI, and a Cargo.toml older than the stable Rust syntax its finder writes every day.
  • He rewrote it from the zlib source with slicing-by-4, -8 and -16 tables, a streaming Digest, a GF(2) combine function, no_std support and Python and Node.js bindings, published as crc32-v2.
  • The crate root sets #![forbid(unsafe_code)], the PyO3 bindings sit behind the python and std features, and 40 integration tests live in tests/byfour.rs.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • decision Anyone who wants the rewrite has to change the dependency name by hand, so no build inherits an unvetted maintainer without a person deciding to trust him.
  • cost Compile-time table generation moves the cost onto every clean build and puts a proc-macro path dependency into the build graph that vendoring and audit policies have to clear.
  • constraint The projects actually shipping the 2015 code are no better off; a fork under a new name cannot patch a dependency tree that still resolves the old name.
  • capability A no_std CRC-32 with unsafe code forbidden at the crate root is usable in firmware and in builds that have to justify every unsafe block, with the Python path kept behind a feature.

Slicing-by-16 needs 17 lookup tables: one 256-entry big-endian table and 16 256-entry little-endian tables at different offsets, which the post works out as 17,408 bytes that have to exist at runtime [7]. Those constants can be committed to git. They can also be built into a static at startup. crc32-v2 generates them during compilation instead: a proc-macro subcrate, crc32-codegen, is invoked from build.rs, generates all 17 tables from the CRC-32 polynomial arithmetic, writes them to $OUT_DIR/crc_tables.rs, and the library include!s the file, so nothing initializes at run time and the tables land in the binary at link time [8].

The whole of build.rs is one call to crc32_codegen::run() [9]. What makes it resolve is a path entry under [build-dependencies] pointing at the subcrate in the same repository [10]. 17,408 bytes is 17 KiB exactly [17]. I would take that trade in a checksum library: the work happens once per clean build, and a proc-macro in the build graph is fine until somebody has to vendor it.

The only throughput figure in the post attaches to the old code. The byte-at-a-time original was "Absolutely not capable of 1 GiB/s throughput", the author wrote [6] [15]. I would expect the wide table variants to show their gain on long buffers and to lose it on short messages, so measure your own input sizes before switching.

CRC-32 is in ZIP integrity checks, Ethernet frames, PNG and every zlib call that computes one internally, according to the post [14]. The author says people were shipping the 2015 crate in their dependency trees and did not know how to reach its maintainer [2]. The crate had no changelog and no CI, and a Cargo.toml older than the stable Rust syntax he uses daily [1].

The post describes publishing the rewrite under the name crc32-v2 and does not report acquiring the crates.io entry for crc32 [16]. A build that lists crc32 today still compiles the 2015 code [18]. Adoption is a deliberate line edit in Cargo.toml, made by a reader who decided a stranger's rewrite beat an unmaintained original.

The rewrite adds a streaming Digest, a crc32_combine() that squares GF(2) matrices in O(log n) [13], PyO3 bindings gated on the python and std features [11], and 40 integration tests in tests/byfour.rs [12]. It ships as no_std, and the crate root sets #![forbid(unsafe_code)] [3] [4]. The 2015 version was three files: a lookup table, a loop, a bitwise XOR [5]. "Because anything worth resurrecting is worth resurrecting properly," the author wrote [20].

What to watch

  • Whether the crates.io entry for crc32 changes hands, since without that no existing dependent picks up the rewrite.
  • Whether #![forbid(unsafe_code)] still holds once the python and std features are on and PyO3 is compiled in.
  • Whether crc32-v2 download counts climb beyond the readers of the post.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories