zstd-stream compresses 4 GB files in the browser using about 35 MB of memory

The 1.6 GB figure follows from the call shape. In a whole-file-in, whole-file-out API, the entire input and the entire output sit in memory at once, according to the library's author, writing on dev.to ^[4]. A browser tab can't hold an ArrayBuffer much past about 2 GB ^[5]. Four gigabytes of input is twice that ceiling before compression starts ^[1]. "No optimisation fixes that. The API shape is the problem," the author wrote ^[6].

Calling compressStream takes a ReadableStream and returns one. That is the same type File.stream(), fetch and Node's Readable.toWeb() already produce ^[8]. The same API runs in Node.js 18 and later ^[21]. I think the interface choice is the best engineering in the package, because it connects to the file picker, the network and Node's file system without adapter code ^[8]. Data moves through fixed buffers 128 KB at a time, so a 4 MB file and a 4 GB file use the same memory ^[9]. When the consumer is slow, the library stops reading the source instead of piling data up in RAM. pipeTo carries that backpressure through, so compression waits for a slow disk ^[10].

The package embeds its WebAssembly binary, with no .wasm file to host and no bundler plugin to configure. It installs at 163 kB with zero dependencies ^[12]. The example that writes to disk imports one anyway: StreamSaver, to receive the piped output ^[13]. The 35 MB figure ^[2] holds end to end only if whatever reads the output also streams. An app that gathers the stream back into a single buffer puts the whole output in memory again. Holding the whole output was half of what broke the older libraries ^[4].

At the author's measured ratios, a 1,000 MB file shrinks to about 182 MB with zstd and about 286 MB with gzip ^[2]. At 147 MB/s that file takes about 6.8 seconds, in line with the author's figure of roughly 7 ^[3]. The post's text does not state the hardware or the compression level behind the comparison. The test data was log-style. For the 5.5x ratio to carry over, a team's files have to look like those logs ^[14].

Upload is the slowest part of the trip, the author wrote ^[15]. In Chrome and Edge, the compressed stream can go straight into fetch with a `Content-Encoding: zstd` header and `duplex: "half"`, provided the server runs HTTP/2 or later ^[18]. Firefox and Safari don't support streaming request bodies yet ^[19]. A single streamed POST covers two of the four browsers the author tested ^[4].