Loading today’s stories
Build1 distinct publisher3 min readUpdated
A Go pipeline over 5 million rows generated 10M-plus allocations and 540 MB of garbage. The fix is not tuning encoding/csv, because its signature guarantees the allocations.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
A developer profiling a Go batch pipeline over a 5-million-row CSV export found the process holding hundreds of megabytes of RAM with the garbage collector taking a noticeable share of CPU, and has published go-zerocsv, a zero-dependency reader and writer built on in-place typed scanning [3][6]. The part worth acting on is not the speedup; it is that the memory profile of `encoding/csv` follows from its signature, so it is not something upstream tuning can remove [1].
Read the promise: `func (r *Reader) Read() (record []string, err error)` [1]. To honour it, every call allocates a fresh `[]string` for the row and copies each parsed field into a newly allocated heap string [2]. Under pprof, `runtime.makeslice` and `runtime.rawstring` dominate the allocation graph [4]. On a 5,000,000-row, 6-column file the author counts more than 10 million allocations and about 540 MB of cumulative heap allocation pushed at the collector [5]. That count reads conservative: one slice header plus six strings per row across five million rows is 35 million allocations, so 10 million is a floor rather than an estimate [13]. In bytes it is roughly 108 of garbage per row, trivial once and expensive five million times [11].
go-zerocsv keeps a single internal 4 KB buffer, compacts it between records and reuses it, which is the mechanism behind the reported flat ~5 KB of heap whether you stream 1,000 rows or 10,000,000 [7][6]. `Read` returns a lightweight `Record`, and `Scan(&id, &name, &score, &active)` parses into your own variables in the pattern `database/sql` users already have muscle memory for [8]. There is an `IsFirst()` helper for skipping a header row and `String(i)` for direct field access [9]. The write path has the mirror-image problem: because `Writer.Write` accepts only `[]string`, emitting an int, float64, bool or time.Time through the standard library costs a `strconv` or `fmt.Sprintf` call per cell and a throwaway string with it [10].
The measurement offered is 5 million rows parsed in about 300 ms while holding 5 KB, on an AMD Ryzen 5 8400F, linux/amd64, Go 1.26 [c11b]. That is roughly 16.7 million rows per second [12]. One machine, one file, the author's own library: a direction, not a specification.
Two cautions before swapping it into a pipeline. The 540 MB and 5 KB numbers do not measure the same quantity; the source describes the first as cumulative heap allocation across the run and the second as steady-state usage [5][6]. Cumulative allocation is what the collector has to chase, not resident set, so the defensible claim is a large cut in GC work, and the 108,000x ratio between the two figures is arithmetic rather than an RSS comparison [14]. Second, zero allocations per record while scanning into a `string` variable implies that string points into the buffer that is compacted and reused for the next row [7][6]. The source's list of alternatives to `Scan` is cut off mid-sentence at copying field bytes [15], which is exactly the API whose lifetime rules decide whether you can retain a scanned string past the next `Read`.
Follow any of these and your For You feed starts watching them — no settings page required.
Ranked by verification strength, evidence, and original report placement.
The reader method signature in Go's encoding/csv is func (r *Reader) Read() (record []string, err error).
Every call to encoding/csv Read() allocates a new []string slice for the row (runtime.makeslice) and copies each parsed byte slice into a newly allocated heap string (runtime.rawstring).
go-zerocsv uses a single internal 4 KB buffer that holds raw bytes and is compacted between records, then reused for the next row.
go-zerocsv's Read() returns a lightweight Record struct whose Scan method scans fields in-place directly into caller variables, mirroring the database/sql pattern.
go-zerocsv provides rec.IsFirst() as a header helper for skipping the header row and rec.String(i) for direct field access.
Because encoding/csv Writer.Write([]string) only accepts strings, writing typed values such as int, float64, time.Time or bool forces calls to strconv.Itoa, strconv.FormatFloat or fmt.Sprintf first, each creating temporary throwaway strings for every cell in every row.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
One self-published post, no artifacts
The structural claims are strong: the encoding/csv reader signature and the string-only Writer.Write are verifiable properties of Go's public API, and the post's code samples demonstrate the proposed API concretely. Everything quantitative is weak: a single dev.to source authored by the library's creator, no benchmark harness or testing.B output, no pprof artifact behind the profiling anecdote, no baseline timing for encoding/csv, an unreconciled gap between the stated 10,000,000+ allocations and the post's own model implying ~35,000,000, and a body that is truncated mid-sentence.
Announcement only, no external users
The supplied material establishes only that the package exists publicly with a documented import path and an author-run benchmark. There is no disclosed version, no dependents, downloads, stars, production deployment, or third-party evaluation, and no indication that anyone other than the author has run it. Adoption is therefore at the floor above zero: a public release event with nothing built on top of it yet.
Headline outruns published proof
Overstated, though not baseless. The correctly diagnosed part - the stdlib API guarantees per-row allocations - is real and understated in most CSV tutorials. But the framing inflates it: '540 MB burned' is cumulative allocation churn presented against a 5 KB steady-state buffer, a ~108,000:1 contrast between non-comparable metrics; the '10,000,000+ allocations' figure does not match the article's own model; and '0 allocations per record' plus '~300 ms for 5M rows' arrive with no benchmark output, no baseline timing, and no correctness or field-lifetime caveats, all from the library's author on a self-publishing platform.
Author promoting own library
The single source is written by go-zerocsv's creator on a developer self-publishing platform, and its structure is a problem-agitate-solve pitch for that package: the standard library's weaknesses are quantified, the author's alternative supplies every favourable number, and the import path is embedded in the examples. There is no disclosure framing, no independent reviewer, and no adversarial number anywhere in the piece. Distribution incentives on dev.to further reward strong headline figures.
Mechanism solid, numbers unverified
Confidence is split by claim type. High confidence that encoding/csv allocates per row and per field by API design and that its writer forces string conversions, since those follow from the public signatures quoted. Low confidence in every magnitude: one self-interested source, no artifacts, an internal arithmetic inconsistency, mismatched memory metrics, and a truncated body. Adoption assessment is high-confidence precisely because the absence of any external signal is unambiguous.
build
A pure-Go DICOM stack takes CGO out of the hospital build pipeline1 distinct publisher
build
Force the tool call, then hand Lightsail a long-lived key1 distinct publisher
build
AI-written code fails the same four ways, and every gate you own reports green1 distinct publisher
build
CSA's 2026 threat list is a flat line, so ask which threats a config snapshot can prove1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 18, 2026