Six-word text matching caught four audiobooks that a filename check had cleared

The check that caught it is cheap. For every clip in the held-out set, ask whether any training clip contains the same run of six consecutive words, then hash the 6-grams and look for collisions ^[10]. Six words is long enough that natural coincidence is rare at this corpus size ^[10]. It is short enough to survive a re-cut, because moving a segment boundary changes where a clip starts and leaves the sentence inside it as it was.

By filename, the eval and training sets showed zero collisions ^[9]. Comparing text found four audiobooks in the material twice, the same recordings segmented by two different tools at two different times under entirely different names ^[11]. "There is little material, so you keep returning to it with new tooling, and every pass produces fresh filenames wrapping old content," the author wrote ^[12].

The failure mode is quiet: no crash, no warning, no odd loss curve ^[14]. The model trains on recordings it is later examined on, different cuts of the same sentences read by the same voices, and posts a good number ^[14]. "The exam questions were handed out in advance," the author wrote ^[13].

The held-out set was designed before any of that surfaced. Adjacent clips here come from the same recording, reader, microphone and afternoon, and the segmenter routinely cuts one sentence across two neighbouring clips, so a random 90/10 split leaves half a sentence in train and the other half in eval ^[6]. Holding out two entire books removed the voice, the microphone, the recording era and one text's vocabulary in a single move ^[8]. The 893 clips average about 7.5 seconds each ^[21]. If the 15.5 hours that survived cleanup is the training side, the 1 hour 52 minutes of eval is about 11% of the combined 17.4 hours ^[22].

One caveat sits under the baseline. The post says 34.6% slightly understates it, because the model writes 1954 where the book spells out Biñ doquz yüz elli dört and the scorer counts four substitutions ^[5]. A digit string scored against a spelled-out target costs four substitutions whether or not a listener would call it an error, so WER on this corpus includes the project's numeral normalisation errors along with the model's recognition errors.

The training run was the small part. The backbone stayed frozen and a small adapter trained alongside it: 31M trainable parameters against roughly 1.5B frozen, about 2% ^[17][23]. Three epochs, 705 steps, an hour and a half on a single consumer GPU ^[18][1]. "Everything that made that hour and a half mean anything took months," the author wrote ^[24].

The chart at the top of the post has three bars, starting model, after fine-tuning, after decode-time tuning, and the post says the third cost zero training ^[19]. The post does not publish the word error rates behind the second and third bars ^[20]. For 34.6% to mean anything about another corpus, that corpus would need its own split on speaker and document and its own content-level dedup pass. Scored against a random split of the same audio, the same model would post a better number off speaker and room memorisation ^[7].