David AI's benchmark finds speech systems 25 points apart on Indic conversation

Both spreads measure how far apart the 14 systems land within a single language on the public evaluation set ^[1][2]. In Bengali, Hindi, Marathi, Tamil and Telugu the average gap is about 6.9 times the gap in the five most consistent languages ^[1]. A choice worth under four points of error rate in one group is worth about 25 points, on average, in the Indic group ^[2][3]. According to Runtimewire's account, the report argues that an English score can make competing models look closer than they are in many other languages ^[16].

The reference transcripts are verbatim. Fillers, repetitions, false starts, interruptions and backchannels stay in, and every transcript went through two annotators and a language expert ^[6]. This is careful annotation work, and it is also a scoring decision. A system built to emit clean, readable text gets charged for each filler it drops, unless the scorer normalises disfluencies before comparing. Runtimewire's write-up does not describe that normalisation step. The scoring code is public ^[12], so its text-normalisation function is the first thing to read before trusting a row of the table.

Every system made its most diarization errors in the conversations with the most overlapping speech ^[11]. The hardest-overlap tier holds 15 recordings ^[8]. A ranking inside that tier rests on a small sample. Runtimewire's own caveat is that the results stay tied to this dataset and its recording conditions ^[17]. For any row to transfer, a team's traffic has to resemble the recording setup: two people talking unscripted, one channel each ^[5][7].

The downloadable slice is smaller than the benchmark. The dataset card lists 1,035 clips and 42.1 hours of two-channel audio from 682 speakers ^[14]. The source corpus is 147 hours ^[5]. The public audio is about 29% of the corpus by hours ^[2]. French, Italian and Korean are held back pending an internal privacy review ^[15], so outsiders can check at most 18 of the 21 languages from the public files ^[3].

YC partner Diana Hu described the founders' thesis as a need for a "Common Crawl for audio" ^[18]. The release contains identifiable voices and requires accepting a data-use agreement ^[13]. That is a sensible call for recordings of real people, and a narrower supply than the broad one Hu's phrase describes ^[18].