A 1.5s speech-to-LLM win came from not sending silence and deleting a classifier

A developer building a desktop overlay that listens to the far side of a video call, transcribes it and streams an LLM answer reports that the naive pipeline took roughly 3.2 seconds from end of sentence to first token, and that four places looked worth attacking ^[1]. According to the write-up, only two of the four turned out to matter, and the total recovered was about a second and a half ^{[2] [3]} - which puts the post-optimization number near 1.7 seconds ^[19].

The first fix was not sending silence, and the author is explicit that the reason is not bandwidth ^[4]. A speech-to-text server that receives continuous audio never observes a clean pause, so it delays its segment commit ^[4]. Client-side voice activity detection therefore runs in an AudioWorklet against the raw signal, with an RMS threshold of 0.005 ^[5]. The ordering is load-bearing: normalizing first amplifies room noise into apparent speech, so detection happens before normalization ^[6]. Hangover is 1.0 second, because cutting the stream the instant level drops removes the tail of every sentence, and because the value has to clear the server's own 0.6 second silence threshold with margin ^{[7] [10]} - here, 0.4 seconds of it ^[21]. Pre-roll is a rolling buffer of 14 chunks, about 600ms at 48kHz, flushed when VAD opens, because a quiet sentence onset sits below the threshold and the first syllable is otherwise gone ^[8]. The author's framing of that failure is the useful part: the difference between "what's a database index" and "at's a database index" is a model that answers confidently and wrongly ^[9].

The second half of the audio fix is server-side endpointing. ElevenLabs Scribe v2 Realtime exposes commit_strategy 'vad' with vad_silence_threshold_secs of 0.6, so the segment commits on a pause rather than on a client request, and combined with the client gate the transcript lands while the speaker is still drawing breath ^{[10] [11]}.

The fourth item is where the framing gets interesting. A cheap model decided whether an utterance was a question worth answering, at a cost of about 200ms ^[15]. It was not removed for latency. It was removed because follow-ups like "And the second letter?" score as not-a-question, so the system went silent exactly when context made the intent obvious; precision was acceptable and the failures clustered in the highest-value moments ^[16]. The gate was not run concurrently with generation - it was deleted, and the decision handed to the model that was already going to be called, which has the context the classifier never had ^[17]. In its place sits a client-side completeness gate with a 900ms silence timer for utterances that do not end in a question mark ^[18].

So the arithmetic: 200ms of the 1,500 came from the classifier, leaving roughly 1.3 seconds attributable to the audio path ^[20]. The author does not itemize further.

Two operational notes worth stealing. Keyterm biasing caps at 50; send 51 and the socket closes with code 1008, invisible unless you log close reasons ^[12]. And a reconnect that races a language switch leaves two live sockets, with the stale one sometimes winning silently - audio flowing, nothing rendering, no error ^[13]. The fix is a four-line generation counter checked in every async continuation, which the author credits with removing a whole class of "stops working after twenty minutes" reports ^[14].

Watch whether the 900ms completeness gate survives contact with slower speakers, and whether configurable server-side endpointing thresholds become table stakes across STT vendors rather than one provider's flag ^{[10] [18]}.