Optimizing for first audio cut nine seconds of caller silence to about 1.5

Nine seconds of dead air sounds like a dropped call, and the dev.to write-up says that gap is long enough that most people would hang up ^[1]. A dashboard shows total response time. A caller hears only the silence before the first syllable, and once audio is playing the rest of the generation runs underneath it.

Three of the four changes in the post attack that gap at the front. Sentence-level streaming hands the first finished sentence to Cartesia while the model is still writing the second ^[9][6]. Warm connections keep the vendor sockets open so the first turn does not pay setup ^[9]. On-device embeddings take a network hop out of the retrieval that remains ^[9]. Conditional RAG removes the retrieval stage entirely for turns that do not need it ^[9].

Nine seconds to about 1.5 is 7.5 seconds removed, a factor of six ^[13]. Each of the four changes carries a condition. Conditional RAG pays in proportion to the share of turns that can skip retrieval, so on a line where every question hits the knowledge base it pays nothing. If volume is low, the first call of the morning still pays connection setup, warm pool or not. On-device embeddings assume the embedding model fits wherever the orchestrator runs. The post reports the two latency figures and the four changes. There is no cost comparison, and no word on whether 1.5 seconds is a median or a tail value ^[14].

"Owning the orchestration layer yourself means you now own barge-in, turn detection, guardrails, and every edge case a managed platform used to absorb for you," the author wrote ^[10]. The post's own list of what a voice system has to handle runs to turn detection, interruptions, conversation flow management, speech recognition and speech generation ^[7]. Deepgram covers recognition and Cartesia covers generation ^[6]. Turn detection, barge-in and flow management are code the team maintains.

The rebuild started from one requirement. The client wanted a more natural, expressive voice able to convey emotion, and at the time that was not straightforward through VAPI ^[4]. The first workaround pulled the available Cartesia voices, stored their IDs in a JSON configuration, and called Cartesia directly during voice selection ^[5]. More requests followed, and the post argues that customization gets harder as requirements get more specific ^[18]. The client then chose what the post calls the "best of the best" combination from the cost estimation document ^[11]. The stated goal was to use each platform for what it does best and keep full control of the orchestration layer ^[12].