Build1 publisher3 min readPublished
Foundry's voice preview swaps the request/response boundary for a persistent WebSocket session
Microsoft treats speech as its own agent kind, with a managed realtime orchestrator holding the socket open. A dev.to walkthrough sets out why turn detection and tool calls had to be restructured around a few hundred milliseconds.
The Engineer · Build desk
What happened
- Microsoft Foundry's Voice Agents is in preview as a first-class agent kind, sitting alongside prompt agents, hosted agents, workflows and external agents in the same project_client.agents management surface.
- Per the dev.to walkthrough, the runtime is a managed realtime orchestrator holding a persistent WebSocket, and agent execution was restructured so tool calling, turn detection and interruption handling work there rather than in a stateless HTTP call.
- The post puts a caller's tolerance at a few hundred milliseconds, with silence past roughly 300ms prompting the caller to ask whether the agent hung up.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Every tool a voice agent can reach now needs a latency class, not just a schema: the ones that return inside the turn budget, and the ones that have to be covered while the caller waits.
- decision Teams with a working cascaded pipeline have to decide whether managed turn-taking is worth giving up the buffers and hops they currently tune themselves.
- capability Because codecs and silence thresholds sit in a versioned governed resource, an organisation can review a change to them through the same path it uses for a prompt change.
- exposure Once the orchestrator owns the socket, a turn-detection or barge-in defect is a platform defect, and the only levers left to the operator are the fields in the agent definition.
A dev.to walkthrough of the preview puts a caller's tolerance at a few hundred milliseconds, and says a pause past roughly 300 milliseconds makes the caller wonder whether the line dropped [2]. The post attributes that to how human conversation works, not to any product spec [15]. Taken as an engineering budget, those 300 milliseconds cover three things: the silence the voice activity detector waits through before declaring the turn over, the model's first token, and the first byte of audio heading back out [13]. Turn detection and reply latency draw on the same window [13].
A tool call does not fit in there. The post describes function calling over a realtime session as requiring a deferred-response pattern that text agents never need [5]. In a chat agent the user is not standing there waiting for the next word, so the agent can think, call a tool, think some more, and answer [16]. On a live socket the caller keeps talking, interrupts, and says "actually never mind" halfway through a sentence [17]. If the tool result lands after that, the agent is answering a question the caller withdrew, which is one of the race conditions the post lists as voice plumbing [10].
The management surface shows how seriously that was taken. Voice agents come back from a kind filter, `project_client.agents.list(kind=AgentKind.VOICE)`, with `AgentKind` imported from `azure.ai.projects.models` [7]. They take the same create_version, get_version, enable, disable and delete_version lifecycle as prompt agents built with create_from_prompt [8]. What differs is the definition, which carries audio codecs, turn detection thresholds and the output voice as fields in a versioned, governed resource [6][12]. A bad silence threshold is at least auditable.
The two earlier paths were a cascaded pipeline (speech-to-text, LLM completion, text-to-speech) where you own every hop, buffer, latency budget and failure mode, or talking to a realtime model's WebSocket yourself and hand-rolling session state, reconnection, tool dispatch and persistence [9]. The post said both work, but both push a huge amount of voice agent plumbing onto every team that wants to ship one [10]. Foundry takes the wire protocol, the turn-taking and, optionally, transcript and audio persistence; the tool logic and the business rules stay with you [11].
The argument that this is a different runtime is the post's own. It says voice is not "chat with an audio codec bolted on" and calls it "a genuinely different runtime model" [4]. The API shape is consistent with that. It is also one practitioner's account: the published piece is an introduction, a section outline and a snippet, without a measured turn latency [14]. For the 300 millisecond budget to transfer to your product, silence has to be the only thing your user has to go on. On a phone call it is. In a client that can render "working", the number you have to hit is whatever your users tolerate.
What to watch
- Whether Microsoft documents the deferred-response pattern at the wire level, including what happens when a tool result arrives after the caller has interrupted.
- General availability terms for Voice Agents, and whether turn-detection thresholds remain editable per agent version.
- Published latency and cost figures for the managed realtime orchestrator, which the preview write-up does not carry.