Build1 publisher2 min readPublished
Restarting the server mutes every visitor mid-conversation until they reload
A voice avatar kept its microphone button lit while the server logged 0.00 volume, and the fix needed three failed hypotheses before a log showed the widget enabling a track that was about to be discarded.
The Engineer · Build desk

What happened
- After a server restart the avatar kept standing with its microphone button lit and simply stopped answering, and the only way visitors could tell it was broken was to reload the page.
- The widget called its connect routine once on page load and had no reconnection path at all, so a deploy, a Wi-Fi switch or a laptop waking from sleep all ended the same way.
- With exponential backoff added, the widget logged a single retry attempt after a deploy and then went quiet, never opening a second connection.
- Once reconnection worked, the server's voice-activity diagnostic reported 1500 judgments, zero speeches and volume 0.00 across 30 seconds.
- Diagnostic logs added on the fourth attempt showed the widget enabling the microphone before the pipeline obtained the new track, which starts disabled.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- exposure Nobody files a support ticket about a muted widget with a lit button, so the server-side volume counter is the only place the failure shows up.
- cost Never giving up means one abandoned open tab asks the server 120 times an hour, a bill the author accepted on the grounds that a reconnection costs almost nothing.
- constraint A re-entry guard written for page load makes the same function useless as a recovery path, so any single-shot connect needs an explicit teardown before it can be called twice.
- capability Since the conversation ID sits in sessionStorage, the recovery path can put the visitor back into the same conversation, which is what makes building it better than prompting for a reload.
The recovery timeline shows the sequence. The server restarted at 15:09:43, the voice pipeline detected the disconnection at 15:10:16, and the reconnect landed at 15:10:28 [12]. Detection took 33 of those 45 seconds [1][2]. The backoff schedule was the straightforward part to write: 1s, 2s, 4s, 8s, 15s and 30s, six attempts and 60 seconds before it settles into a 30-second loop the author deliberately left without a timeout [5][5][6].
One line at the top of connect() explains the retry that logged once and stopped: `if (pcRef.current) return;` [8]. The author wrote that "As a function called only once when the page is opened, this guard is correct to prevent double connections" [9]. On a reconnect, the object it tests for is exactly the one the dropped connection left behind, so the call returned with no error and no log [10]. Calling disconnect() before reconnecting fixed that half [11].
The server-side counter narrowed the second fault. 1500 judgments in 30 seconds works out to 50 a second, about one every 20 milliseconds, and all of them read volume 0.00 [15][3]. The detector was running and no audio was arriving [15].
The mute was an ordering problem. The widget re-enabled the microphone under `if (voicepipe.status === 'connected')`, and that status is updated from `pc.oniceconnectionstatechange` [21]. The status flips on that event before connect() has obtained the new capture track, so the enable applied to the track from the dead connection, and connect() then built a new track that starts disabled [20][16]. The excerpt stops inside that ICE handler, so the post does not show the final fix [22].
Three fixes went in before those logs existed [17]. Each came from reading the source and choosing a plausible culprit, and by the author's account all three assumed something had muted the microphone [17][20]. The fourth attempt printed the enable call and the track creation with timestamps, in that order [18][19].
Both faults are internal to the widget. The stale microphone was its own track, created by the previous connection and already marked for disposal, and no device enumeration or hardware picker is involved [20]. The pattern reproduces in another codebase only under three conditions. connect() is called once on page load and guards against re-entry [3][8]. The capture track is created inside connect() and starts disabled [16]. The mute state shown in the UI lives outside the track, so it has to be reapplied whenever the track is replaced [16].
What to watch
- Whether a follow-up post shows the fix for the enable-before-track ordering, since the excerpt breaks off inside the ICE state handler.
- Whether indefinite 30-second retries hold up under a real deploy with many tabs left open, on the server side of the connect path.
- Whether the 33-second detection delay can be cut with an application-level heartbeat instead of waiting on ICE connection state.