Build1 publisher3 min readPublished
Fixed a-i-u-e-o test order shrank the first vowel's signal in a stateful lip-sync estimator
One developer's lip-sync estimator measured its first test vowel against a running average dominated by that vowel, across three TTS voices. Rotating which vowel leads, from a reset state, is the control a stateful audio model needs before its score means anything.
The Engineer · Build desk
What happened
- The vowel estimator compares each frequency band's current level with a long-term average that updates whenever audio arrives, so the same sound yields different features depending on what came before.
- Test audio was sustained vowels synthesized with the author's company's Style-Bert-VITS2, three speakers by five vowels, so every clip's correct label was known.
- The first evaluation always played the vowels in the order a, i, u, e, o, an order the author found conflicted with how the long-term average initializes.
- The author's fix rotates which vowel plays first, shifting a, i, u, e, o to i, u, e, o, a, with the estimator reset to the same initial state for every run.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint An accuracy figure from one fixed order carries over to production only if real speech opens on the same vowel after the same warm-up, so a single-order score describes the test sequence as much as the estimator.
- decision For a five-vowel set, fair per-vowel comparison takes five runs from a reset state, one per leading vowel, because a random shuffle leaves which vowel goes first to chance.
- cost Synthetic test audio needs its own validation pass before it can grade anything, since a silent or malformed clip gets counted as an estimator error.
The first warning sign came before the diagnosis. While tuning the estimator, the author wrote, "I found myself writing code to accommodate the quirks of the evaluator." [15]
Subtracting a running average is a cheap way to normalize a live signal. Whatever stays constant in each band drops out, and the classifier sees the change [3]. The price is that a sound's features depend on its history. The author put it this way: "Simply preparing evaluation audio does not fix the measurement conditions." [16]
The problem is in the warm-up. The average is seeded from the start of the input so it is usable from the first frame, and it follows the input faster early on [10]. If あ plays first, the average learns the spectrum of あ almost at once. Subtract that from the current あ and the band deviations the templates are built from get smaller [10][8]. Each later vowel meets an average that already holds the vowels before it [10]. A tidy a, i, u, e, o sequence looked like good hygiene [9]. "The most significant discovery this time was that the measurement method was creating the answer," the author wrote [1].
Rotation is the right control. The author is precise about why a shuffle falls short: "What's important is not just shuffling, but also making the initial state of the estimator uniform and measuring the condition where each vowel becomes the first vowel." [12] The order generator is a short `map` over the vowel array in JavaScript [13].
Cyclic rotation balances who goes first. It does not change who follows whom. Across the five shifted orders, i is either first or directly after a [3]. Later vowels are compared against an average that includes their predecessors [10]. If that average weights recent audio heavily, part of the i score is really a score of the a-to-i transition. In my view the next check is an ordering design that also varies each vowel's predecessor, so the effect can be read per pair.
The quality checks got the same suspicion, and this is the part of the post I would copy. Before listening, the author tested each clip's length, RMS, peak, voicing rate, fundamental frequency and formants [5]. A clipping check that looked only at peaks flagged clips that turned out to be peak-normalized [6]. LPC formant estimates for high-pitched speakers locked onto harmonics. The author dropped them from band design and read the spectrum directly, noting this is not a verdict on LPC in general [7]. Templates were validated leave-one-speaker-out: built from two voices, scored on the third, with the roles rotated [8].
The text of the post available for review ends at the rotation code. It does not include accuracy before and after rotation, or the results for clip duration, the second pitfall the title names [14].
What to watch
- Accuracy from the rotated runs, broken out by which vowel led, would show how large the order effect actually was.
- The post's duration results would show whether clip length moves scores through the same running-average warm-up.
- A test with orders that also vary each vowel's predecessor, beyond cyclic rotation, would isolate transition effects.