Scores and Performances.
Chenzi Xu · chenzi.xu@ntu.edu.sg
1. Week 10’s normalizer TODO: what should “$2.50” become before a voice reads it?
“two dollars fifty”
2. What did G2P do with “makan”, a word missing from the dictionary?
guessed it by English letter-to-sound rules (/ˈmækən/)
3. In KlattGrid, what did you set by hand?
F1 and F2 (the 120 Hz buzz and F3 were preset)
4. Unit selection chose recordings by two costs. Which two?
target cost and join cost
5. PSOLA moved the pieces closer together. What changed, and what stayed?
the pitch changed, the formants stayed
| Tacotron 2 | FastSpeech 2 | |
| TRAINING | learns the timing itself | durations from an aligner |
| SYNTHESIS | one frame at a time | all frames at once; editable |
A mel spectrogram is the score,
not the performance.
It says which frequencies, when, and how loud.
It leaves out the phase (where each wave peaks), so the vocoder has to supply it.
Synthetic: 100 Hz tone, A all in phase, B random phases (the same as listening pair 2). Band widths and shapes after Glasberg & Moore (1990).
Same sound
Each harmonic is alone in its ear band, so there is nothing for phase to change.
Different: A buzzy, B smoother
The upper harmonics (gold) share ear bands; their phases make the loudness pulse or not. A lower voice packs its harmonics closer: more per band.
Different: B rough and wobbly
Tone 1 again, rebuilt from 50 ms pieces; in B the pieces are out of step and partly cancel.
The ear hears phase only as loudness changes inside one band: when harmonics share a band, at bursts, and when pieces are out of step.
Synthetic tones made in the page, all at the same loudness (RMS). A teal, B coral; gold: a harmonic that shares its ear band. Sources: Patterson (1987); Carlyon & Shamma (2003); Laroche & Dolson (1999).
In 2016, listeners rated it above splicing (US English MOS: WaveNet 4.21, unit selection 3.86, natural speech 4.55). The same loop returns in Lecture II, predicting audio tokens.
The acoustic model draws the mel; the vocoder performs it as a waveform.
Durations: the Week 9 aligner on “kopi” (k 100, ɔ 90, p 60, i 110 ms), in 10 ms frames; mel and wave drawn by hand.
A voice, from two networks.
Next: the newer designs.
| model | family | what it predicts | what makes the wave | voice chosen by |
| VoxCPM2 | one by one | continuous audio frames (no tokens) | its own audio decoder (frames → wave) | a description, a clip, or a prompt to continue |
| Pocket TTS | one by one | continuous audio frames | its own audio decoder (frames → wave) | preset voices, or a clip made into a speaker embedding |
| CSM-1B | one by one | Mimi codec tokens | the Mimi codec decoder | the conversation so far, and a speaker tag |
| Parler-TTS Mini | one by one | DAC codec tokens | the DAC codec decoder | a text description of the voice |
| XTTS-v2 | one by one | audio codes (made from a mel) | a HiFi-GAN-style decoder, from the GPT’s hidden states (not the codes) | a 6-second clip |
| OmniVoice | in steps | codec tokens, unmasked in 32 steps | the codec decoder | a clip, listed attributes, or nothing |
| F5-TTS v1 | in steps | a mel (flow matching) | Vocos, a separate vocoder | a clip and its transcript |
| Kokoro-82M | one pass | no mel: durations, pitch and energy (phones in) | a built-in decoder, no separate vocoder (iSTFTNet) | preset voices only |
one by one an audio language model: WaveNet’s loop, on tokens or frames · in steps flow matching or unmasking, all frames at once · one pass durations, FastSpeech-style, the wave made directly
Any other model: find its paper and ask what it predicts and what makes the wave.
“The next train to Tampines will arrive in two minutes.” All five voices are synthetic.
5 excellent · 4 good · 3 fair · 2 poor · 1 bad
Under each voice: a MOS predictor’s guess (UTMOS, a network trained on listeners’ ratings) and Whisper’s word errors. A and D share one mel: only the vocoder differs. Did the room agree with the predictor?
| measure | how | what you report |
| Intelligibility | Whisper (whisper-small or whisper-large-v3-turbo) transcribes the clip; score it with your Assignment 1 normalizer | WER per group, with clip and word counts |
| Predicted naturalness | A MOS predictor (UTMOSv2, or UTMOS22 via SpeechMOS) guesses listeners’ 1 to 5 rating | Mean and spread per group |
| Prosody | Praat: F0 (a question rise, pitch on a particle), durations, pauses | Change per group, in semitones or Hz, and ms |
| Voice similarity (clones) | Each clip against your reference, with SpeechBrain ECAPA | Cosine similarity per group (1 = identical) |
| Your expert listening | Log each failure: sentence, what you heard, a narrow transcription where it helps | Failures, grouped by where they arose |
Formants: tiny, controllable, robotic. Splices: natural on common text, brittle on rare, and huge.
Close to human naturalness, any voice from data, but mostly a black box: you can change durations or pitch, not the fine detail.
Fine control, explanation, and an easy way to tell a real voice from a generated one.
Hear the vocoder.
Then judge the voice.
In the break: open the Week 11 notebook (link on NTULearn) in Colab;
run the setup cell and the Part 1 loading cell.
Every clip is synthetic. If a model fails to load, pre-made clips and stored results take over. Take-home (optional): three test sentences for your own A2 model.
Colab's free CPU is enough; models download part by part.
The network that maps a phone or character sequence to a sequence of acoustic frames, usually a mel spectrogram, deciding durations either by attention or by an explicit duration predictor.
Course wording. Shen et al. (2018), Tacotron 2; Ren et al. (2021), FastSpeech 2.The component that turns acoustic frames such as a mel spectrogram into a waveform, supplying the phase and fine detail the frames leave out.
Course wording. Griffin & Lim (1984); van den Oord et al. (2016), WaveNet; Kong, Kim & Bae (2020), HiFi-GAN.A single network trained to map text directly to a waveform, learning the alignment, the acoustics and the vocoding together.
Course wording. Kim, Kong & Son (2021), VITS.The average of listeners' ratings of each stimulus on a five-point scale from 1 (bad) to 5 (excellent), used to judge the naturalness or quality of speech.
Course wording. ITU-T P.800 (1996), Annex B: absolute category rating, written for telephone speech quality.Tacotron 2 and FastSpeech 2 both turn a sequence of letters or phones into mel frames. Which one needs phone durations before it can be trained, where do they come from, and what does that buy it?
Why can a mel spectrogram not simply be turned back into sound? Name the missing ingredient and two ways a vocoder can supply it.
A model card says: “turns text into audio tokens one by one, continuing a 5-second voice prompt; a codec decoder makes the waveform.” Which family is it? Would it rather clone by a vector or by a prompt?
A TTS model scores WER 0% and a predicted MOS of 4.2 on “Can lah, see you later”, but to your ear the “lah” is flat and too short. What do you report, and which measure supports your ear?
Worked answers in the appendix at the end of this deck.
1. Classic neural TTS is two networks split at the mel: an acoustic model draws it, a vocoder performs it; one-network voices and audio language models skip the mel
2. The acoustic model either finds the alignment by attention or learns durations from forced alignment
3. The mel has no phase: the vocoder invents it, sample by sample (WaveNet) or all at once (HiFi-GAN)
4. Judge a voice on intelligibility, naturalness, prosody and similarity; tools stand in for listeners, and your ear checks the tools
The models used in today's practical. huggingface.co/learn/audio-course/chapter6/introduction
§§1–2 only, then the audio samples. arxiv.org/abs/1712.05884
The blog, not the paper: listen to the embedded audio. deepmind.google/blog/wavenet-a-generative-model-for-raw-audio
Skim the discriminator design: you used this vocoder today. arxiv.org/abs/2010.05646
Introduction and demo samples; the math is heavy. arxiv.org/abs/2106.06103
Why MOS is indispensable and flawed. isca-archive.org/ssw_2019/wagner19_ssw.html
Skim the taxonomy figure; read two case studies. arxiv.org/abs/2402.01708
On voice banking: a synthetic voice built for one person. ted.com/talks/rupal_patel_synthetic_voices_as_unique_as_fingerprints
§1 and the speaker-embedding plots. arxiv.org/abs/1806.04558
Text becomes a picture.
The picture speaks.
Shen, J., Pang, R., Weiss, R. J., et al. (2018). “Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions.” Proc. ICASSP 2018. arXiv:1712.05884. Tacotron 2.
Ren, Y., Hu, C., Tan, X., et al. (2021). “FastSpeech 2: Fast and high-quality end-to-end text to speech.” ICLR 2021. arXiv:2006.04558.
McAuliffe, M., Socolof, M., Mihuc, S., Wagner, M. & Sonderegger, M. (2017). “Montreal Forced Aligner.” Proc. Interspeech 2017, 498–502. FastSpeech 2’s durations.
Griffin, D. W. & Lim, J. S. (1984). “Signal estimation from modified short-time Fourier transform.” IEEE TASSP 32(2), 236–243.
Patterson, R. D. (1987). “A pulse ribbon model of monaural phase perception.” JASA 82(5), 1560–1586. When the ear hears phase.
Carlyon, R. P. & Shamma, S. (2003). “An account of monaural phase sensitivity.” JASA 114(1), 333–348.
Laroche, J. & Dolson, M. (1999). “Improved phase vocoder time-scale modification of audio.” IEEE Trans. Speech and Audio Processing 7(3), 323–332. Pieces out of step.
Glasberg, B. R. & Moore, B. C. J. (1990). “Derivation of auditory filter shapes from notched-noise data.” Hearing Research 47(1–2), 103–138. The ear’s band widths.
van den Oord, A., Dieleman, S., Zen, H., et al. (2016). “WaveNet: A generative model for raw audio.” arXiv:1609.03499.
Kong, J., Kim, J. & Bae, J. (2020). “HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis.” NeurIPS 2020. arXiv:2010.05646.
Kim, J., Kong, J. & Son, J. (2021). “Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech.” ICML 2021. arXiv:2106.06103. VITS.
Pratap, V., Tjandra, A., Shi, B., et al. (2024). “Scaling speech technology to 1,000+ languages.” JMLR 25(97), 1–52. MMS.
Ao, J., Wang, R., Zhou, L., et al. (2022). “SpeechT5: Unified-modal encoder-decoder pre-training for spoken language processing.” ACL 2022. arXiv:2110.07205.
Wang, C., Chen, S., Wu, Y., et al. (2023). “Neural codec language models are zero-shot text to speech synthesizers.” arXiv:2301.02111. VALL-E.
Défossez, A., Copet, J., Synnaeve, G. & Adi, Y. (2022). “High fidelity neural audio compression.” arXiv:2210.13438. EnCodec.
Chen, Y., Niu, Z., Ma, Z., et al. (2025). “F5-TTS: A fairytaler that fakes fluent and faithful speech with flow matching.” arXiv:2410.06885.
Zhu, H., Ye, L., Kang, W., et al. (2026). “OmniVoice: Towards omnilingual zero-shot text-to-speech with diffusion language models.” arXiv:2604.00688.
Zhou, Y., Zeng, G., Liu, X., et al. (2026). “VoxCPM2 technical report.” arXiv:2606.06928.
Rouard, S., Orsini, M., Roebel, A., Zeghidour, N. & Défossez, A. (2025). “Continuous audio language models.” arXiv:2509.06926. Pocket TTS.
Sesame (2025). “Crossing the uncanny valley of conversational voice.” Research post, 27 Feb. CSM-1B.
Lyth, D. & King, S. (2024). “Natural language guidance of high-fidelity text-to-speech with synthetic annotations.” arXiv:2402.01912. Parler-TTS.
Casanova, E., Davis, K., Gölge, E., et al. (2024). “XTTS: A massively multilingual zero-shot text-to-speech model.” Proc. Interspeech 2024. arXiv:2406.04904.
Li, Y. A., Han, C., Raghavan, V. S., Mischler, G. & Mesgarani, N. (2023). “StyleTTS 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models.” NeurIPS 2023. arXiv:2306.07691. The base of Kokoro-82M (hexgrad, model card).
ITU-T (1996). Recommendation P.800: Methods for subjective determination of transmission quality. Annex B: absolute category rating.
Kirkland, A., Mehta, S., Lameris, H., Henter, G. E., Székely, É. & Gustafson, J. (2023). “Stuck in the MOS pit: A critical analysis of MOS test methodology in TTS evaluation.” Proc. SSW 12.
Saeki, T., Xin, D., Nakata, W., et al. (2022). “UTMOS: UTokyo-SaruLab system for VoiceMOS Challenge 2022.” Proc. Interspeech 2022. arXiv:2204.02152.
Radford, A., Kim, J. W., Xu, T., et al. (2023). “Robust speech recognition via large-scale weak supervision.” ICML 2023. Whisper.
Siuzdak, H. (2023). “Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis.” arXiv:2306.00814. F5-TTS’s vocoder.
Desplanques, B., Thienpondt, J. & Demuynck, K. (2020). “ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification.” Proc. Interspeech 2020. SpeechBrain ECAPA.
FastSpeech 2 needs durations to train: they come from forced alignment (MFA, Week 9). In return it draws all frames at once (fast), almost never skips or repeats words, and lets you edit a duration or a pitch by hand. Tacotron 2 learns the alignment itself, by attention.
The mel keeps the energy in each band but drops the phase (and blurs fine frequency detail). A vocoder supplies it by estimating it (Griffin-Lim: trial and repair) or by learning what real waves look like (WaveNet sample by sample; HiFi-GAN all at once, trained against discriminators).
An audio language model: it predicts discrete audio tokens one by one, and a codec decoder turns them into a waveform. It clones by prompt: it continues the reference recording.
Report all three: WER 0% (a machine understood the words), predicted MOS 4.2 (it sounds natural overall), and your observation that the particle is flat and short. Support your ear with prosody measures in Praat: the F0 contour and the duration of “lah”, compared with how the particle is usually said. Say plainly that WER and predicted MOS cannot see this.