HG4052 · Speech Synthesis & Recognition
m
HG4052 · Speech Synthesis & Recognition / Week 12

The Melody
and the Mask.

Tunes and Voices.

Chenzi Xu · chenzi.xu@ntu.edu.sg

→ / space to advance · F for fullscreen · M for the slide menu
t
The shape of the morning · quiz day

Today, by the clock

09:40  Quiz 2 (45 minutes, Weeks 6 to 11)
10:25  Scripts in · break (15 min)
10:40  Going through the answers: immediate feedback (60 min)
11:40  Break (10 min)
11:50  Lecture: the tune of a sentence, then cloning and consent
12:35  Wrap-up and Assignment 2 questions
12:50  Optional practical: start it here, or do it at home
Q
09:40 · pens out

Quiz 2: 45 minutes, Weeks 6 to 11

  • Covers Weeks 6 to 11: fill in the blanks, short written answers, reading code and a figure
  • Closed book; no calculator needed; devices away and silenced
  • Finished early: check your answers, then wait quietly
  • 10:25: pens down, scripts passed to the aisle

The weekly notebooks were the revision.

AFTER THE QUIZ

Break until 10:40; we go through the answers when we return.

Weight: 20% of the course grade. This is the last quiz.

k
10:25 · Scripts in, break (15 min)

The quiz is done.
Back at 10:40.

When we return: the answers to the quiz, then the lecture: the tune of a sentence, and whose voice it is.

Δ
11:50 · Today's lecture

Roadmap

PART I

The tune of a sentence

  • What prosody does
  • Why TTS voices flatten it
  • How to measure it
PART II

Whose voice is it?

  • A voice from a few seconds
  • Harms and safeguards
  • Voices given back
  • What consent must cover
PRACTICAL · OPTIONAL

Does the tune move?

  • Your statement and question
  • Three TTS voices, same pair
  • Particles: lah, meh, lor
  • In class if time, or at home
Warm-up after the quiz · 2 minutes, tell your neighbor

Three echoes from Week 11

1. FastSpeech 2 takes phones in. What does it predict before drawing the mel?

how long each phone lasts, and the pitch and energy of each frame

2. A TTS model can copy a voice from a few seconds of audio without retraining. What are the two ways?

a speaker vector (like SpeechT5's voice vector in last week's notebook), or a prompt that an audio language model (such as VALL-E) continues

3. How did last week's notebook check whether a question rises?

pitch over the last 100 ms of voicing against the sentence's median, in semitones

2
Assignment 2 · 30% · one week to go

Assignment 2: the poster fair is next week

  • Explain your model: how it turns text into sound, and what is new
  • Evaluate it on sentences you design for Singapore speech, with automatic tools and your own ears
  • Pitch: 3 minutes, hard stop. Play the model's voice, then its key idea, your question, result, why, and your ethical position
  • Bring a laptop or phone with a speaker the class can hear; rehearse aloud with a timer
DATES
  • Poster PDF emailed by Thu 12 Nov, 09:30
  • Poster fair in class, Fri 13 Nov
  • Abstract and poster on NTULearn by Fri 13 Nov, 23:59
Three TTS voices · one statement, one question

Three TTS voices read a question; none asks

Prosody · above the phones

Prosody: pitch, duration and loudness

PITCH · F0

The tune

How the voice rises and falls over a phrase: a final rise, a final fall, a peak on one word

DURATION

The timing

How long each syllable lasts, and where the pauses fall: lengthening marks stress and phrase ends

ENERGY

The loudness

Stressed and accented syllables are louder; function words are quieter

★ The hardest idea this week

One sentence, four tunes, four meanings

Only the pitch changes; the words never do.
Where pitch carries more of the meaning

Singapore English particles carry meaning in pitch

  • English: pitch works over the phrase (intonation); the words stay the same words
  • Mandarin: pitch tells words apart (lexical tone): mā, má, mǎ, mà are four different words
  • Singapore English particles (lah, lor, meh) end the phrase and show the speaker's attitude; some came from Cantonese with their tones (Lim 2007)
  • meh, a skeptical “really?”, is described with a high level pitch (Soh, Lee & Tan 2022); none of the three TTS voices ends it high
MEASURED · THREE TTS VOICES

End of “The shop is open …”, in semitones from each sentence's median

Teal: the sentence ends below its median. Synthetic voices, made in advance; the end is the last 100 ms of voicing.

Why TTS voices sound flat

Averaging many tunes gives a flatter tune

  • The same words were read with different tunes in the training data
  • A model trained to be close on average learns the average tune: smaller rises and falls than a typical real reading
  • Getting the tune back: predict pitch and let people edit it (FastSpeech 2); copy the tune of a recording; or mark it in the text with tags (SSML)
How to measure a tune

Measure the tune in semitones, not hertz

semitones = 12 × log2( f2 / f1 )

f1 = the starting pitch, f2 = the ending pitch, both in Hz

  • Pitch is heard by ratio: 100 → 120 Hz (+20 Hz) and 200 → 240 Hz (+40 Hz) sound like the same step
  • 12 semitones = one octave = double the frequency
  • To ask “does the sentence rise?”, compare the last 100 ms of voicing with the sentence's median pitch (its middle value): positive = ends higher, negative = ends lower
WORKED OUT

Coral: a rise; teal: a fall. Each value computed from the formula. In the practical, this formula is the one line of code you write.

Part II · whose voice is it?

The tune says how it is meant.
The voice says who is saying it.

Zero-shot voice cloning

A few seconds of audio is enough to clone a voice

A few seconds of someone's voicea phone call, a video, a voice note
→
A speaker vector, or a promptno retraining
→
TTS modelany text you type
→
Any sentence, in that voicewords the person never said
  • VALL-E (Microsoft, 2023): 3 seconds; demos only, never released
  • Qwen3-TTS (Alibaba, Jan 2026): “3-second” voice cloning; open weights, free to download
  • Pocket TTS (Kyutai, Jan 2026): about 5 seconds, on a laptop, no GPU
  • VibeVoice (Microsoft, Aug 2025): released openly, then its voice code withdrawn after misuse
A VOICE CAN BE PERSONAL DATA

Nothing in the model checks whose voice it is: any check has to be added by people.

In Singapore, a recording can be personal data under the PDPA if a person can be identified from it (PDPC advisory guidelines, revised 2024).

What has already happened

Cloned voices are already used to deceive

ELECTION · US · JAN 2024

Fake Biden voice tells voters to stay home

Cloned-voice robocalls went out before the New Hampshire primary. The FCC fined the consultant US$6 million.

Source: FCC, 26 Sept 2024
FRAUD · HONG KONG · JAN 2024

Deepfaked bosses on a video call: firm loses HK$200 million

An Arup employee paid out after a call with fake senior colleagues, voices included.

Source: CNN, 16 May 2024
SCAM ADS · SINGAPORE · 2023–2025

Cloned voices of Singapore's leaders sell fake investments

Deepfakes of Lee Hsien Loong (as PM, then as Senior Minister) and of PM Lawrence Wong, often a cloned voice over real footage.

Source: The Straits Times via The Star, 2 June 2024
SECURITY · UK · FEB 2023

AI copy of his voice gets a reporter into his bank account

A free voice clone passed the bank's phone voice ID; the bank also asked for his date of birth.

Source: Vice, 23 Feb 2023

Headlines written for this slide; the links open the original reports.

What can stop misuse

Four safeguards, and none is enough alone

1 · before cloning

Consent

Ask the speaker first, in writing

Apple Personal Voice builds only your own voice; the US actors' union (SAG-AFTRA) requires consent and pay

Gap: only honest users ask

2 · when audio is made

Marking

Hide a watermark in the audio, or attach a signed label of where it came from

AudioSeal (2024); C2PA labels

Gap: labels can be stripped; watermarks attacked

3 · when audio arrives

Detection

A classifier checks whether the speech is synthetic

the ASVspoof challenge, since 2015

Gap: much worse on generators it has not seen

4 · after harm

Law

Penalties and takedowns, which also deter

US FCC (2024): AI voices in robocalls need consent; Singapore (2024 Act): no faked candidates online during an election; EU AI Act: marking from Aug 2026

Gap: slow, and stops at borders

Each closes one gap. Together they raise the cost of misuse; none makes it impossible.

The same technology, the other direction

Cloning also gives people their own voice back

  • Voice banking (VocaliD, this week's TED reading): people with ALS (motor neuron disease) or facing removal of the larynx record phrases while they can still speak
  • Apple Personal Voice (2023): 150 phrases, about 15 minutes, built on the phone; 10 phrases since 2025
  • July 2024: US Representative Jennifer Wexton, who lost much of her speech to a brain disease (progressive supranuclear palsy), spoke in Congress in an AI clone of her earlier voice
ONE TREND, TWO USES

Voice banking once needed 350 to 1,600 recorded phrases; now 10 can be enough.

The less audio a voice needs, the easier it is for patients, and for scammers.

A hard case · hands up, then argue

A voice built from many speakers: whose consent?

A start-up trains a “Singapore English” voice on hundreds of hours of local podcasts, without asking the speakers.

A made-up case.

A
Fine
no one's own voice was copied
B
Not fine
the speakers had to agree first
C
Fine only if
the speakers are credited and paid
A rests on harmB on consentC on a fair shareUnderneath: does a way of speaking belong only to each speaker, or also to the community that speaks that way?
Informed consent, for a voice

Consent must say what, for how long, and who else

Voice consent formFIVE THINGS IT MUST SAY
What: each use is stated; a new use needs new consent
How long: a time limit; withdrawal at any time
Who else: whether others may sell it, pass it on, or train AI on it
Payment: when the voice earns money
Disclosure: listeners are told the voice is synthetic
Speaker's signature
Date
WHERE THESE COME FROM
  • Voice actors' contract terms (NAVA, 2026)
  • SAG-AFTRA's 2023 agreement: consent must describe the use
  • EU GDPR Art. 7: consent can be withdrawn at any time
  • EU AI Act Art. 50: deepfakes must be disclosed
Course definitions, with sources

This week, formally

DEFINITION
Prosody

The pitch, duration and loudness patterns of speech above the level of single phones, across syllables, words and phrases; in J&M's words, “the use of F0, energy, and duration to convey pragmatic, affective, or conversation-interactional meanings”.

Jurafsky & Martin, Speech and Language Processing (3rd ed. draft), ch. 15 §15.3 (Aug 2026 draft; §14.3 in the Jan 2026 release).
DEFINITION
Intonation (tune)

The rise and fall of F0 over a phrase, used to signal meanings such as statement versus question, focus and attitude; a final rise on a yes-no question is called a question rise.

Course wording. J&M ch. 15 §15.3.3 “Tune” (Aug 2026 draft).
DEFINITION
Voice cloning

Making a speech synthesizer speak in a particular person's voice from recordings of that person; zero-shot cloning does it from a few seconds of audio, without retraining the model.

Course wording. Jia et al. (2018); Wang et al. (2023), VALL-E.
DEFINITION
Informed consent (for a voice)

Agreement, given freely and in advance, by a person who understands what their voice will be used for, by whom, for how long, and how to withdraw.

Course wording. NAVA (2026), voice actors' AI contract terms; SAG-AFTRA (2023); EU GDPR Art. 7 (withdrawal).
Exercises: pencils out · ~7 minutes, in pairs

Try it: tunes and voices

EX 12.1

A voice goes from 180 Hz to 270 Hz at the end of a sentence. How many semitones is that? Would it count as a rise?

EX 12.2

A TTS model reads “You're leaving?” and “You're leaving.” the same way. What has it missed, and which one number would show it?

EX 12.3

Why does a model trained to be close on average produce a flat tune? Give one way to get the tune back.

EX 12.4

A model clones any voice from a 6-second clip. Name two harms it makes easier, and one safeguard for each.

Worked answers are in the appendix.

Before you go

Four things to remember

1. Prosody (pitch, duration, loudness) carries meaning the words do not: question or statement, focus, phrasing, attitude; Singapore English particles carry theirs partly in pitch

2. TTS voices often get the words right and the tune wrong: one text allows many tunes, and a model trained on averages flattens them

3. Measure a tune in semitones: 12 × log2(f2/f1), the end of the sentence against its median

4. A few seconds of audio can clone a voice; consent, watermarks, detection and law each help, and none is enough alone

Radar
This week

This week's readings

REQWagner et al. (2019), “Speech synthesis evaluation: state-of-the-art assessment and suggestion for a novel research program”, Proc. SSW10

Why MOS is indispensable and flawed. isca-archive.org/ssw_2019/wagner19_ssw.html

REQHutiri, Papakyriakopoulos & Xiang (2024), “Not my voice! A taxonomy of ethical and safety harms of speech generators”, FAccT 2024

Skim the taxonomy figure; read two case studies. arxiv.org/abs/2402.01708

REQPatel (2013), “Synthetic voices, as unique as fingerprints”, TED

On voice banking: a synthetic voice built for one person. ted.com/talks/rupal_patel_synthetic_voices_as_unique_as_fingerprints

OPTJia et al. (2018), “Transfer learning from speaker verification to multispeaker text-to-speech synthesis”, NeurIPS 2018

§1 and the speaker-embedding plots. arxiv.org/abs/1806.04558

OPTJurafsky & Martin, Speech and Language Processing, ch. 15 §15.3 “Prosody” (Aug 2026 draft)

Prominence, phrasing and tune, in under three pages. web.stanford.edu/~jurafsky/slp3

12:50 · Optional: in class if there is time, or at home

Measure the tune.
Yours first, then the machine's.

Open the Week 12 notebook (link on NTULearn) in Colab and run the setup cell; it works at home too.
On your phone, record “You paid for the kopi.” and “You paid for the kopi?”; name the files statement and question.

The practical · optional · about 40 minutes

Measure your own tune, then three TTS voices

1.  Write the semitone line, then upload your two sentences: does your question end higher? (20 min)
2.  Three TTS voices, the same pair: does the ? change anything? (10 min)
3.  Particles: how do three TTS voices end lah, meh? and lor? Your own reading is optional (10 min)

MMS runs live; SpeechT5 and Kokoro clips are made in advance.

IF SOMETHING GOES WRONG
  • No recording? The notebook uses a synthetic pair: Kokoro's statement, and the same clip with a final rise added by hand (PSOLA)
  • MMS fails to load: pre-made MMS clips take over
  • The tracker finds no pitch: speak closer to the microphone and record again
m

A synthetic voice can say anything.
Ask whose voice it is, and who agreed.

Sources

References

Jurafsky, D. & Martin, J. H. Speech and Language Processing, 3rd ed. draft, ch. 15 §15.3 “Prosody” (Aug 2026; §14.3 in the Jan 2026 release).

Lim, L. (2007). “Mergers and acquisitions: on the ages and origins of Singapore English particles.” World Englishes 26(4), 446–473.

Soh, Y. Q., Lee, J. & Tan, Y.-Y. (2022). “Ethnicity and tone production on Singlish particles.” Languages 7(3), 243. meh with a high level tone.

Wang, Y., Stanton, D., Zhang, Y., et al. (2018). “Style tokens: unsupervised style modeling, control and transfer in end-to-end speech synthesis.” ICML 2018. Averaged prosody.

Ren, Y., Hu, C., Tan, X., et al. (2021). “FastSpeech 2: Fast and high-quality end-to-end text to speech.” ICLR 2021. arXiv:2006.04558.

W3C (2010). Speech Synthesis Markup Language (SSML) Version 1.1. W3C Recommendation, 7 September 2010.

Pratap, V., Tjandra, A., Shi, B., et al. (2024). “Scaling speech technology to 1,000+ languages.” JMLR 25(97). MMS.

Ao, J., Wang, R., Zhou, L., et al. (2022). “SpeechT5: Unified-modal encoder-decoder pre-training for spoken language processing.” ACL 2022. Kokoro-82M (hexgrad, 2024), Apache-2.0.

Mauch, M. & Dixon, S. (2014). “pYIN: a fundamental frequency estimator using probabilistic threshold distributions.” ICASSP 2014.

Wang, C., Chen, S., Wu, Y., et al. (2023). “Neural codec language models are zero-shot text to speech synthesizers.” arXiv:2301.02111. VALL-E.

Qwen Team (2026). “Qwen3-TTS technical report.” arXiv:2601.15621; model cards on Hugging Face (Apache-2.0).

Kyutai (13 Jan 2026). “Pocket TTS” (blog and model card). Microsoft (2025). VibeVoice GitHub README, entry of 5 Sept 2025 (TTS code removed after misuse).

Jia, Y., Zhang, Y., Weiss, R. J., et al. (2018). “Transfer learning from speaker verification to multispeaker text-to-speech synthesis.” NeurIPS 2018.

PDPC Singapore (2024). Advisory Guidelines on the PDPA for Selected Topics, revised 23 May 2024. Recordings as personal data.

Wagner, P., Beskow, J., Betz, S., et al. (2019). “Speech synthesis evaluation: state-of-the-art assessment and suggestion for a novel research program.” Proc. SSW 10, 105–110.

Hutiri, W., Papakyriakopoulos, O. & Xiang, A. (2024). “Not my voice! A taxonomy of ethical and safety harms of speech generators.” FAccT 2024, 359–376.

FCC (2024). Declaratory Ruling FCC 24-17, 8 Feb 2024 (AI voices in robocalls); forfeiture order against S. Kramer, 26 Sept 2024.

CNN (16 May 2024). Arup revealed as the victim of a US$25 million deepfake scam. The Straits Times via The Star (29 Dec 2023; 2 June 2024) and Bernama via The Star (7 Mar 2025): deepfakes of Singapore's leaders.

Cox, J. (23 Feb 2023). “How I broke into a bank account with an AI-generated voice.” Vice.

San Roman, R., Fernandez, P., Elsahar, H., et al. (2024). “Proactive detection of voice cloning with localized watermarking.” ICML 2024. AudioSeal.

C2PA (2026). Content Credentials deployment guidance. Coalition for Content Provenance and Authenticity.

Müller, N. M., Czempin, P., Dieckmann, F., et al. (2022). “Does audio deepfake detection generalize?” Interspeech 2022. ASVspoof: asvspoof.org.

Singapore (2024). Elections (Integrity of Online Advertising) (Amendment) Act 2024 (No. 34 of 2024), in force 22 Jan 2025. EU AI Act, Regulation (EU) 2024/1689, Art. 50.

SAG-AFTRA (2023). TV/Theatrical agreement: digital replica provisions. NAVA (2026). “Synthetic/AI voices.” navavoices.org. EU GDPR Art. 7.

Patel, R. (2013). “Synthetic voices, as unique as fingerprints.” TEDWomen. RCSLT (2020). Voice banking factsheet.

Apple (16 May 2023; 13 May 2025). Newsroom: Personal Voice. 9to5Mac (19 Mar 2026): 150 prompts before iOS 26, 10 since. Fox 9 (25 July 2024): Rep. Jennifer Wexton's speech in an AI-cloned voice.

Appendix: worked answers

Answers

EX 12.1

12 × log2(270 / 180) = 12 × log2(1.5) ≈ +7.0 semitones. Yes: a large rise, more than half an octave.

EX 12.2

It has missed the intonation: in “You're leaving?” only a final rise makes it a question. The one number: the question's end minus the statement's end (each the last 100 ms against the sentence's median), in semitones; it should be clearly positive.

EX 12.3

The same text appears with different tunes in training; the tune that is closest to all of them on average is their average, which rises and falls less than a typical reading. To get it back: predict pitch explicitly and edit it (FastSpeech 2), copy the style of a reference recording, or mark it in the text with SSML.

EX 12.4

For example: scam calls in a relative's voice (safeguard: call back on a number you know; consent checks in the product) and fake statements by a politician (safeguard: watermarks and signed records of where a file came from; election laws against deepfakes). Bank voice logins (safeguard: a second factor) also work.