HG4052 · Speech Synthesis & Recognition
ɑ
HG4052 · Speech Synthesis & Recognition / Week 10

Making It Talk.

Spelling and Sound.

Chenzi Xu · chenzi.xu@ntu.edu.sg

→ / space to advance · F for fullscreen · M for the slide menu
t
The shape of the morning

Today, by the clock

09:40  Five echoes from Week 9, then Assignments 1 and 2 (15 min)
09:55  Lecture I: the front end, formant synthesis, then KlattGrid by hand
10:45  Break (10 min)
10:55  Lecture II: concatenative synthesis, then articulatory synthesis
11:40  Break + Colab setup (10 min)
11:50  Practical: a normalizer rule, a cost lattice, a vocal tract
12:50  Wrap-up and five minutes of Assignment 2 questions
Before new material · 3 minutes, tell your neighbor

Five echoes from Week 9

1. “Okay lah, we'll order 2 kopi”: what did the normalizer do to “2” before scoring?

rewrote it as “two” on both sides, so a correct transcript went from 100% WER to 0%

2. What can one overall WER hide?

a gap between groups: report WER for each group, with utterance and word counts

3. Forced alignment is Viterbi with what already known?

the words: Viterbi decides only when each phone starts and ends

4. What does a forced alignment give every phone?

a start and an end time, so a duration

5. What happens to a word missing from the aligner's dictionary, like “makan”?

it gets no phones (MFA labels it spn, spoken noise); the fix is one dictionary line

A
Assignments · one due Sunday, one released today

Assignment 1 ends; Assignment 2 begins

ASSIGNMENT 1 · EXTENDED: DUE SUN 25 OCT, 23:59
  • Three files via the A1 submission link on NTULearn: notebook or scripts, report (PDF), results file (.csv)
  • Attach the plagiarism declaration form
  • Late work loses 5% per day; not accepted after the seventh day without a valid reason
ASSIGNMENT 2 · RELEASED TODAY · 30%
  • Individual: present one open-weights speech synthesis model at the Week 13 poster fair
  • Explain how it turns text into sound; test it on sentences you design for Singapore speech
  • A one-page abstract, a digital poster, a three-minute pitch
  • If you clone, use only your own voice, a synthetic voice or a public-domain recording
  • Poster PDF emailed by Thu 12 Nov, 09:30; abstract and poster on NTULearn by Fri 13 Nov, 23:59

A1 put a recognizer on trial; A2 does the same for a synthesizer. Everything task-specific is in the briefs on NTULearn.

Δ
Today

Roadmap

Recognition and synthesis, side by side

Recognition and synthesis are inverses

The classical pipeline

The front end: from raw text to a plan

raw text“Dr Tan paid $2.50”
→
normalizeexpand numbers, symbols
→
G2Pwords → phones
→
prosodypitch, duration, stress
→
specificationphones, durations, F0
→
back endmake the wave
  • The front end is linguistics before acoustics: phonology, morphology and syntax decide the plan
  • The back end only renders the plan
  • Each arrow can fail on its own
  • A front-end error is never repaired later: if G2P gives the wrong phone, the back end pronounces the wrong phone clearly

This is the classical pipeline. Neural models (Week 11) learn some of these steps inside the network, but most still normalize text first.

Normalization, for speaking

Text normalization, for speaking

TO SCORE · WEEK 9

“2 kopi” → “two kopi” on both sides

Any fixed rule works

TO SPEAK · TODAY

“2 kopi” → which words?

The reading must fit the context

writtenusual readinganother readingwhat decides
“$2.50”two dollars fiftytwo fiftyhouse style
“Dr”Doctor (Dr Tan)Drive (Holland Dr)position: before a name, or after a street name
“23/10”the 23rd of Octobertwenty-three over tena date or a number?
“Blk 210”Block two-tenBlock two-one-zerolocal habit
“MRT”M R T“murt”an initialism, read as letters

The surrounding words, not the token, decide the reading. An A2 example question: does the model read “S$3.50, 12/11, Blk 123” as a Singaporean would?

g
The heart of the front end

Grapheme-to-phoneme (G2P): from letters to phones

  • English orthography hides the sounds: “through”, “though”, “tough” share letters, not phones
  • Dictionary first (CMUdict); letter-to-sound rules or a trained model for words not in it (out-of-vocabulary, OOV)
  • Heteronyms (one spelling, two pronunciations) need the sentence: “a record” vs “to record” is a part-of-speech decision; “I read it yesterday” vs “I will read it” needs the tense
When a word is not in the dictionary
  • G2P falls back on letter-to-sound rules: written by hand (as in eSpeak NG) or learned from a dictionary such as CMUdict
  • eSpeak NG (US English) reads makan as /ˈmækən/, kopi as /ˈkoʊpi/, Tampines as /ˈtæmpaɪnz/
  • The same gap as Week 9, where MFA turned a missing word into spn
One spelling, two pronunciations

the desert / to desert
ˈdɛzərt   dɪˈzɜrt

record (noun) / record (verb)
ˈrɛkərd   rɪˈkɔrd

stress and vowels flip with the syntax

The back end before neural models: three classical approaches

Where does the sound come from?

FORMANT
F1F2F3

Build the output from rules

Set the source and the formants by hand; the sound is calculated.

Try it: KlattGrid in Praat

CONCATENATIVE

Reuse real recordings

Cut one speaker’s recordings into small units and join them into new sentences.

Try it: a cost lattice

ARTICULATORY
tongueglottislips

Simulate the physical cause

Shape a virtual vocal tract, push air through it, and physics makes the sound.

Try it: Pink Trombone

All three are pre-neural (neural models: Week 11).
Each renders the same plan from the front end: phones, durations and F0.

Formant synthesis, in one line

Formant synthesis is
source-filter theory, run as a generator.

Week 3: a vowel is a source shaped by a filter.
Here: choose the source, set the formants by hand, and a vowel comes out.

Formant synthesis, from the 1950s

Formant synthesis: draw the resonances

  • Source: a buzz at F0 with all its harmonics (voiced), or noise (voiceless)
  • Filter: resonators at F1, F2 and F3; their frequencies set the vowel
  • No recordings at all: the voice is a handful of numbers over time, drawn by a phonetician
  • The Klatt synthesizer (1980; revised 1990), in Praat as the KlattGrid: you try it at the end of this section
  • Fully controllable; clearly robotic
Why linguists use it: stimuli for perception experiments
hear step:
  • One cue at a time: only the start of F2 moves, in equal steps (illustrative values)
  • Listeners' “ba or da?” answers switch suddenly between two steps: categorical perception (Liberman et al. 1957)
  • Like Week 2's Ganong continuum: where does /ba/ become /da/?
Worked example: one buzz, three vowels

Sweep the formants, hear the vowel change

  • The gray comb is the source: F0 harmonics, unchanging
  • The coral curve is the filter: peaks at F1, F2, F3
  • The teal comb is the output: harmonics shaped by the filter, the spectrum you hear

Only the filter changes; the source is the same for Week 1's /i/, /ɑ/ and /u/. The sound: Praat's KlattGrid making the same sweep.

Formant synthesis

Formant synthesis: strengths and weaknesses

Strengths
  • Full control of every parameter
  • Small: kilobytes, no stored audio
  • Any sound, human or not
  • Every parameter has a phonetic meaning
Weaknesses
  • Sounds robotic: a small rule set misses the detail of real voices
  • Each language and voice is tuned by hand
  • Natural prosody is very hard to write as rules
A VOICE KEPT FOR LIFE
  • Stephen Hawking kept his 1980s formant voice until his death in 2018: it had become part of his identity
  • Formant speech is highly intelligible, even when it is not natural

A preview of Week 12: a voice is a person.

i
Praat
PRAAT
15 minutes · Praat on your laptop

Hands-on: make three vowels from formants

1.  In Praat: Open Praat script… → klatt_vowels.praat (Week 10 notebook, Part 1) → Run
2.  Choose /i/, type F1 270 and F2 2290; listen, then click View & Edit
3.  Repeat for /ɑ/ (730, 1090) and /u/ (300, 870), or use your own Week 3 values
4.  Try a pattern no human vocal tract makes: F1 1400, F2 1500. What do you hear?

Which formant moved /i/ toward /ɑ/: F1, F2, or both?

IF SOMETHING GOES WRONG
  • The notebook's Part 1 plays ready-made recordings of all four vowels
  • Red formant dots in View & Edit can be wrong on synthetic vowels: read the dark bands instead
k
10:45 · Break

Built from numbers.
Next: built from recordings.

After the break: concatenative synthesis, which joins pieces of recorded speech, then articulatory synthesis.

The opposite philosophy

Why synthesize sound you could just replay?

  • Record one speaker for hours, cut the audio into small units, and join them into new sentences
  • Every unit is real human speech; the difficulty is the joins
  • The standard unit: the diphone, from the middle of one phone to the middle of the next
WHY DIPHONES, NOT PHONES
  • Coarticulation (Week 3): each phone's edges change with its neighbors
  • Phone middles are the most stable part, so diphones are cut there
  • Each transition stays inside one recorded unit, never at a join
Concatenative synthesis: a short history

From a talking clock to unit selection

1933
A talking clock
Paris: words recorded on photographic strips, played in order
1953
One piece per phoneme
Harris (Columbia): mostly unintelligible, because coarticulation is lost
1958
The diphone
Peterson, Wang & Sivertsen (Michigan): cut at phone middles, the most stable part
1967–77
Diphones as numbers
Stored as synthesizer settings, not audio: IBM, then Bell Labs
1990demo in Praat
PSOLA
CNET, France: change a recording’s pitch and duration
1996★ hardest idea
Unit selection
Hunt & Black (ATR, Japan): thousands of candidates, chosen by target and join cost
2001
Unit selection goes commercial
AT&T Natural Voices; Siri still used a hybrid in 2017
2016Week 11
WaveNet
A neural network beats splicing in listening tests

Sources: Abgrall et al. (2013); Harris (1953); Peterson, Wang & Sivertsen (1958); Dixon & Maxey (1968); Olive (1977); Klatt (1987); Moulines & Charpentier (1990); Hunt & Black (1996); Burdick (2003); Capes et al. (2017); van den Oord et al. (2016).

Changing the prosody of a recording · Moulines & Charpentier (1990)

PSOLA: new pitch, same vowel

  • A diphone keeps the speaker’s pitch and timing; the plan asks for others
  • Cut the wave into overlapping pieces, one per pitch pulse
  • Pieces closer together: higher pitch. Repeat or drop pieces: longer or shorter
  • Each piece keeps its shape, so the formants and the vowel stay (nearly) the same, if the change is moderate
How it works: one synthetic vowel, five steps

PSOLA, step by step: pitch up by a third

ɑ
Praat
PRAAT
10 minutes · Praat on your laptop

Hands-on: raise the pitch of a vowel by a third

1.  Select a Sound: your /ɑ/ from the KlattGrid hands-on, or record yourself saying “kopi” (New → Record mono Sound…)
2.  Manipulate → To Manipulation… → OK, then View & Edit: the blue dots are the pitch
3.  Drag across the whole picture to select it, then Pitch → Multiply pitch frequencies… → 1.333
4.  Click a bar below the picture to listen; Shift-click plays the original
5.  Now try 2 (an octave up) and 0.5 (an octave down). Where does it start to sound wrong?

Did the vowel change, or only the pitch? (The KlattGrid /ɑ/ goes from 120 Hz to 160 Hz.)

IF SOMETHING GOES WRONG
  • No blue dots: the sound has no voicing; choose a vowel or a voiced word
  • A jumpy pitch line: Pitch → Stylize pitch (2 st)
  • To keep the result: File → Publish resynthesis (a new Sound)
  • Praat calls the method overlap-add
★ The hardest idea this week

Unit selection: the cheapest path
through real recordings.

Target cost: does this recording fit the plan? Join cost: does it join smoothly to its neighbor?
Viterbi finds the sequence with the lowest total.

Concatenative synthesis, scaled up (Hunt & Black 1996)

Unit selection: a huge library, one best path

  • Not one recording per unit but thousands, spoken in every context
  • Two costs to minimize: target cost (does this unit fit the plan?) and join cost (do neighboring units join smoothly?)
  • Choose the sequence of units with the lowest total cost
  • The table trick's fourth appearance: DTW kept the minimum (Week 5), Viterbi the maximum (Week 6), CTC the sum (Week 8)
  • Here Viterbi keeps the lowest total cost, and so chooses recordings
  • Lattice: one column per target diphone, one node per candidate recording
YOU HAVE HEARD THIS
  • Car navigation voices and automated phone menus of the 2000s
  • Siri's voices as late as 2017 (hybrid unit selection; Capes et al. 2017)
  • Natural when the right units exist; an audible jump when a rare word forces a poor join
Worked example (invented rules): where a target cost comes from

Target cost: compare the plan with each recording

The plan for /ɔ-p/ in “kopi”, from the front end

PENALTY RULES (INVENTED)
    IN REAL SYSTEMS
    • Many more features: neighboring phones, position in the phrase, and more
    • The weights are tuned, by hand or automatically
    • Newer systems measure the distance from what a statistical model predicts (Google, Gonzalvo et al. 2016; Siri, Capes et al. 2017)
    Worked example (illustrative costs): three diphones for “kopi”

    The lattice, and the cheapest path through it

    • Each row: one recording of “kopi”; the number in a node is its target cost: how badly it fits the plan (0 = perfect)
    • Each edge: the join cost: 0 inside one recording, otherwise 0.3
    • Total = target costs + join costs; the table fills like DTW's (Week 5) and traces back like Viterbi's (Week 6)
    V(u) = T(u) + minp [ V(p) + J(p, u) ]

    V: cheapest total from the start to u (a running total)
    u: a candidate in this column · p: one in the previous column

    Concatenative synthesis

    Concatenative synthesis: strengths and weaknesses

    Strengths
    • With the right units, close to the recorded speaker
    • No acoustic model to get wrong
    • The voice is that speaker's own voice
    Weaknesses
    • Hours of studio recording per voice
    • Gigabytes of audio
    • Audible join glitches in rare contexts
    • Only the recorded style: a calm speaker's units cannot whisper
    HYBRIDS, THEN NEURAL
    • Later systems used a statistical model to smooth the joins
    • From 2016, neural networks generated the whole waveform (WaveNet), and it sounded better than splicing
    • Week 11 tells that story
    ə
    The third approach

    Articulatory synthesis: model the vocal tract, not the sound

    • Formant synthesis models the output; concatenation reuses recordings
    • Articulatory synthesis models the vocal tract itself: tongue, lips, glottis, as a physical tube
    • Push air through the simulated tube, shape it, and sound comes out by physics, not by rules or replay
    ʃ
    Articulatory synthesis · what to try

    Play a vocal tract with your hands

    • Drag the tongue body and watch a vowel slide around the space you drew in Week 1
    • Pinch a constriction for a fricative; tap a full closure for a stop burst
    • The articulators from phonetics class, as controls you can hear
    • In class: /i/, /ɑ/, /u/, then /ʃ/, then a stop
    • Then a whole word: much harder, because every articulator must move in time
    STRENGTHS AND WEAKNESSES
    • Closest to how speech is produced
    • Hardest to control, slowest to run, rarely used in products
    • Lets you hear an articulatory diagram

    Research-grade: Birkholz's VocalTractLab.

    Live demo · Pink Trombone v1.1 · © 2017 Neil Thapen, MIT License

    Pink Trombone: a vocal tract you can play

    Pink Trombone: a cross-section of the vocal tract with tongue and voicebox controls

    Press start, then click once inside to begin the sound; drag the tongue control or touch the tract. Works offline; the original is at dood.al/pinktrombone.

    Three classical back ends

    The three answers, compared

    FORMANT

    Model the output

    Build the spectrum from rules. Tiny, controllable, robotic. Any sound, no data.

    CONCATENATIVE

    Reuse the recordings

    Join recorded speech fragments. Natural on common text, brittle on rare, and huge.

    ARTICULATORY

    Model the cause

    Simulate the tract. Physically realistic; slow and hard to control; rarely used in products.

    ?
    Vote first, then discuss · hands up

    You must build a Singlish TTS next month. Which classical approach?

    A  formant: hand-tune the vowels and particles yourself

    B  unit selection: record a Singlish speaker for many hours

    C  articulatory: simulate a Singaporean vocal tract

    V
    11:40 · Break, then hands on keyboards

    Fix the text.
    Then play the tract.

    The practical

    A front end and two back ends, by hand

    • Front end, the one code TODO: add one expansion rule to the provided normalizer (for example “$2.50” → “two dollars fifty”); run it on five Singapore strings
    • Unit selection, no code to write: run the provided cheapest-path cell on a small lattice; change one join cost; listen to a good join and a bad join
    • Articulatory: in Pink Trombone, make /i ɑ u/, a fricative and a stop; screenshot the tract shape for each

    Stretch (take-home): add “kopi” and “lah” to the front-end dictionary with phone strings, and synthesize “can lah” in your KlattGrid voice.

    You leave with this, made by you

    illustrative layout: F1 and F2 for /i ɑ u/ (Peterson & Barney means), as you will set them in KlattGrid

    The practical hour

    The practical hour, stage by stage

    1.  The normalizer TODO: one expansion rule, then five Singapore strings (20 min)
    2.  Unit selection: run the cheapest-path cell, change one join cost; a good join and a bad join (15 min)
    3.  Pink Trombone: three vowels, a fricative, a stop (20 min)

    KlattGrid (notebook Part 1) was done in Lecture I; start at Part 2. The take-home is optional.

    YOU LEAVE WITH
    • A normalizer rule you wrote
    • A cheapest path, and a bad join you can hear
    • Tract shapes for five sounds
    • Plus, from Lecture I: three vowels you set in KlattGrid
    Course definitions, with sources

    This week, formally

    DEFINITION
    Grapheme-to-phoneme (G2P)

    The front-end task of mapping written words to their pronunciations, by dictionary lookup with a rule-based or statistical model for unseen words and context for heteronyms.

    Course wording. Taylor (2009), Text-to-Speech Synthesis, ch. “Pronunciation”, 192–224; lexicon: CMUdict.
    DEFINITION
    Formant synthesis

    Rule-based synthesis that generates speech from an explicit source (voicing or noise) and a set of resonators at the formant frequencies: the source-filter model, run as a generator.

    Klatt (1980), JASA 67(3), 971–995; Praat's KlattGrid implements Klatt & Klatt (1990).
    DEFINITION
    Unit selection synthesis

    Concatenative synthesis that chooses, from a large recorded database, the unit sequence minimizing a combination of target cost (fit to the specification) and join cost (splice quality), by dynamic programming.

    Hunt & Black (1996), Proc. ICASSP 1996, vol. 1, 373–376; J&M Ch 17, Historical Notes (Aug 2026 draft).
    DEFINITION
    Articulatory synthesis

    Synthesis that simulates the physical vocal tract, generating sound from a model of the articulators and airflow rather than from acoustic targets or recordings.

    J&M Ch 17, Historical Notes (Aug 2026 draft); tools: Thapen (2017), Pink Trombone; Birkholz (2013), PLoS ONE 8(4), e60603 (VocalTractLab).
    Exercises: pencils out · ~7 minutes, in pairs

    Try it: front end and back end

    EX 10.1

    Expand for speech, and give one alternative reading each: “Dr Tan lives at 21 Jln Besar, unit 03-11, and paid $4.50.” Which expansions need context the string alone does not provide?

    EX 10.2

    A dictionary lists “read” with two pronunciations. Why can G2P not choose between them by lookup alone, and what information resolves it? Name one other English heteronym.

    EX 10.3

    Two target diphones, two candidates each. Target costs: col 1 [0.2, 0.6]; col 2 [0.5, 0.3]. Join cost: 0 if both come from the same recording (i = j), otherwise 0.1. Find the minimum-total path and its cost.

    EX 10.4

    Match each approach (formant, unit selection, articulatory) to “models the output / reuses recordings / models the cause”. Which degrades most gracefully on a brand-new Singlish word in no database, and why?

    Worked answers in the appendix at the end of this deck.

    Before you go

    Four things to remember

    1. Synthesis is recognition reversed: the text leaves out pitch, timing and voice, so the synthesizer must supply them

    2. Half of classical TTS is the front end: normalization and G2P, linguistics before acoustics

    3. Three back ends: model the output, reuse recordings, model the cause

    4. Unit selection is the table trick's fourth appearance: the cheapest path of target plus join costs through a lattice of recordings

    Radar
    This week

    This week's readings

    REQJurafsky & Martin, SLP (3rd ed., free online): Ch 17, “Text-to-Speech”

    Chapter numbers follow the Aug 2026 draft. For today: §17.1 and the Historical Notes (formant, concatenative and articulatory synthesis; unit selection). §§17.2–17.4 preview Week 11; §17.4 (MOS, WER, speaker similarity) is also A2 reading. web.stanford.edu/~jurafsky/slp3/17.pdf

    REQKing, S., speech.zone course materials (University of Edinburgh)

    Front end: Speech Processing, Module 5. Unit selection: Speech Synthesis, Module 2.

    REQPraat manual, “KlattGrid” pages (Weenink)

    Skim them so the interface is familiar before the practical. fon.hum.uva.nl/praat/manual/KlattGrid.html

    OPTKlatt (1987), “Review of text-to-speech conversion for English”, JASA 82(3), 737–793

    By the synthesizer's creator. Free copy (sail.usc.edu)

    OPTTaylor (2009), Text-to-Speech Synthesis, Cambridge University Press

    The classical text-to-speech textbook (library copy).

    Next week

    Readings for Week 11

    REQHugging Face Audio Course, Unit 6, “From text to speech”

    The models used in the Week 11 practical. huggingface.co/learn/audio-course/chapter6/introduction

    REQShen et al. (2018), “Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions” (Tacotron 2), Proc. ICASSP 2018

    §§1–2 only, then the audio samples. arxiv.org/abs/1712.05884

    REQ“WaveNet: A generative model for raw audio”, DeepMind blog (2016)

    The blog, not the paper: listen to the embedded audio. deepmind.google/blog/wavenet-a-generative-model-for-raw-audio

    OPTKong, Kim & Bae (2020), “HiFi-GAN”, NeurIPS 2020

    Skim the discriminator design. arxiv.org/abs/2010.05646

    OPTKim, Kong & Son (2021), VITS, ICML 2021

    Introduction and demo samples; the math is heavy. arxiv.org/abs/2106.06103

    ɑ

    A voice is a
    filter, a library, or a tract.

    Sources

    References

    Jurafsky, D. & Martin, J. H. Speech and Language Processing, 3rd ed. (Aug 2026 draft). Ch. 17, “Text-to-Speech”: §17.1 and the Historical Notes. The reading.

    Klatt, D. H. (1980). “Software for a cascade/parallel formant synthesizer.” JASA 67(3), 971–995. The formant synthesizer.

    Klatt, D. H. & Klatt, L. C. (1990). “Analysis, synthesis, and perception of voice quality variations among female and male talkers.” JASA 87(2), 820–857. The version Praat's KlattGrid implements.

    Klatt, D. H. (1987). “Review of text-to-speech conversion for English.” JASA 82(3), 737–793.

    Liberman, A. M., Harris, K. S., Hoffman, H. S. & Griffith, B. C. (1957). “The discrimination of speech sounds within and across phoneme boundaries.” Journal of Experimental Psychology 54(5), 358–368. Categorical perception.

    Peterson, G. E. & Barney, H. L. (1952). “Control methods used in a study of the vowels.” JASA 24(2), 175–184. Formant values in the vowel sweep.

    Abgrall, M., Blondé, P., Lours, M., Prodhomme, M. & Dimarcq, N. (2013). “Les 80 ans de l'horloge parlante.” l'Astronomie, February, 32–37. The 1933 Paris speaking clock.

    Harris, C. M. (1953). “A study of the building blocks in speech.” JASA 25(5), 962–969.

    Peterson, G. E., Wang, W. S-Y. & Sivertsen, E. (1958). “Segmentation techniques in speech synthesis.” JASA 30(8), 739–742. The diphone (“dyad”).

    Dixon, N. R. & Maxey, H. D. (1968). “Terminal analog synthesis of continuous speech using the diphone method of segment assembly.” IEEE Trans. Audio and Electroacoustics 16(1), 40–50.

    Olive, J. P. (1977). “Rule synthesis of speech from dyadic units.” Proc. ICASSP 1977, vol. 2, 568–570.

    Moulines, E. & Charpentier, F. (1990). “Pitch-synchronous waveform processing techniques for text-to-speech synthesis using diphones.” Speech Communication 9(5–6), 453–467. PSOLA.

    Burdick, A. (2003). “The mathematics of artificial speech.” Discover, January. AT&T Natural Voices.

    Hunt, A. J. & Black, A. W. (1996). “Unit selection in a concatenative speech synthesis system using a large speech database.” Proc. ICASSP 1996, vol. 1, 373–376. Target and join costs.

    Gonzalvo, X., Tazari, S., Chan, C., Becker, M., Gutkin, A. & Silen, H. (2016). “Recent advances in Google real-time HMM-driven unit selection synthesizer.” Proc. Interspeech 2016, 2238–2242.

    Capes, T., Coles, P., Conkie, A., et al. (2017). “Siri on-device deep learning-guided unit selection text-to-speech system.” Proc. Interspeech 2017, 4011–4015.

    Taylor, P. (2009). Text-to-Speech Synthesis. Cambridge University Press. “Pronunciation” (192–224); “Unit-selection synthesis” (474–516).

    Carnegie Mellon University. The CMU Pronouncing Dictionary (CMUdict). speech.cs.cmu.edu/cgi-bin/cmudict. The G2P lexicon.

    King, S. Speech Processing (Module 5) and Speech Synthesis (Module 2) course materials. speech.zone, University of Edinburgh.

    Thapen, N. (2017). Pink Trombone, v1.1. dood.al/pinktrombone. MIT License; the live demo runs from a copy inside these slides (four page-layout sizes given “px” units; the author link opens in a new tab; otherwise unchanged).

    Birkholz, P. (2013). “Modeling consonant-vowel coarticulation for articulatory speech synthesis.” PLoS ONE 8(4), e60603. VocalTractLab: vocaltractlab.de.

    van den Oord, A., Dieleman, S., Zen, H., et al. (2016). “WaveNet: A generative model for raw audio.” arXiv:1609.03499.

    Medeiros, J. (2015). “How Intel gave Stephen Hawking a voice.” Wired, 13 January. Hawking's formant voice.

    Appendix: worked answers

    Answers

    EX 10.1

    “Dr” → Doctor (before a name), not Drive; “21” → twenty-one; “Jln” → Jalan; “03-11” → oh three, eleven (a unit number), not “three minus eleven”; “$4.50” → four dollars fifty. Context decides “Dr” and “03-11”: the token alone cannot.

    EX 10.2

    One spelling, two pronunciations, so lookup returns both. The choice needs the tense (present “read” /riːd/, past “read” /rɛd/), which comes from the surrounding words (yesterday, will), not from the word itself. Another heteronym: “lead” (verb /liːd/, the metal /lɛd/), where the part of speech decides; or “tear”, “bow”, “live”.

    EX 10.3

    Paths (col1 → col2): 1→1 = 0.2 + 0 + 0.5 = 0.7; 1→2 = 0.2 + 0.1 + 0.3 = 0.6; 2→1 = 0.6 + 0.1 + 0.5 = 1.2; 2→2 = 0.6 + 0 + 0.3 = 0.9. Minimum is 1 → 2, cost 0.6: the cheap target in column 1 is worth a small join penalty to reach the cheap target in column 2.

    EX 10.4

    Formant synthesis models the output, unit selection reuses recordings, articulatory synthesis models the cause. Most graceful on a new word: formant (or articulatory), since rules or physics need no recording of the word; unit selection can only recombine what was recorded, so a new context forces a poor join.