%

HG4052 · Speech Synthesis & Recognition / Week 9

The Audit.

Errors and Alignments.

Chenzi Xu · chenzi.xu@ntu.edu.sg

t

The shape of the morning

Today, by the clock

09:40 Five echoes from Week 8, then Assignment 1 (15 min)

09:55 Lecture I: normalization, WER by group, and a published audit

10:40 Break (10 min)

10:50 Lecture II: the National Speech Corpus, then forced alignment

11:40 Break + Colab setup (10 min)

11:50 Practical: an audit table, an error log, and a TextGrid

12:50 Wrap-up and five minutes of Assignment 1 questions

Before new material · 3 minutes, tell your neighbour

Five echoes from Week 8

1. The CTC collapse rule, in order?

merge repeats, then delete blanks

2. What does attention compute for each output token?

a probability distribution over the encoder frames, then their weighted sum

3. How do you score a transcript against its reference?

lowercase and strip punctuation on both sides, then WER = (S + D + I) ÷ reference words

4. What can Whisper print for five seconds of silence?

fluent text that nobody said: a hallucination

5. With no lexicon file and no separate language model, what remains to change in a recognizer?

its training data

A

Assignment 1 · one week to go

Assignment 1: steps 5 and 6 today

Steps 2 to 4 (run both systems, one normalizer, WER) were the Week 8 practical
Steps 5 and 6 are today's practical; in A1 you repeat them with your own numbers
Step 5, the audit table: overall WER for both systems, then WER split by your contrast, with utterance and reference-word counts in each cell; a table and a chart
Step 6, the error analysis: listen to the group with the highest WER; log each error with reference, hypothesis and one category from the brief

KEY DATES

Today: 16 Oct (Week 9)
Due: Fri 23 Oct, 23:59
Weight: 30%

Everything task-specific is in the brief on NTULearn. Corpus audio and transcripts stay inside the module.

Δ

Today

Roadmap

Normalization: why a correct transcript can score 100% WER
Disaggregation: what one overall WER hides; Koenecke et al. (2020)
The National Speech Corpus: parts, speakers, metadata, licence
Forced alignment: Viterbi with the words known
Practical: an audit table, an error log, a TextGrid

Hold all morning: a WER gap describes the system and the test, never a deficit in the speakers.

Normalization, before scoring

A correct transcript that scores 100% WER

Before normalizing

WER = (5 + 0 + 1) ÷ 6 = 100%

REF

Okay

lah,

we’ll

order

2

kopi.

HYP

okay

lah

we

will

order

two

kopi

EDIT

S

I

✓

S

After normalizing: lowercase, strip punctuation, we’ll → we will, 2 → two

WER = 0 ÷ 7 = 0%

REF

okay

lah

we

will

order

two

kopi

HYP

okay

lah

we

will

order

two

kopi

EDIT

✓

Ssubstitution Iinsertion ✓match D, deletion: none here N: words in the reference (6, then 7: we’ll became two words)

Normalization, before scoring

The normalizer is part of the metric

The normalizer

Fixed rules applied to reference and hypothesis before scoring
Each rule changes the WER
Two papers with different normalizers report different quantities, both called WER

In practice

Whisper ships its own normalizer: it turns “two” into “2”, the opposite of the class rule
Either rule works if both sides get the same one
In A1: fix yours before seeing results; state it in the report

Limits of WER

What WER cannot see

NO SPACES, NO WORDS

Use CER (character error rate)

Same formula, over characters
Mandarin and Japanese text has no spaces between words
One Singapore code-switched line can mix spaced English words and unspaced Chinese characters

EVERY ERROR COSTS 1

Meaning does not count

“gonna” → “gotta” and “gonna” → “banana”: 1 error each
“gonna” → “going to”: 2 errors (1 substitution + 1 insertion)
Dropping “lah” and dropping “not”: 1 error each

WHOSE ORTHOGRAPHY?

Reference text is a choice

Is the reference “going to” or “gonna”? “makan” or a translation?
The corpus's transcription conventions become the definition of correct

Rule of the day

Fix the normalizer before you look at any WER
Use the same one for both systems
Report it with the reference conventions and the test population

Illustrative numbers · one number hides two

Split the average

overall WER = w₁ × WER on the majority accent + w₂ × WER on code-switched speech, where each w is the group's share of the reference words (here w₁ = 0.75, w₂ = 0.25)
A different test-set mix can change which system has the lower overall WER, with no change to either model
Disaggregation: WER reported separately for each named group of speakers, with utterance and word counts

Averaging WER · illustrative numbers

Two ways to average WER

Pooled WER

WER_pooled=Σ_i E_iΣ_i N_i

Total errors divided by total reference words: long utterances count more
What jiwer returns for a list of utterances

Mean of per-utterance WERs

WER_mean=1UΣ_iE_iN_i

Each utterance's WER first, then their average: every utterance counts equally, however short

E_i: errors in utterance i N_i: reference words in utterance i U: number of utterances Σ_i: add up over the utterances

Example: utterance 1 has 2 words and 2 errors; utterance 2 has 18 words and 0 errors

POOLED

2 + 02 + 18=220=10%

MEAN

12(22+018)=12(100% + 0%)=50%

Report which average you use, with utterances and reference words for every cell
Pooled over groups, it is the weighting in Split the average: each group's WER × its share of the words

Disaggregation, in one line

An average can hide
a gap between groups.

Nothing in the WER formula is biased. The bias is in what gets averaged,
over whom, and which number a team chooses to report.

The published audit · Koenecke et al. (2020)

Five recognizers, one racial gap: 35% vs 19% WER

Five commercial recognizers (Amazon, Apple, Google, IBM, Microsoft) transcribing interview snippets
Black speakers from CORAAL (Corpus of Regional African American Language), white speakers from Voices of California: 2,141 snippets per group, matched on age, gender and duration
Average WER 35% vs 19%; all five systems show the gap
On 206 short phrases spoken by both groups, WER was still about twice as large for Black speakers
Among Black speakers, WER rose with dialect density (the share of African American English features in the speech)

WHY IDENTICAL PHRASES MATTER

Same phrases give both groups the same words and the same language-model prior
The remaining gap points to how the words were pronounced and heard: the acoustic model and its training audio

The authors: the gap arises “primarily from a performance gap in the acoustic models”.

Koenecke et al. (2020), PNAS 117(14), 7684–7689 (quoted).

Diagnosis, in this course's vocabulary

Where a WER gap can come from

LEXICON

Missing words or pronunciations

Names, loanwords, particles absent from the vocabulary
A word listed with one pronunciation only (the Singapore English /ɛ/ and /æ/ merger)
Matched phrases rule out missing words, not missing pronunciations

LANGUAGE MODEL

Improbable word sequences

A language model that gives one group's grammar low probability
Controlled by matching content
Still a possible cause for code-switching and particles in Singapore

ACOUSTIC MODEL

Voices missing from the training audio

Vowel spaces, prosody and voice quality the training audio never contained
The matched-phrase evidence points here
Only data changes it

Week 1's sentence, now explained: the system learned someone's phonology; an audit shows whose
End-to-end model: all three sit inside one network trained on one dataset
So the remaining option is better training data

The Whisper-era question

Does 680,000 hours of training audio close the gap?

Averages improve with scale. Per language, “WER halves for every 16× increase in training data” (Radford et al. 2023)
Gaps remain in current systems: J&M Fig. 16.1 lists 16.2% WER on CORAAL interviews against 5.8% to 11% on other conversational English
The same audit applies to L2 accents, children's and older voices, and atypical speech. For any “supported language”, ask: which test set, which speakers, what WER?

SUMMARY

Usable is not equitable.

A model can be good enough for most users on average
It can still have its highest WER on the same groups
Only a per-group table shows the good average and the group gap together

The same audit in Singapore

What is the Singapore version of this study?

The Singapore axis is language use, not race in the American sense: particles (lah, leh), Singlish grammar, switching languages within a sentence
NSC metadata (information stored with each recording: speaker gender and ethnic group, recording channel, corpus part) makes per-group tables possible
Read speech vs conversation is itself an axis: Part 3 (conversation) is usually the hardest for recognizers
In the practical you build this kind of table on class data; in A1, on your own test set

AUDIT ETHICS

A high-WER cell describes the model and its training data, not the speakers in it
Metadata categories are administrative, not phonetic: ethnicity is not an accent
A cell's WER does not describe any one speaker in it
Report which model, which data, which group

m

10:40 · Break

Ten minutes.
Then: whose speech is in the data.

After the break: the National Speech Corpus, then forced alignment, which uses the classical HMM, lexicon and Viterbi as a measuring tool.

N

The corpus

The National Speech Corpus

A corpus is audio + transcripts (whose conventions?) + metadata (who is speaking) + licence (who may use it, for what)
Commissioned by IMDA (Infocomm Media Development Authority), built with NTU linguists; before it, the largest Singapore English speech corpus had about 125 hours
Parts 1 and 2 oversample Malay and Indian speakers: about 60% Chinese, 19% Malay, 19% Indian (citizen population: 74%, 13%, 9%)
Singapore Open Data Licence v1.0: research and commercial use, on request; name IMDA and the NSC when you use it

Speaker metadata

What the metadata makes possible

Audits

Per-group WER tables become possible: by gender, ethnic group or corpus part
Without these columns, no group gap can be measured

Phonetics at scale

In Week 3 you measured F1 and F2 of eight vowels by hand
With metadata plus forced alignment, the same measurement runs over thousands of speakers
Vowel spaces, particles and prosody by group

WHAT IT CANNOT TELL YOU

Ethnic group is not language dominance
“Part 3” is not one register
Metadata labels groups (for example female, Malay), not individual people
Speakers within one group can differ as much as the groups do

Age band and education were collected (Koh et al. 2019); check your copy's speaker file before planning an age contrast.

Reading a corpus critically

Four limits of most corpora

Read speech is not conversation

Read sentences (NSC Parts 1 and 2) give clean transcripts
But they lose reduction, overlap and natural particle use
Conversation (NSC Part 3) is the harder test, and the harder to transcribe

Prompted local words

Part 2's prompts include local names and food on purpose: Tampines (a town in eastern Singapore), kopi peng (iced coffee with milk)
Good for coverage, but not evidence of how often anyone says them

Who is missing

Every recruitment method misses some people: ages, occupations, residents who fit no census box
Groups missing from the corpus cannot appear in the audit

Consent has a scope

NSC speakers were recruited and agreed to a stated use under a licence
Whisper's 680,000 hours are audio found on the web, with no record of who the speakers are or what they agreed to
Week 12 returns to consent

Check these four points for any corpus before you use it.

★ The hardest idea this week

Forced alignment is recognition
with the transcript given.

The words are fixed. Viterbi decides only when each phone starts and ends.

V

The classical recognizer, used for measurement

Forced alignment: Viterbi, with the words known

Recognition asked: which words? Alignment already knows the words and asks only: when?
Week 6's machinery without the language model: lexicon → phones → one chain of phone HMMs → Viterbi places the boundaries
Two inputs besides the audio: a pronunciation dictionary and an acoustic model
Output: a TextGrid (Praat's annotation file: labelled time intervals on tiers), with a word tier and a phone tier
You have made one already: the boundary between /n/ and /o/ in Week 6's backtraced state path

WHY IT IS STILL USED

End-to-end models replaced the pipeline for transcription
CTC models can align too, but each label peaks on one or two frames between blanks, so phone boundaries are less precise
Measurement needs boundaries from a documented, repeatable method: HMM aligners such as MFA remain standard in phonetic research

Worked example (illustrative): “kopi”, frames x₁ to x₂₀

Viterbi places the phone boundaries

The path may only stay in a phone or move to the next: the word is fixed, so the only freedom is when each phone ends
Viterbi fills the whole table first; the backtrace from x₂₀ gives this path, drawn left to right

Worked example (illustrative): “kopi”, 4 phones, 8 frames

The table behind the path, with numbers

V(s, t) = P(x_t | s) × max[ V(s, t−1), V(s−1, t−1) ]

s: a phone, in the word's order; the path can only stay in s or move on from s−1
Start in /k/ at x₁, end in /i/ at x₈; stay and move are equally likely here, so they drop out

f

The tool

Montreal Forced Aligner, from the command line

montreal-forced-aligner.readthedocs.io

# 1. download the acoustic model: how each phone sounds mfa model download acoustic english_mfa # 2. download the pronunciation dictionary: how each word is pronounced mfa model download dictionary english_mfa # 3. align, in order: corpus in, dictionary, acoustic model, TextGrids out mfa align corpus/ english_mfa english_mfa aligned/

In: one .wav and one .lab (its transcript) per utterance, same file name
Plus: a pronunciation dictionary and an acoustic model, both downloaded by the commands above
Out: one TextGrid per utterance, with a word tier and a phone tier
Pin the MFA version you tested: commands change between versions

LEARN IT STEP BY STEP

A Gentle Guide to Montreal Forced Aligner (Xu 2024): set up, prepare TextGrids, fix out-of-vocabulary words, align
Chodroff (2018), Corpus Phonetics Tutorial: in this week's readings

The tool, read critically

What MFA needs, and what to distrust

WHAT IT NEEDS

Every word in the dictionary: mfa validate lists the missing ones (out-of-vocabulary, OOV)
An OOV word like “makan” becomes one block of noise (spn), with no phones inside
Fix: add the word to the dictionary yourself
Pronunciations must fit the speaker: if a speaker merges /ɛ/ and /æ/, so must the dictionary

WHAT TO DISTRUST

Alignments are hypotheses
They fail on: missing words, the wrong variety's dictionary, untranscribed hesitations or laughter, code-switches, heavy reduction, overlap
Spot-check in Praat before trusting a measurement

The practical gives you a TextGrid made with MFA; running MFA yourself is optional.

ɔ

Uses of forced alignment

What alignment makes possible

SCALE

From eight vowels to thousands of speakers

Week 3's hand measurement, run across thousands of NSC speakers
Formants, durations and VOT by group, without annotating by hand

DOCUMENTATION

Fieldwork recordings

Align fieldwork recordings to rough transcripts, then search and measure them
Including languages without commercial ASR

THIS COURSE

Your own pipeline

NSC + MFA + Praat scripts: a complete research stack for a thesis on Singapore English
All three are free to use (the NSC after an access request to IMDA)

Alignment errors pass silently into every measurement
Report the aligner, its models and your spot-check procedure, like any other method
Synthesis needs a duration for every phone: Week 10's front end predicts them
In Week 11, duration models learn them from forced alignments

P

11:40 · Break, then the practical

Run the audit.
Then listen to the errors.

Open the Week 9 notebook (link on NTULearn)
No GPU today: the transcripts are pre-run
In the next hour: an audit table with counts, an error log for the worst group, and one vowel measured from a TextGrid

The practical

Your audit table, the worst group, and a TextGrid

Normalize first: finish the normalizer (one line); the lecture pair falls from 100% to 0%
The class file: real Common Voice clips, 19 from Singapore English and 19 from US English speakers; transcripts from both Week 8 models, pre-run
Audit table (A1 step 5): WER for both models, overall and by accent, with counts; one chart
Error log (A1 step 6): listen to three clips from the higher-WER group; one category from the brief per error
TextGrid: one Singapore English clip aligned by MFA; check one boundary on the spectrogram, measure one vowel

Before comparing two models, read their model cards: a model trained on your test corpus scores too well on it
Take-home (optional): run the same audit on your own A1 results file

You leave with this, made by you

illustrative layout: overall WER per system, then the same WER split by condition

The practical hour

The practical hour, stage by stage

1. The normalizer TODO; check the lecture pair (10 min)

2. Audit table: overall WER, then by your contrast with counts; the chart (15 min)

3. Error log: listen to three clips from the higher-WER group, one category per error (20 min)

4. TextGrid: check one boundary, measure one vowel (10 min)

Stages 1 to 4 are the in-class core; the take-home is optional.

YOU LEAVE WITH

A normalizer you completed
An audit table with counts and a chart
A three-clip error log
One vowel measured from machine-placed boundaries

A1 steps 5 and 6, rehearsed

No results file yet? The class file runs every stage.

In the textbook's words

This week, formally

DEFINITION

Word error rate

From the alignment of the two strings, “the minimum number of word substitutions, word insertions, and word deletions”; the rate “can be greater than 100%”. Course gloss: WER = 100 × (S + D + I) ÷ N, with N the words in the reference.

Jurafsky & Martin, SLP 3rd ed., §16.6 (quoted; Aug 2026 draft).

DEFINITION

Text normalization (for scoring)

“It’s normal for systems to normalize text before computing word error rate.” Course gloss: one fixed clean-up applied to reference and hypothesis (lowercase, punctuation, contractions, numerals); it is part of the metric and is reported with it.

J&M §16.6 (quoted). Whisper's normalizer: Radford et al. (2023), appendix on text standardization.

DEFINITION

Disaggregated evaluation

Course wording: the same metric reported separately for each named group in the test set, with utterance and word counts per group. The standard scoring tool already allows it: “sclite also gives error rates by speaker”.

J&M §16.6 (quoted). Exemplar: Koenecke et al. (2020), PNAS 117(14), 7684–7689.

DEFINITION

Forced alignment

Course wording: given audio and its known transcript, the assignment of time boundaries to the words and phones, by running the recognizer's Viterbi search over the fixed transcript.

Tool: McAuliffe et al. (2017), Proc. Interspeech 2017, 498–502 (Montreal Forced Aligner).

Exercises: pencils out · ~7 minutes, in pairs

Try it: score, split, align

EX 9.1

REF: “Can lah, we’re going at 2 pm.” HYP: “can lah we are going at two pm”. Compute the raw WER, then the WER after the lecture's normalizer.

EX 9.2

System A: 8% on group M (majority accent), 25% on group C (code-switched). System B: 13% and 15%. A share p of the reference words is from group M. Write both overall WERs in terms of p; at which p does the ranking flip?

EX 9.3

“makan” is missing from the aligner's dictionary. What does MFA write for it, what can you no longer measure, and what one-line fix restores it?

EX 9.4

Using NSC metadata (gender, ethnic group, recording channel, part), name two audits it makes possible and two questions it cannot answer, and why.

Worked answers in the appendix at the end of this deck.

Before you go

Four things to remember

1. A WER needs its normalizer, its reference conventions and its test population stated beside it

2. An overall WER is a weighted average over groups; report WER for each group as well, with counts

3. In Koenecke et al. (2020) the gap stayed on matched phrases: it points to the acoustic model and its training audio

4. Forced alignment is Viterbi with the words known: a start and end time, so a duration, for every phone

Radar

Assignment 1 is due next Friday, 23 October, 23:59
Assignment 2 is released in Week 10
Quiz 2 (start of Week 12) covers Weeks 6 to 11: today is week four of six

This week

This week's readings

REQKoenecke et al. (2020), “Racial disparities in automated speech recognition”, PNAS 117(14), 7684–7689

Short and open access; read the methods as a model of an audit. pnas.org/doi/10.1073/pnas.1915768117

REQChodroff (2018), “Corpus Phonetics Tutorial”, arXiv:1811.05553

The Montreal Forced Aligner sections; updated web version at eleanorchodroff.com/tutorial. arxiv.org/abs/1811.05553

REQHugging Face Audio Course, Unit 5

WER, CER and text normalization. huggingface.co/learn/audio-course/chapter5/introduction

OPTMcAuliffe et al. (2017), “Montreal Forced Aligner”, Proc. Interspeech 2017, 498–502

Readable now that you know HMMs. isca-archive.org/interspeech_2017/mcauliffe17_interspeech.html

OPTMarkl (2022), “Language variation and algorithmic bias”, FAccT ’22, 521–534

Bias beyond US English, written by a sociolinguist. dl.acm.org/doi/10.1145/3531146.3533117

Next week

Readings for Week 10

REQJurafsky & Martin, SLP (3rd ed., free online): Ch 17, “Text-to-Speech”

Chapter numbers follow the Aug 2026 draft. web.stanford.edu/~jurafsky/slp3/17.pdf

REQKing, S., Speech Synthesis course materials, speech.zone (University of Edinburgh)

The text-processing front end and the unit-selection videos. speech.zone/courses/speech-synthesis

REQPraat manual, “KlattGrid” pages (Weenink)

Skim them so the interface is familiar before the practical. fon.hum.uva.nl/praat/manual/KlattGrid.html

OPTKlatt (1987), “Review of text-to-speech conversion for English”, JASA 82(3), 737–793

By the synthesizer's creator.

OPTTaylor (2009), Text-to-Speech Synthesis, Cambridge University Press

The classical text-to-speech textbook (library copy).

9

One number is not enough.
Split it by group.

Score it, split it, listen to the worst group
Week 10, “Making It Talk”: text normalization, grapheme-to-phoneme conversion, and three classical ways to build a voice
Assignment 1 is due Fri 23 Oct, 23:59

Sources

References

Jurafsky, D. & Martin, J. H. Speech and Language Processing, 3rd ed. (Aug 2026 draft). Ch. 16, §16.1 (Fig. 16.1, error rates by task) and §16.6 (word error rate; text normalization before evaluation). Quoted on the formal slide.

Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z., Toups, C., Rickford, J. R., Jurafsky, D. & Goel, S. (2020). “Racial disparities in automated speech recognition.” PNAS 117(14), 7684–7689. The audit in Lecture I.

Koh, J. X., Mislan, A., Khoo, K., Ang, B., Ang, W., Ng, C. & Tan, Y.-Y. (2019). “Building the Singapore English National Speech Corpus.” Proc. Interspeech 2019, 321–325. The corpus.

IMDA. National Speech Corpus. Singapore Open Data Licence v1.0. The data.

Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C. & Sutskever, I. (2023). “Robust speech recognition via large-scale weak supervision.” Proc. ICML 2023, PMLR 202, 28492–28518. Whisper; Fig. 3 and the text normalizer.

McAuliffe, M., Socolof, M., Mihuc, S., Wagner, M. & Sonderegger, M. (2017). “Montreal Forced Aligner: trainable text-speech alignment using Kaldi.” Proc. Interspeech 2017, 498–502. The aligner.

Chodroff, E. (2018). Corpus Phonetics Tutorial. arXiv:1811.05553; online edition eleanorchodroff.com/tutorial. MFA commands.

Markl, N. (2022). “Language variation and algorithmic bias: understanding algorithmic bias in British English automatic speech recognition.” Proc. FAccT ’22, 521–534.

Hugging Face. Audio Course, Unit 5: evaluation metrics for ASR. huggingface.co/learn/audio-course. WER, CER and normalization in code.

Montreal Forced Aligner documentation (v3.x). montreal-forced-aligner.readthedocs.io. Commands and out-of-vocabulary handling.

Appendix: worked answers

Answers

EX 9.1

Raw tokens: 5 S (Can≠can; lah,≠lah; we’re≠we; 2≠two; pm.≠pm) + 1 I (+are) over 7 reference words = 6 ÷ 7 ≈ 86%; going, at ✓
After normalization (lowercase, strip punctuation, we’re → we are, 2 → two) the strings are identical: 0%
Same audio, same transcript, two numbers: the normalizer is part of the metric

EX 9.2

A(p) = 8p + 25(1 − p) and B(p) = 13p + 15(1 − p), with p = group M's share and 1 − p = group C's share of the reference words. Equal when 25 − 17p = 15 − 2p: 15p = 10, p = 2/3. Above two thirds group M, A has the lower average; below, B does. The models never changed: only the mix did.

EX 9.3

mfa validate flags “makan” as out-of-vocabulary; MFA aligns it as spoken noise (spn): one interval with no boundaries between its phones, so its vowels cannot be measured, and its edges with the neighbouring words are less reliable. Fix: add one tab-separated dictionary line (makan, then its phones in the acoustic model's phone set) and re-align. A fixed lexicon again, but here the fix is one line you add yourself.

EX 9.4

Possible, for example: WER by gender within Part 3; base-vs-fine-tune gains by ethnic group on a part the fine-tune was not trained on (it used Part 2). Not answerable, for example: whether errors track how many Singlish features (particles, Singlish grammar) an utterance contains (no such column: register is not recorded per utterance), or speakers' language dominance (ethnic group is an administrative category, not a linguistic profile). Metadata names cells; it does not describe repertoires.