HG4052 · Speech Synthesis & Recognition
σ
HG4052 · Speech Synthesis & Recognition / Week 7

A Matrix and
a Squash.

Neurons and Gradients.

Chenzi Xu · chenzi.xu@ntu.edu.sg

→ / space to advance · F for fullscreen · M for the slide menu
Q
Quiz 1 debrief

Four points worth a second look

1. The cepstrum as a second transform: what gets transformed, and why the axis is a time

2. Likelihood versus prior in Bayes' rule: which factor is which, and which model supplies each

3. Reading the DTW table: where a cell's value comes from, and what the backtrace means

4. Source and filter: which parts of the spectrum come from the vocal folds (the harmonics) and which from the vocal tract (the formants)

Δ
Today

Roadmap

Also today: Assignment 1 is released. In the practical you measure slopes, run gradient descent, and train a vowel classifier.

t
The shape of the morning

Today, by the clock

09:40  Quiz 1 debrief, then Assignment 1: the brief and key dates (15 min)
09:55  Lecture I: the neuron, layers, and what they detect
10:40  Break (10 min)
10:50  Lecture II: the loss, slopes, descent, and a classifier that learns
11:40  Break + Colab setup (10 min)
11:50  Practical: slopes, gradient descent, a vowel classifier
12:50  Wrap-up
Before new material · 3 minutes, tell your neighbour

Five echoes from Week 6

1. The fundamental equation of classical ASR?

Ŵ = argmaxW P(X | W) · P(W): the acoustic model's likelihood times the language model's prior

2. In the HMM, what did the Gaussian bells do?

emission scores: how likely a frame is under each state; the bell shape was chosen by hand, its centre and width estimated from data

3. What does one Viterbi cell hold?

the probability of the best path that ends in that state at that frame

4. How were the HMM's probabilities trained?

Baum-Welch: start from guesses, work out how probably each frame belongs to each state, re-estimate, repeat

5. What was the classical stack's main weakness?

each frame is scored on its own, so the order of frames is invisible; deltas and triphones were the patches

A
Released today · 30% of the course

Assignment 1: automatic speech recognition

  • Individual and hands-on, on speech recognition
  • Due Friday 23 October, the end of Week 10
  • Everything you need is in the brief on NTULearn: task, data, deliverables, marking
  • This week's only job: read the brief and start the data-access request; the corpus takes time to arrive
KEY DATES

Released: today, 25 Sep
Data request: this week
Due: Fri 23 Oct (Week 10)
Weight: 30%

Recess is next week: a quiet stretch to read the brief properly.

From Week 4

The mel filterbank line is half a layer

Week 4's two lines: a matrix times a vector, then the log
TODAY IN ONE LINE

Take the weighted sum. Add a squash. Let the data choose the weights.

mel_frame = melfb @ power_spec
log_mel = librosa.power_to_db(mel_frame)

  • The first line is a matrix times a vector: many weighted sums at once, one per triangle
  • The second line is the squash: one curve, here the log, applied to each number
  • The filterbank's weights were fixed by us: triangles on the mel scale, chosen from hearing research
  • A neural layer is the same two lines with the weights left free, to be learned from data
y
The unit, drawn

One neuron: a weighted sum and a squash

y = σ( w · x + b )

  • x is the input (say a vowel's F1 and F2), w the weights: the dot product w · x scores similarity to a learned pattern, Week 5's template idea with the pattern learned
  • The bias b shifts the threshold; the squash σ turns the score into a value between 0 and 1
  • Read the output as graded evidence: how /i/-like is this vowel?
Why the squash is not optional

Without a nonlinearity, extra layers add nothing

two squashes in common use
  • Two linear maps in a row collapse into one linear map; a stack of matrices is still one matrix
  • The nonlinearity (the squash) between layers is what lets depth build new features
  • Sigmoid squashes any z into 0 to 1: σ(0) = 0.5, σ(1) ≈ 0.73, σ(−1) ≈ 0.27 (e ≈ 2.718). ReLU keeps positive values and sets negative ones to zero, and trains fast
Layers

A layer is a learned matrix; a stack is a hierarchy

  • A layer runs many neurons on the same input: one matrix, one squash
  • Stacked layers detect patterns of patterns: spectral peaks first, vowel categories later
  • The top layer is a softmax: scores become a probability over classes, summing to one

Each layer's output is a new set of coordinates for the input, an embedding: nearby points are similar. The F1 × F2 vowel chart is an embedding phoneticians built by hand; a network learns its own, with more dimensions.

?
Commit before you see it · hands up

Train a small net on vowels. What do its hidden units become?

A  one tidy unit per phoneme, like labelled buttons

B  useful acoustic pattern detectors, formant-like but unlabelled

C  uninterpretable noise that happens to work

Mostly B, with some C: studies that inspect trained networks find early layers acting like filterbanks and later ones grouping sounds much as phonetic features do, although nobody built that in (Sainath et al. 2013; Nagamine et al. 2015).

m
10:40 · Break

Ten minutes.
Then the one new piece of maths this term.

So far the network is frozen: weights fixed, nothing learned. After the break: how to score its mistakes (the loss), then how the weights are learned.

First: what learning should reduce

The loss: wrongness as one number

  • Run the net on labelled data; collect its probability for each correct answer
  • The loss is the negative log of those probabilities, averaged: the cross-entropy loss. Confident and right costs little; confident and wrong costs a lot
  • Training means changing every weight to make the loss as small as possible
  • Logs again: they keep products of small probabilities from underflowing to zero
the penalty curve: loss = −log P(correct)
P(correct answer)1.00.90.50.10.01
loss = −log P00.110.692.304.61

natural log; the dots on the curve mark P = 0.9, 0.5 and 0.1

Where the name comes from

Why it is called cross-entropy

entropyH(p)    = − Σ p(x) · log p(x)
cross-entropyH(p, q) = − Σ p(x) · log q(x)

p(x): how often outcome x really occurs (the data)

q(x): the probability the model gives to x

−log: surprisal: small for expected outcomes, large for surprising ones

− Σ p(x) · …: add up over all outcomes, each weighted by how often it occurs: an average

  • Entropy: the average surprise when the true odds do the scoring, the lowest possible
  • Cross-entropy: the world follows p, the model's q does the scoring; it equals the entropy only when q = p
One token of /i/
vowel/i//ɑ//u/
p (label)100
q (softmax)0.710.190.10

H(p, q) = −(1 · log 0.71 + 0 · log 0.19 + 0 · log 0.10) = −log 0.71 = 0.34

The zeros remove every term except the correct vowel: cross-entropy = −log P(correct answer), the loss.

★ The hardest idea this week

Learning is one question:
which way is downhill?

Downhill means: the direction that lowers the loss.
Measure the slope, take one step against it, repeat.

d
The new idea, gently

The derivative is a slope at a point

  • Zoom in on any smooth curve and it straightens; the slope of that line is the derivative there
  • It answers one question: nudge the input a little, how much does the output move?
  • At a valley floor the slope is zero: nothing to gain in either direction
No formalism required · type it with me

Measure a slope by nudging

The function: f(x) = x². In Python, ** means “to the power of”: 3.01**2 is 3.01².

(3.01**2 - 3**2) / 0.01

6.009999…  the slope of x² at x = 3 is 6

(3.0001**2 - 3**2) / 0.0001

6.0000999…  smaller nudge, cleaner slope

Calculus gives the exact answer without nudging: for x² the slope at x is 2x. We will take such recipes as given; the point is what a slope means, and the nudge test always works.

THE STEP THAT LEARNS

x ← x − η · slope

Move against the slope: downhill. The small number η (eta) is the learning rate: the size of each step.

One weight, one loss curve

Gradient descent, live

  • The curve: loss(w) = ½ (w − 2)² + 0.4, lowest at w = 2; its slope at w is w − 2
  • The ball repeats one move: w ← w − η · slope
  • Small η crawls; a good η settles quickly; too large a step overshoots the valley
  • At η = 2.3 each step overshoots further than the last: the run diverges
  • A network does this in a million directions at once; the vector of slopes is called the gradient
Backpropagation, briefly

Backpropagation, in three sentences

The chain rule on one weight: three functions in a row, three slopes multiplied

Illustrative values: input x = 2 and weight w = 0.5 give z = 1, so y = σ(1) ≈ 0.73 (bias left out). σ: the sigmoid, 1 / (1 + e−z).

  • A network is functions inside functions; the chain rule says slopes of nested functions multiply
  • Backpropagation is the bookkeeping that computes every weight's slope in one backward sweep, reusing shared work
  • You met that trick twice already: dynamic programming

For the curious: two extra slides on the chain rule, with worked examples, in the appendix at the end of the deck.

IN PRACTICE

loss.backward()

One line: the library differentiates the whole network for you. Understanding the slope is your job; computing a million of them is the machine's.

Real vowels, real speakers

The data: Hillenbrand et al. (1995)

  • 1,668 vowel tokens in /hVd/ words (heed, hid, … who'd): 12 vowels from 139 Michigan speakers, 45 men, 48 women and 46 children
  • Inputs: each token's F1 and F2, so the space is the vowel chart; the figure shows three of the twelve vowels
  • Targets: the vowel each speaker was asked to say
  • The machine: 2 inputs, a small hidden layer, a softmax over the vowels

Week 5's recognizer stored one speaker's token. This one sees many speakers: it learns each vowel's distribution, which the template lacked.

/i/, /ɑ/, /u/ from 70 speakers
An illustrative run on the same vowels

A training run, epoch by epoch

  • Dots are the vowel tokens; they stay put. One epoch = one pass through every token
  • The three moving rings are the model's current guess of where each vowel sits; every point goes to its nearest ring, which draws the tinted decision regions
  • Early epochs are wrong everywhere; training moves the rings onto the clusters, and the loss curve falls
  • The model was never told where /i/ is: the boundary is what it learned

A simplified model for the picture: one centre per vowel, so the boundaries are straight. The practical's network draws its own, curved ones.

Week 2's held-out rule, again

Overfitting: memorising is not learning

two losses during one training run
  • Training loss keeps falling: the network can always memorise its training speakers a little more
  • Held-out loss falls, then turns upward: memorisation has replaced generalisation
  • The standard remedy: stop training at the turn (early stopping)
Held-out data, split in two

Training, development and test sets

for example, the 139 Hillenbrand speakers, each speaker in one set only
training84 speakers
development20 speakers
test35 speakers
TRAINING

Learns the weights

Gradient descent uses it at every step

DEVELOPMENT (DEV)

Makes the choices

When to stop, how many hidden units, which learning rate. Checked many times

TEST

Gives the final score

How well the model does on new speakers. Used once, at the end, never for choices

Jurafsky & Martin, §3.2: evaluating any machine learning model needs “at least three distinct datasets”: training, development and test.

Early stopping: the rule

When to stop training

the rule on an illustrative dev-loss curve, with N = 5
  • 1After every epoch, compute the loss on the development (dev) set
  • 2When it is the lowest so far, save the weights
  • 3After N epochs with no improvement, stop. N is the patience (scikit-learn's default: 10)
  • 4Go back to the saved weights

Why wait N epochs: the loss goes up and down a little, so one rise is not yet the turn (epochs 6, 9 and 12 in the figure).

No fixed number of epochs: the turn depends on the network's size, the amount of data and the learning rate.

Where this plugs in

Neural networks inside the classical recognizer

audiowaveform in
→
log mel framesWeek 4's front end
→
neural netreplaces the Gaussian bells
→
state scoresper frame, per state
→
Viterbi + n-gramWeek 6, unchanged
→
wordsthe argmax

This is the hybrid system on last week's timeline: the HMM structure stays and the network replaces only the emission scores. It reads a window of neighbouring frames at once, which eases last week's frame-independence weakness; error rates fell sharply (Hinton et al. 2012).

At scale: today's example has two inputs and three outputs; production networks take dozens of log-mel channels over many frames and have millions of weights. The same parts: matrix, squash, loss, slope.

After the recess

Next: one network replaces the whole pipeline

  • Week 8: end-to-end ASR: one network from audio to text, no lexicon, no separate LM, Whisper included
  • Its central mechanism is attention: for each output, the network learns which frames to use
  • Homework over recess: watch the 3Blue1Brown attention video before Week 8 (required)
  • Read the Assignment 1 brief; post questions on the forum
RECESS · 28 SEP – 4 OCT

No class on Friday 2 October. Before Week 8 (Friday 9 October): the attention video and the Assignment 1 brief.

η
11:40 · Break, then the practical

Nudge. Descend.
Then train a vowel classifier.

Open the Week 7 notebook (link on NTULearn). In the next hour you measure slopes, run gradient descent, and train a vowel classifier yourself.

The practical

Slopes, descent, and a vowel classifier

  • Slopes and steps: measure the slope of x² by nudging; your TODO is the update line x = x - eta * slope; run three step sizes
  • The vowel chart: load the Hillenbrand data and plot F1 against F2, axes flipped as phoneticians draw them
  • One neuron: a logistic regression for /i/ vs /ɑ/; draw its boundary on the chart and read its two weights
  • A network: sklearn's MLPClassifier on all 12 vowels; watch the loss curve fall, then try a learning rate 100 times larger
  • Held-out speakers: train on some speakers, test on the rest; which vowels does the confusion matrix mix up?

Stretch (pick one, take-home): overfit on purpose (a large network on 10 tokens per vowel) · sweep the hidden-layer size from 1 to 100 · add your own three vowels measured in Praat: does a model trained on Michigan English classify your accent?

You leave with this, made by you

decision regions over the vowel chart, tested on speakers the network never saw

The practical hour

The practical hour, stage by stage

1.  Slopes by nudging; the update line; three step sizes (15 min)
2.  The vowel chart from the Hillenbrand data (5 min)
3.  One neuron for /i/ vs /ɑ/: its boundary and weights (10 min)
4.  MLPClassifier on 12 vowels; the loss curve; a 100× learning rate (15 min)
5.  Held-out speakers and the confusion matrix (15 min)

The stretch (pick one) is take-home; the five stages above are the in-class core.

YOU LEAVE WITH

A slope you measured by nudging, a gradient-descent run you broke on purpose, and a vowel classifier scored on speakers it never saw.

If the update line is unfinished, a fallback keeps the descent running.

In the textbook's words

This week, formally

DEFINITION
Unit

“The building block of a neural network is a single computational unit. A unit takes a set of real valued numbers as input, performs some computation on them, and produces an output.” Here: a weighted sum plus a bias, then a squash.

Jurafsky & Martin, SLP 3rd ed., §6.1 (quoted; Aug 2026 draft); last sentence: course gloss.
DEFINITION
Feedforward network

“A feedforward network is a multilayer network in which the units are connected with no cycles; the outputs from units in each layer are passed to units in the next higher layer, and no outputs are passed back to lower layers.”

Jurafsky & Martin, SLP 3rd ed., §6.3 (quoted; Aug 2026 draft).
DEFINITION
Derivative

The slope of a function at a point: the rate at which the output changes for a small nudge of the input. At a local minimum the derivative is zero.

Standard definition, course wording; see 3Blue1Brown, Essence of Calculus, chs. 1–2.
DEFINITION
Gradient descent

“[G]radient descent requires knowing the gradient of the loss function, the vector that contains the partial derivative of the loss function with respect to each of the parameters.” Each step moves every parameter against its slope, scaled by the learning rate.

Jurafsky & Martin, SLP 3rd ed., §6.6 (quoted; Aug 2026 draft); last sentence: course gloss.
Exercises: pencils out · ~7 minutes, in pairs

Try it: neurons and slopes

EX 7.1

A neuron has weights w = [2, −1], bias b = −4, and input x = [3, 1]. Compute w · x + b, then the output σ(w · x + b), given that σ(1) ≈ 0.73.

EX 7.2

Measure the slope of f(x) = x³ at x = 2 by nudging: compute (2.01³ − 2³) / 0.01. The calculus recipe says the slope is 3x². Do the two answers agree?

EX 7.3

For f(x) = x², the slope at x is 2x. Starting at x = 3, take one gradient-descent step with η = 0.1, and another (from x = 3 again) with η = 1.1. Which step helps, which harms, and how can you tell?

EX 7.4

Your vowel net scores 98% on its training speakers and 71% on a new speaker. Name the phenomenon, the data that revealed it, and one remedy.

Worked answers in the appendix at the end of this deck.

Before you go

Four things to remember

1. A neuron is a weighted sum, a bias, and a squash: Week 5's template made soft and learnable

2. The derivative is a slope; the nudge test measures it with arithmetic alone

3. Learning is sliding downhill on the loss; the learning rate decides whether it crawls, settles, or diverges

4. Held-out speakers are the test: a network that memorises its training speakers has not learned the vowels

Radar: recess next week, no class on 2 October. Over the break: the attention video, the Assignment 1 brief, and the data-access request. Quiz 2 (Week 12) covers Weeks 6 to 11, so today's neuron is on it.

This week

This week's readings

REQJurafsky & Martin, SLP (3rd ed., free online): Ch 6, “Neural Networks”

§6.1 (units), §6.3 (feedforward networks) and §6.6 (training); skim the backpropagation maths. Chapter numbers follow the Aug 2026 draft. web.stanford.edu/~jurafsky/slp3/6.pdf

REQ3Blue1Brown, “But what is a neural network?” (19 min) and “Gradient descent, how neural networks learn” (21 min)

Layers and weights, then this week's hardest idea; watching the second video twice is a good use of time. youtube.com/watch?v=aircAruvnKk · youtube.com/watch?v=IHZwWFHWa-w

OPTTensorFlow Playground

Watch decision boundaries form in the browser, no code. playground.tensorflow.org

OPT3Blue1Brown, Essence of Calculus, chapters 1–2

The derivative from first pictures, if today's slope needs a slower treatment. 3blue1brown.com/topics/calculus

After the recess

Readings for Week 8

REQ3Blue1Brown, “Attention in transformers, step-by-step” (26 min)

Required before Week 8; watch it over the recess. 3blue1brown.com/lessons/attention

REQHannun (2017), “Sequence Modeling with CTC”, Distill

Read through the collapsing rule; the animations are the point. distill.pub/2017/ctc

REQHugging Face Audio Course, Unit 3: “Transformer architectures for audio”

The CTC and Seq2Seq pages. huggingface.co/learn/audio-course/chapter3/introduction

REQJurafsky & Martin, SLP (3rd ed.): Ch 16, §16.3 “The Encoder-Decoder Architecture for ASR” and §16.5 “CTC”

web.stanford.edu/~jurafsky/slp3/16.pdf

OPTRadford et al. (2023), Whisper, and Baevski et al. (2020), wav2vec 2.0

The introduction and approach sections of each. arxiv.org/abs/2212.04356 · arxiv.org/abs/2006.11477

∂

One matrix. One squash.
Then slide downhill.

That is a neural network, and you trained one on real vowels.  Week 8, after the recess, “The End of the Assembly Line”: end-to-end recognition, attention, and Whisper.

Sources

References

Jurafsky, D. & Martin, J. H. Speech and Language Processing, 3rd ed. (Aug 2026 draft). Ch. 6, §6.1 (units), §6.3 (feedforward networks), §6.6 (training, the cross-entropy loss). Definitions quoted on the formal slide.

Rumelhart, D. E., Hinton, G. E. & Williams, R. J. (1986). “Learning representations by back-propagating errors.” Nature 323(6088), 533–536. Backpropagation.

Hillenbrand, J., Getty, L. A., Clark, M. J. & Wheeler, K. (1995). “Acoustic characteristics of American English vowels.” JASA 97(5), 3099–3111. The vowel data.

Barreda, S. (2023). phonTools: Tools for phonetic and acoustic analyses. R package version 0.2-2.2, CRAN. The copy of the Hillenbrand data used here.

Hinton, G., Deng, L., Yu, D., Dahl, G. E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N. & Kingsbury, B. (2012). “Deep neural networks for acoustic modeling in speech recognition.” IEEE Signal Processing Magazine 29(6), 82–97. The hybrid recognizer.

Sainath, T. N., Kingsbury, B., Mohamed, A. & Ramabhadran, B. (2013). “Learning filter banks within a deep neural network framework.” IEEE ASRU 2013, 297–302.

Nagamine, T., Seltzer, M. L. & Mesgarani, N. (2015). “Exploring how deep neural networks form phonemic categories.” Proc. Interspeech 2015, 1912–1916.

Pedregosa, F. et al. (2011). “Scikit-learn: machine learning in Python.” Journal of Machine Learning Research 12, 2825–2830. The practical's classifiers.

Sanderson, G. (3Blue1Brown). Neural networks and Essence of Calculus, video series. 3blue1brown.com

Appendix: worked answers

Answers

EX 7.1

w · x + b = 2·3 + (−1)·1 − 4 = 1. Output σ(1) ≈ 0.73: moderately confident evidence for the pattern this neuron detects.

EX 7.2

(2.01³ − 8) / 0.01 = (8.120601 − 8) / 0.01 = 12.06. The recipe gives 3 × 2² = 12. They agree to the accuracy of the nudge; shrink the nudge and the gap shrinks with it.

EX 7.3

Slope at 3 is 6. With η = 0.1: x becomes 3 − 0.6 = 2.4, closer to the minimum at 0: the step helps. With η = 1.1: x becomes 3 − 6.6 = −3.6, farther from 0 than where it started: the step overshoots and the run diverges. The tell is distance from the minimum (or simply the loss) increasing after the step.

EX 7.4

The phenomenon is overfitting: the net memorised its training speakers. The data that revealed it is held-out data, here a held-out speaker. Remedies: stop training earlier, shrink the network, or train on more speakers; all three trade memorisation for generalisation.

Appendix: extra for the curious · not core

The chain rule: slopes multiply

One nudge, passed along two functions
A curve: y = (3x + 1)² at x = 1

inside u = 3x + 1 → slope 3 outside y = u² → slope 2u = 8 (u = 4 here) whole 8 × 3 = 24

((3*1.001 + 1)**2 - (3*1 + 1)**2) / 0.001

→ 24.009 the nudge test agrees

Leibnizdy/dx = dy/du × du/dx
Lagrangey′    = f′(g(x)) × g′(x)

dy/dx, y′: the slope of y for x, the whole chain

dy/du, f′(g(x)): the outside's slope, at the inside's current value

du/dx, g′(x): the inside's slope

Lagrange writes y = f(g(x)): g is the inside, f the outside, and the prime ′ means “the slope of”. Here g(x) = 3x + 1 and f(u) = u², so y′ = 2(3x + 1) × 3 = 24 at x = 1.

Appendix: extra for the curious · not core

Backpropagation: the chain rule, run backwards

One hidden neuron, one output neuron: values forward on top, slopes of the loss backward underneath

Illustrative values: input x = 2, w1 = 0.5, w2 = 1.5 (biases left out). Local slopes, as on the backpropagation slide: y(1 − y) for the sigmoid σ, −1/y for −log.

When a value feeds several later boxes, the slopes arriving along each path are added.