Probability and Uncertainty.
Chenzi Xu · chenzi.xu@ntu.edu.sg
1. What is a vector?
a list of numbers; a point with a direction
2. What question does the dot product answer?
do two arrows point the same way?
3. In the recognizer pipeline, which two stages turn features into text?
the acoustic model, then the decoder with its language model
4. The course mantra?
one list of numbers into another
Today we build the mathematics inside those last two boxes: the arithmetic of guessing.
When do I use cosine, and when distance?
Cosine compares direction and forgives overall size: a child's /i/ and an adult's /i/ point the same way. Distance cares about the actual Hz gap. Neither is wrong; they answer different questions, which is why /u/ ranked differently under each in the practical.
Why is two samples per cycle the limit?
A 100 Hz wave goes up and down 100 times a second. The samples must come faster than the wiggles: at least one on the way up and one on the way down of every cycle, so more than 200 samples per second. Sample more slowly and whole wiggles fall between the samples; the dots that remain trace out a slower wave instead (Week 3 calls this aliasing). So a recording at rate R holds frequencies only up to R/2: the Nyquist frequency.
In the acoustic phonetics textbook's words: “The Nyquist frequency is always one-half the sampling rate.” Johnson, Acoustic and Auditory Phonetics, 2nd ed. (2003), p. 22. The full sampling theorem is Coleman (2005) §2.5, in the same chapter as Week 1's quantisation ladder.
Do I have to convert binary in my head?
No. Practise a couple of 4-bit conversions once, then own the idea: n bits give 2ⁿ levels, and every extra bit doubles the ladder. The table is a reference, not a memory task.
“recognize speech”
[ˌɹɛkəɡnaɪz ˈspiːtʃ]
“wreck a nice beach”
[ˌɹɛkə ˈnaɪs ˈbiːtʃ]
The acoustics barely differ. Word boundaries are not in the signal.
So a recognizer cannot know; it can only rank its guesses.
Your perceptual system does exactly the same thing, all day, effortlessly. This week is the arithmetic of guessing: recognition as inference under uncertainty.
“Every time I fire a linguist, the performance
of the speech recognizer goes up.”
Attributed to Fred Jelinek (IBM), c. 1988: the folk version. His own telling was milder, and no linguist was actually fired; see the references.
What he meant: hand-written rules were losing to probabilities counted from data. That turn is paradigm 2 on last week's timeline, HTK to Kaldi, and every paradigm since has taken it further.
A probability is a count of some unit in some context. In that era linguists chose both: phones as the units, the previous phone or two as the context.
End-to-end and foundation models count letters, word pieces, even self-invented sound units. Your choices move outward: which speech is in the training data (today's Singlish demo), and whose errors get counted.
Its arithmetic is still today's arithmetic: every network is trained to minimise surprisal and graded by perplexity.
Which phone is the most frequent
in spoken English?
Write your guess down. Hands up: who picked a vowel? A consonant? Which one?
The chart on the next slide settles it.
≈ share of phone tokens, conversational English (approximate)
Frequency is what you counted.
Probability is what you believe about the next one.
(Goldsmith's distinction; it is in your reading)
Schwa wins. The most frequent sound in English is the most reduced one. Hold that thought for the surprisal slide.
Not zero in general: perfectly fine in Cantonese (ŋɔː⁵ 我 ‘I’), in Vietnamese (Nguyễn), and in the Singaporean surname Ng. Phonotactics is a probability distribution conditioned on the language.
P(/ɹ/ | /t/): given a /t/, how often is the next phone /ɹ/? A different question from the joint P(/t/ then /ɹ/): “pick any adjacent pair at random; is it /tɹ/?”
The bridge between them: P(A then B) = P(A) × P(B | A)
Read aloud: “the chance of the pair is the chance of the first, times the chance of the second given the first.”
Toy counts: 200 word-initial pairs, first phone by row, second by column. Phonotactically honest: /tɹ/ thrives, /tl/ and /sɹ/ starve.
Pick any pair at random: is it /tɹ/?
| ɹ | l | w | Σ | |
|---|---|---|---|---|
| t | 40 | 2 | 18 | 60 |
| s | 1 | 24 | 35 | 60 |
| k | 29 | 36 | 15 | 80 |
| Σ | 70 | 62 | 68 | 200 |
P(t then ɹ) = 40 / 200 = .20
Ignore the second phone: starts with /t/?
| ɹ | l | w | Σ | |
|---|---|---|---|---|
| t | 40 | 2 | 18 | 60 |
| s | 1 | 24 | 35 | 60 |
| k | 29 | 36 | 15 | 80 |
| Σ | 70 | 62 | 68 | 200 |
P(t first) = 60 / 200 = .30
Given it starts /t/: is /ɹ/ next?
| ɹ | l | w | Σ | |
|---|---|---|---|---|
| t | 40 | 2 | 18 | 60 |
| s | 1 | 24 | 35 | 60 |
| k | 29 | 36 | 15 | 80 |
| Σ | 70 | 62 | 68 | 200 |
P(ɹ | t) = 40 / 60 ≈ .67
P(t) × P(ɹ|t) = 60/200 × 40/60 = (60 × 40) / (200 × 60) = 40/200 = P(t then ɹ)
P( stɹɛŋθs ) = P(s) · P(t|s) · P(ɹ|st) · P(ɛ|stɹ) · P(ŋ|stɹɛ) · P(θ|stɹɛŋ) · P(s|stɹɛŋθ)
P( stɹɛŋθs ) ≈ P(s) · P(t|s) · P(ɹ|t) · P(ɛ|ɹ) · P(ŋ|ɛ) · P(θ|ŋ) · P(s|θ)
Why truncate? Each extra phone of context multiplies the number of probabilities to estimate, and most longer contexts never occur in a finite corpus at all: data sparsity. Two phones of context give a trigram; more in Week 6.
P( word | sound ) ≠ P( sound | word )
“How would this word sound?” and “which word was that sound?” are different questions.
Bayes' rule is the machine for flipping one into the other.
P(word | sound) ∝ P(sound | word) × P(word)
What we want: belief in each candidate word, after hearing the sound.
How well does each word explain the sound? In ASR this is the acoustic model (Weeks 5–8).
How plausible was the word anyway? In ASR this is the language model (Week 6).
These are exactly the two boxes from last week's pipeline diagram. Bayes' rule is the referee that multiplies their opinions.
P(A|B) = P(A,B) / P(B)
P(B|A) = P(A,B) / P(A)
P(A|B) · P(B) = P(A,B) = P(B|A) · P(A)
P(A|B) = P(B|A) · P(A) / P(B)
Bayes’ rule. No new assumption anywhere: the definition of conditional probability, rearranged.
| ɹ | l | w | Σ | |
|---|---|---|---|---|
| t | 40 | 2 | 18 | 60 |
| s | 1 | 24 | 35 | 60 |
| k | 29 | 36 | 15 | 80 |
| Σ | 70 | 62 | 68 | 200 |
by row: P(t) · P(ɹ|t) = 60/200 × 40/60 = 40/200
by column: P(ɹ) · P(t|ɹ) = 70/200 × 40/70 = 40/200
Two different conditionals reach the same cell. Setting them equal and dividing by one marginal is exactly step 3.
A noise-smeared fricative before [ɪp]:
sip or ship?
No context given. Hands up for sip; hands up for ship.
Hold your vote; the arithmetic is next, and it may change your mind twice.
The frication is ambiguous between [s] and [ʃ], leaning /s/: P(sound|sip) = 0.6 P(sound|ship) = 0.4
Corpus priors, “ship” about 9× more frequent:
P(ship) = 0.90 P(sip) = 0.10
a two-word universe: priors renormalized over the only candidates
ship: 0.4 × 0.90 = .360 sip: 0.6 × 0.10 = .060
Ship wins: the prior steamrolls the acoustics.
The context flips the priors:
P(sip) = 0.95 P(ship) = 0.05
same sound, same arithmetic, new expectations
sip: 0.6 × 0.95 = .570 ship: 0.4 × 0.05 = .020
Sip wins, 97%: the verdict flips back.
The sound never changed; only the prior did. Context decides what you hear.
No context:
P(ship|sound) = .360/(.360+.060) = 86%
“Take a ___ of tea”:
P(sip|sound) = .570/(.570+.020) = 97%
Widths are priors: how much of the world each word owns before you listen.
Heights are likelihoods: how well each word explains this sound.
Shade is their product, the joint. The posterior is a share of shade: your word's patch, out of all the shade there is.
Context redraws the widths. The heights never move. The verdict flips.
This is the picture in this week's 3Blue1Brown reading; the numbers are the café's.
A garbled signal is equally consistent with “their” and “there”: P(signal | their) = P(signal | there) = 0.5. Corpus priors: P(their) = 0.3, P(there) = 0.7. Use Bayes' rule to find P(there | signal). What decided the outcome?
Same recipe as the café: likelihood × prior for each candidate, then compare. Check with your neighbour before the walkthrough; the worked answer is in the appendix.
Predict first, word for word. Then we dictate this into the podium recognizer:
“He kena scolded by his boss lah.”
A US-English language model prior rewriting Singlish acoustics: kena becomes the nearest English word it expects, lah vanishes. The sip/ship arithmetic, happening in the wild.
To hear Singlish you must count Singlish: the IMDA National Speech Corpus is exactly the corpus you would count. Week 9 audits this properly.
Stretch. Refill.
Then: we measure your ears.
When we return, your own perception becomes the data set.
If perception were acoustics only, the two boundaries would land in the same place. If your lexicon is a prior, the boundary shifts: more “g” before -ift (gift is a word), more “k” before -iss (kiss is a word). Your vote curve is the posterior, drawn by a room of linguists.
An ambiguous sound between /ɡ/ and /k/ is heard as /ɡ/ before “-ift” (gift is a word) but /k/ before “-iss” (kiss is a word). Same acoustics, different percept: the prior at work. You have just replicated this.
Replace the /s/ in “legi*lature” with a cough and listeners still hear the /s/, and cannot even locate the cough. Perception fills in what the prior expects.
The noisy-channel model is not a metaphor borrowed by engineers; it is a serious theory of human word recognition (see Norris & McQueen's Shortlist B, if you want the deep end).
Loudness: each +10 dB ≈ ×10 in power. A log axis for ears.
Pitch perception: roughly logarithmic in frequency above ~1 kHz. Returns in Week 4.
Musical intervals: equal steps = equal frequency ratios. Logs again.
Logs do two jobs: they compress huge ranges into honest axes, and they turn multiplication into addition:
log(a × b) = log(a) + log(b)
Doubling x adds 1 to log₂ x: the curve keeps climbing, ever more slowly.
2y = x ⇔ y = log₂ x log₂ 8 = 3
log 1 = 0. Between 0 and 1 the log is negative; it never reaches x = 0. Probabilities live in that strip, so log P ≤ 0.
log(a × b) = log a + log b
log(aⁿ) = n · log a
log₂ counts bits (this course); log₁₀ counts decades (decibels); ln is the calculus one. To get log₂ from a calculator's log₁₀:
y = log₂ x means x = 2y
log₁₀ x = log₁₀(2y) = y · log₁₀ 2 the power rule
y = log₂ x = log₁₀ x / log₁₀ 2 divide by 0.301
A 20-word sentence, each word P ≈ 0.01:
0.01 × 0.01 × … × 0.01 = (10⁻²)²⁰ = 10⁻⁴⁰ → absurdly small, but still alive
A two-minute transcript (200 words):
0.01 × … × 0.01 = 10⁻⁴⁰⁰ = exactly 0.0 → below the float floor, near 10⁻³⁰⁸ (float64)
Week 1's exponent bits, out of room; the next slide closes that loop.
log₂P = (−6.6) + (−6.6) + … = −133 for 20 words, −1329 for 200 → alive at any length
From Week 5 onward, every score in this course (acoustic, lexical, syntactic) is a log probability being added. When you see scores added in a decoder, you are watching probabilities being multiplied.
>>> 0.01 ** 20
1e-40 tiny, but alive
>>> 0.01 ** 200
0.0 a two-minute transcript, gone without an error message
>>> 200 * math.log2(0.01)
-1328.77… the same number, alive and well, in logs
Remember float32: 1 sign + 8 exponent + 23 fraction bits, floor near 10⁻³⁸. Python's float is float64: 11 exponent bits buy a floor near 10⁻³⁰⁸. But 0.01²⁰⁰ = 10⁻⁴⁰⁰ needs an exponent no float can hold, so the hardware rounds it to 0.0 and says nothing. log₂P sidesteps the whole machinery: it keeps only the exponent, as an ordinary mid-sized number; −1329 fits comfortably where 10⁻⁴⁰⁰ cannot.
No crash, no warning, just a wrong answer: the log scale is a survival rule, not a taste.
“the”, out of the blue: P ≈ .05 → 4.3 bits
“the” right after “of”: P ≈ .6 → 0.7 bits
“sesquipedalian”: P ≈ 10⁻⁸ → ≈ 27 bits
A surprise meter: the rarer, the bigger. A certainty scores 0 bits; every halving of the probability adds exactly one bit.
Predictable words and segments are produced shorter, more reduced, more often deleted, as measured across corpora (Aylett & Turk 2004; Bell et al. 2009).
“probably” → [pɹɒbli] exactly when it's expected. Surprisal is in the signal, not just in the model.
MacKay defines the Shannon information content of an outcome x as h(x) = log₂ 1/P(x), measured in bits: the same number as −log₂ P(x). Psycholinguists call it the surprisal of a word in its context (Hale 2001; Levy 2008): the name is literal, it measures how surprised you should be by the outcome that actually happened.
MacKay (2003), Information Theory, Inference, and Learning Algorithms, §2.4 (free online). The name goes back to Tribus (1961).Which phone? Which word? Which particle? The outcomes are separate options, and each gets a probability. Bar heights, and they sum to 1.
P(x) ≥ 0 Σ P(x) = 1 a probability mass function
F1 in Hz, a vowel's duration in ms, pitch. Any value on the line is possible, so no single exact value can carry probability. Instead a density curve: probability is the area under it between two values; the whole area is 1.
f(x) ≥ 0 total area = 1 a probability density function
Everything so far was discrete (phones, words). Acoustic measurements are continuous, and Week 6's recognizer scores them with densities: next slide.
F1 of /i/ tokens, one speaker: histogram + fitted normal density
What's the probability your next /i/ has F1 of exactly 280.000000 Hz? Zero.
For continuous measurements we use a density: a curve whose height says which regions are likely. Fit a bell curve to F1 and you have a model of a vowel.
In Week 6, each HMM state scores incoming acoustic frames with exactly this kind of curve. You now own every ingredient.
I have a sentence. You guess it letter by letter; I only say yes or no. We count the guesses; those counts estimate the language's per-letter information.
Strict phonotactics → fewer real options → lower entropy per segment. After /st/, English allows very little. Structure = predictability.
H(X) = −Σx p(x) log₂ p(x), in bits: Jurafsky & Martin introduce entropy as a measure of information and, intuitively, as “a lower bound on the number of bits it would take to encode a certain decision”. The definition is Shannon’s (1948).
Jurafsky & Martin, SLP 3rd ed., §3.7, eq. (3.32); Shannon (1948).4.70 bits
computed: log₂ 26
≈ 4.1 bits
the formula, fed corpus letter counts; you compute it on a novel in the practical
≈ 1 bit / letter
measured with the guessing game, next slide; his bounds: 0.6 to 1.3 bits
Mid-word, after strong context, one guess suffices: low surprisal. Right after the switch point, guesses pile up: surprisal made audible. (Strictly, guess counts give bounds on entropy, not the number itself; close enough to feel it.)
P(x) → −log₂ → surprisal → average → entropy H → 2H → back to a number of options
The 2H undoes the −log₂. For a single outcome the round trip returns 1/P(x): a one-in-32 event counts as 32 live options; perplexity averages those across outcomes. Check: k equally likely options → H = log₂ k → 2H = k exactly.
Perplexity converts bits back into an intuition: “the model is as confused as if it were choosing among k equally likely options.”
Letter frequencies alone: 24.1 ≈ 17. Predicting with the preceding text (the guessing game): 21 = 2, nearly a coin flip per letter.
In Week 6 we grade language models by exactly this number, on text they have never seen.
We always examine a model
on sentences it never saw.
The text the model counted. Its textbook.
The exam: text it could not have memorized.
Your bigram model's perplexity on a fresh paragraph vs the same paragraph shuffled. Commit now: which is higher, and why?
Acing your own textbook proves memory, not knowledge. From here to Week 13, every number we trust is measured on held-out data.
Open your laptop.
The Week 2 Colab link is on NTULearn.
Same drill as last week: browser tab, no installs. Your notebook already knows pwd and ls.
Stretch (or take-home): trigrams (better babbling); or compare the entropy of English letters vs English phones. Which writing system is the corpus hiding?
your bigram heatmap: hunt for the bright q→u cell and the phonotactic dark zones
def clean_text(raw):
text = raw.lower()
text = remove_punctuation(text)
return text
We read it aloud together, once: the name, the input, two steps, the output. That is the whole grammar of a function.
This is Week 1's input → output machine, in code: clean_text(raw) is spectrogram(waveform)'s cousin. From Week 5 onward you must recognize functions on sight; you will never be asked to write one from scratch.
counts = Counter()
for ch in text:
counts[ch] += 1
Read aloud: for each character in the text (call it ch), add one to that character’s count. The indented body runs once per item; ch is a name you choose, fresh each round.
letters = [ch for ch in text
if ch.isalpha()]
A list comprehension: collect ch, for each ch in text, keeping only the letters. The same loop in one line; the notebook uses both shapes.
As with def: you read these on sight; you are never asked to write one from scratch. Counting the novel is the left loop run 846,301 times, once per character of Emma.
✓ a bigram heatmap with the q→u cell found
✓ five pseudo-words that look eerily English
✓ perplexity higher on the shuffled text, and you can say why
This bigram model returns in Week 6 as the language model inside a real recognizer. The stretch tasks are take-home if the hour runs out.
P(A | B) = P(A, B) / P(B): the probability of A once B is known. Read “the probability of A given B”.
Standard; see Goldsmith (2007), “Probability for linguists”.P(A | B) = P(B | A) · P(A) / P(B). It turns “how a word sounds” (likelihood) and “how expected the word is” (prior) into “which word, given the sound” (posterior).
Jurafsky & Martin, SLP 3rd ed. (noisy-channel framing).Surprisal of an event: I(x) = −log₂ P(x) bits. Entropy is its expected value: H(X) = −Σₓ P(x) log₂ P(x), the average surprisal.
Shannon (1948); Jurafsky & Martin, SLP 3rd ed., ch. 3.An n-gram model estimates P(word | n−1 previous words). Its perplexity on a test set W is PP(W) = P(w₁…w_N)−1/N; “the lower the perplexity… the better the model.”
Jurafsky & Martin, SLP 3rd ed., §3.1, §3.7 (quoted).One word has probability ¼; another has probability 1/32. Give each one's surprisal in bits. Which is more surprising, and by how many bits?
A toy language has 4 phonemes. (a) If all four are equally likely, what are its entropy and perplexity? (b) Now one phoneme has P = 0.7 and the other three share 0.1 each. Is the entropy higher or lower than (a)? Compute it.
Utterance-final particles in a Singlish corpus: P(lah) = .5, P(leh) = .2, P(lor) = .15, P(meh) = .1, P(sia) = .05. Compute the entropy and the perplexity. Compared with EX 2.3(b): which system keeps its listeners guessing more?
EX 2.1 was the pencil break earlier. Worked answers for everything are in the appendix.
Written exactly for you: linguists with secondary-school math. At least the opening sections: probability vs frequency, distributions over words and phones. journals.openedition.org/msh/7933
Skip smoothing for now; we return to it in Week 6. web.stanford.edu/~jurafsky/slp3 (chapter numbers refer to the Jan 2026 release)
Watch before the practical: the area-based picture is the one we draw on the board. 3blue1brown.com/lessons/bayes-theorem
The original guessing-game paper: short and historically delightful. Free at archive.org/details/bstj30-1-50
Play with “Basic Probability” and “Compound Probability” if conditioning still feels slippery. seeing-theory.brown.edu
Meet the course’s second free resource: short pages with runnable audio examples. Unit 1 is waveforms, sampling and spectrograms; the course follows us to Week 13. huggingface.co/learn/audio-course
The sine, sampling and aliasing pages: Week 1’s samples-per-cycle slides, animated and draggable. jackschaedler.github.io/circles-sines-signals
Stop at feature extraction: that is Week 4’s whole story. (Chapter numbers refer to the Jan 2026 release.)
The winding-machine picture behind next week’s spectrograms.
The spectrogram-settings and formant sections: a lab manual you will keep using through Week 5. wstyler.ucsd.edu/praat
Next week: the sound itself becomes numbers: sampling in full (aliasing included), the Fourier idea, spectrograms, and the vocal tract as source and filter.
Hear a sound, weigh every explanation,
pick the least surprising.
That's Bayes, and that's a recognizer. Next week: the sound itself becomes numbers (sampling, spectrograms, source–filter), and you will hear what a 4 kHz telephone ceiling removes from /s/.
Goldsmith, J. (2007). “Probability for linguists.” Mathématiques et sciences humaines 180, 73–98. The probability-vs-frequency distinction, and this week’s required reading.
Jelinek, F. (2005). “Some of my best friends are linguists.” Language Resources and Evaluation 39(1), 25–34. His own account of the “fire a linguist” quote and of IBM’s statistical turn; the LREC 2004 talk slides are free at lrec-conf.org/lrec2004/doc/jelinek.pdf.
Shannon, C. E. (1951). “Prediction and Entropy of Printed English.” Bell System Technical Journal 30(1), 50–64. The guessing game and the ≈1 bit-per-letter estimate.
Ganong, W. F. (1980). “Phonetic categorization in auditory word perception.” JEP: Human Perception and Performance 6(1), 110–125. Warren, R. M. (1970). “Perceptual restoration of missing speech sounds.” Science 167, 392–393.
Aylett, M. & Turk, A. (2004). Language and Speech 47(1), 31–56. Bell, A. et al. (2009). Journal of Memory and Language 60(1), 92–111. Probabilistic reduction: predictability shortens words.
Norris, D. & McQueen, J. M. (2008). “Shortlist B: A Bayesian model of continuous speech recognition.” Psychological Review 115(2), 357–395.
Johnson, K. (2003). Acoustic and Auditory Phonetics, 2nd ed. Blackwell, p. 22 (the Nyquist sentence). Coleman, J. (2005). Introducing Speech and Language Processing. CUP, §2.5.
Jurafsky, D. & Martin, J. H. Speech and Language Processing, 3rd ed. (Jan 2026 release), Ch 3. Definitions quoted on the formal slide.
3Blue1Brown, “Bayes theorem, the geometry of changing beliefs”: the area picture. Seeing Theory (Kunin et al., Brown University): interactive probability.
P(there | signal) = (0.5 × 0.7) / (0.5 × 0.7 + 0.5 × 0.3) = 0.35 / 0.50 = 0.70. The likelihoods tied, so the prior alone decided: context doing the work your ear does automatically.
−log₂(¼) = 2 bits; −log₂(1/32) = 5 bits. The 1-in-32 word is more surprising by 3 bits, and 3 bits means 2³ = 8× less probable.
(a) H = log₂4 = 2 bits; perplexity = 2² = 4 (“choosing among 4”). (b) H = −[0.7 log₂0.7 + 3(0.1 log₂0.1)] = −[−0.360 − 0.997] = 1.36 bits: lower. Skew makes a language more predictable; perplexity falls to 21.36 ≈ 2.6.
H = −(.5 log₂.5 + .2 log₂.2 + .15 log₂.15 + .1 log₂.1 + .05 log₂.05) ≈ .500 + .464 + .411 + .332 + .216 = 1.92 bits; perplexity 21.92 ≈ 3.8, as confused as choosing among about four equally likely particles. Higher than EX 2.3(b)'s 1.36 bits: despite the strong lah bias, the particle system keeps listeners guessing more.