Time and Frequency.
Chenzi Xu · chenzi.xu@ntu.edu.sg
1. Bayes' rule, in one line?
posterior ∝ likelihood × prior: how well the word explains the sound, times how expected the word was
2. Why do decoders add scores instead of multiplying?
log probabilities: products become sums, and nothing underflows to zero
3. A probability of ¼ is how many bits of surprisal?
surprisal = −log₂ P, so −log₂(¼) = log₂ 4 = 2 bits: ¼ is two halvings of certainty, one bit each
4. The held-out rule?
models are evaluated on data they never saw
Today the sound itself finally becomes the list of numbers Week 1 promised.
Cycles per second (Hz). F0 is the physical repetition rate of the folds (~100–250 Hz); pitch is what your ear makes of it: the melody.
Pressure swing (physical), heard logarithmically as loudness. Decibels measure amplitude ratios: dB = 20 log₁₀(A/Aref), so ×2 = +6 dB and ×10 = +20 dB.
Two sounds with the same pitch and loudness can still differ: the waveform's shape carries vowel quality and consonant identity.
LIVE: a pure 120 Hz tone vs an [ɑː] at the same 120 Hz: same frequency, utterly different sound. Whatever “quality” is, it lives in the shape. By the end of today you'll know its real name: the spectrum.
Air molecules bunch up (compression) and thin out (rarefaction); each molecule only jiggles in place while the pattern races outward at ~343 m/s. The middle curve is pressure across space at one instant; the bottom one is pressure at one point across time (the waveform).
The radius wrapped along the rim fits 2π ≈ 6.28 times: that is all a radian is.
angle in radians = arc length / radius
No degrees anywhere: the circle measures itself with its own radius.
| turn | radians | degrees |
|---|---|---|
| quarter | π/2 | 90° |
| half | π | 180° |
| full | 2π ≈ 6.283 | 360° |
A tone at f Hz makes f full turns per second, and each turn is 2π radians: so the angle grows as 2πft. That is the 2π inside sin(2πft): bookkeeping, not magic.
A, the wheel's radius → how far the wave swings: loudness.
f, laps per second → how fast it repeats: pitch, in Hz.
2πf, radians per second → f laps × 2π radians per lap (last slide); ×t gives the angle now. Its name: angular frequency, ω.
φ, the starting angle → where in the lap you began. Ears mostly ignore it (we test this by ear later today); machines keep track.
y(t) = A sin(2πft + φ): the waveform of perfectly steady oscillation, fixed by three numbers: amplitude, frequency, phase. The building block of all sounds.
Jurafsky & Martin, SLP 3rd ed., §15.4.1, eq. (15.3).A tone is described by y(t) = 0.4 · sin(2π · 250 · t). Read off: the amplitude, the frequency, and the period.
A wheel spins 100 times per second with radius 2, and starts a quarter-turn ahead of the usual starting point. Write y(t). Does the tone sound any different for it?
Worked answers in the appendix at the end of this deck.
A repeating pattern has a period; 1/period is the fundamental frequency, F0: how many times the pattern repeats per second. Pitch is what your ear makes of F0.
Noise has no period, so no F0: just energy smeared across frequencies. The ear still hears where that energy sits: /s/ is brighter than /ʃ/.
Every segment you transcribe is some mix of these two ingredients.
continuous pressure wave → snapshots (dots) → quantized steps (the ruler's precision)
Sample rate: how often we look. 16,000 samples per second for speech research; 44,100 for CDs.
Bit depth: how precisely we measure each snapshot. 16 bits ≈ 65,536 ruler marks.
That's the whole trick. A WAV file is these dots, nothing more: Week 1's “list of numbers”, born.
Same sentence, coarser and coarser rulers: 16 bits, then 8, then 4, then 3. Predict first: at which step do you start to hear it?
Playback from pre-rendered clips (bitdepth_16/8/4/3.wav); the practical notebook has the code that made them.
Not softer, but grainier: the rounding errors behave like added noise. Bit depth buys dynamic range (~6 dB per bit); 16 bits is plenty for speech.
Fine for vowels (formants live below ~3.5 kHz). Risky for sibilants: /s/ keeps much of its energy at 4–8 kHz and beyond.
That's why “sailing” vs “failing” is a phone-call classic: the /s/–/f/ cue lives exactly in the band the channel throws away.
You will hear this happen in the practical: resample your own /s/ to a 4 kHz sample rate, and everything above 2 kHz is discarded: the sharp hiss goes dull. A phonetic fact, manufactured by an engineering decision.
The same 16 rungs, two spacings (drawn with µ = 15; the real µ = 255 crowds harder). Both encodings are unpacked on the next slide.
Spend rate and bits where the information is. Speech lives below 8 kHz and rarely spans 50 dB in a real room, so 16 kHz · 16-bit captures essentially everything; anything higher only adds safety margin for editing, not intelligibility.
µ-law spaces its rungs logarithmically, finer near silence, because loudness is heard logarithmically (decibels, Week 2). Even 8 telephone bits are spent where the ear cares.
The samples in Week 1: an integer ladder with equal steps top to bottom. 16 bits, 65,536 rungs, every rung the same size.
The name, unpacked: measure the wave at regular pulses, write each measurement as an integer code. That recipe is PCM; linear adds that the steps are even.
The catch: even steps waste ruler on loud samples. Hearing is more sensitive at small intensities, but linear PCM measures a whisper and a shout with the same coarseness.
F(x) = sgn(x) · log(1 + μ|x|) / log(1 + μ) μ = 255 for 8 bits · J&M eq. (15.5)
Compress with a log, then quantise: 8 bits placed logarithmically sound far better than 8 even ones. Week 2’s log, in every phone call.
Which encoding a file uses is written in the WAV header’s compression field (next slide: 1 = linear PCM, 7 = μ-law).
One box per byte (1 byte = 8 bits); each field is a fixed 4 bytes (2 for the four short ones). Teal = fixed ASCII labels; gold = numbers that differ per file. After Jurafsky & Martin, SLP 3rd ed. (Aug 2026 draft), Fig. 15.12: a simple file with one data chunk; RIFF also allows more chunks, so data need not sit at byte 36.
RIFF · 84036 · WAVE · fmt · 16 · 1 · 1 · 16000 · 32000 · 2 · 16 · data · 84000
Compression 1 = PCM, 1 channel, 16,000 Hz, 16 bits; 84,000 data bytes = 42,000 samples × 2 bytes, 42,000 ÷ 16,000 per second = 2.625 s of audio, and 84,036 = file size − 8 (the 8: the RIFF tag and the size field itself, already read). All the numbers above are stored as 4- or 2-byte binary integers.
“The .wav format is a subset of Microsoft’s RIFF format for multimedia files; RIFF is a general format that can represent a series of nested chunks of data and control information” (Jurafsky & Martin, §15.4.2). Apple’s AIFF is the same idea.
At 4 kHz sampling, who dies first:
/s/ /f/ /i/ /ɑ/ ?
Rank all four with your neighbour and write the order down.
You will test your ranking on your own voice at 12:15. Nyquist has opinions.
a 5 kHz tone (thin) sampled at 8 kHz: the dots fit a 3 kHz wave (dashed) exactly
An undersampled frequency does not vanish: the samples are perfectly consistent with a lower one, so that is what you get back.
Which one? Infinitely many waves fit the same dots (k × 8,000 ± 3,000 Hz); playback returns the unique one below Nyquist: |5,000 − 8,000| = 3,000 Hz.
Same illusion as wagon wheels spinning backwards on film: the wagon-wheel effect, for your ears.
Recorders kill everything above Nyquist before sampling (an anti-aliasing filter).
At 8,000 samples/s: 3,000 Hz steps 135° per tick; 5,000 Hz steps 225°. Each tick stores one number: the height of the spoke’s tip.
+225° forward lands exactly where −135° backward lands, tick after tick. The stored heights fit both wheels: the slow reading is the 3,000 Hz impostor.
Below Nyquist a wave steps less than half a turn (under 180°), and two different such steps can never land alike: so at most one below-Nyquist wave fits any set of samples.
These buttons synthesize audio inside this very slide (no internet needed). Listen in pairs: which two buttons sound identical?
The trapped 5 kHz tone is built sample by sample at an 8 kHz rate, exactly as the figure showed; the samples fit a 3 kHz wave, so a 3 kHz wave is what you hear. And the sweep never reaches its target: it bounces off the 4 kHz ceiling and comes back down.
Stretch. Refill.
Then: the hardest idea, with sound on.
When we return, you will hear a voice assemble itself out of pure tones.
Every sound is a sum of sine waves, and we can recover the recipe.
A prism un-mixes white light into a rainbow. Fourier analysis is the prism for sound.
Each ingredient is trivial: a pure tone with one frequency, one amplitude.
The sum is already complex, voice-like, periodic at the lowest frequency; that is why harmonics sit at multiples of F0.
Hiss fits too: noise is also a sum of sines, just very many of them, at every frequency, with random strengths and phases: no harmonic ladder, energy smeared across the band.
Each button adds one more harmonic of 100 Hz. Listen to a hum become a buzz:
The scrambled version has a wildly different waveform shape, yet it sounds the same: the promised proof that ears mostly ignore phase. The recipe (which frequencies, how strong) is what perception keeps; that recipe is the spectrum.
Same information, different question. Time view: what is the pressure doing? Frequency view: which sines, how strong? Three ingredients, three spikes. The spectrum is the answer to “what's in it?”
To ask “how much 200 Hz is in this sound?”: multiply the signal by a 200 Hz probe sine, sample by sample, and add everything up: literally Week 1's dot product, between two long vectors. Repeat for every probe frequency; the list of answers is the spectrum. (Full honesty: each frequency gets two probes, a sine and a cosine, so a phase-shifted wave cannot hide; that pair is where phase lives. The famous FFT is just a fast way of taking all these dot products at once.)
The coral dot walks the wave; its twin wraps the same heights around the circle.
The centre of mass swings far from the middle only when the winding rate w matches a frequency inside the signal. Sweeping w and recording the swing gives the spectrum: that is the Fourier transform, drawn.
The bump at w = 0 is just the average level of the signal. Credit: after Grant Sanderson (3Blue1Brown), “But what is the Fourier transform?” (2018).
Speech changes every few milliseconds, and one Fourier analysis of a whole utterance averages it all together. So: analyse one frame at a time, and the spectrum becomes a movie.
A sharp cut is itself a click, and a click contains every frequency: its energy smears across the whole spectrum (leakage). Multiply the frame by a bell-shaped window and the smear disappears.
Vocabulary for the practical: frame length (25 ms), step (10 ms), window (the taper). One windowed frame → one spectrum → one column of the next slide’s spectrogram. The taper quiets each frame’s edges; that is why frames overlap: every sample gets a full-weight turn near the centre of some frame.
The taper quiets each frame’s edges, so frames are laid down overlapping. The marked moment is nearly silent in frames 1 and 3 but sits at the centre of frame 2: some frame always hears each moment at full weight. Each windowed frame yields one spectrum: one column of the next slide’s spectrogram.
A spectrum every few milliseconds, painted as a column; darkness = energy. You have read these in Praat for years; now you know exactly what the picture is.
horizontal striations = harmonics of F0
vertical striations = glottal pulses; fat bands = formants
A long analysis window pins down frequency but smears time; a short one does the reverse. Not a software limitation but a law (the same trade-off as Heisenberg's). Choose the view that answers your question.
Why “band”: a window of length T analyses with a bandwidth of about 1/T. 25 ms ≈ 40 Hz, narrower than the spacing between harmonics, so they separate; 5 ms ≈ 200 Hz, wider than F0, so harmonics merge and single pulses show.
Three real Praat spectrograms go up on screen. In teams of three, decode them using the cue cards on the reading-spectrograms slide (press M and jump to it): stops, fricatives, formants.
No circuits needed: averaging each sample with its neighbours already smooths fast wiggles away, a low-pass filter in one line of code. Every filter in this course is a small computation like that.
The telephone’s 4 kHz wall is exactly this picture: multiply the spectrum by a response that is 1 below 4 kHz and 0 above, and /s/ loses its energy. An engineering decision with phonetic consequences.
Keep everything below a cutoff. The anti-aliasing filter; the telephone’s 4 kHz wall.
Keep everything above a cutoff. Strips rumble and microphone-handling thumps.
Keep one band. The telephone channel passes about 300–3,400 Hz.
Remove one band. Kills 50 Hz mains hum without touching the speech.
Same machinery, four response curves: multiply the spectrum by the shape you want. The curve on each card is the frequency response: what survives at each frequency.
A cavity of air favours the frequencies that fit it: energy near its resonant frequency passes amplified; the rest is damped. Change the cavity (a fuller bottle, a different tongue position) and the peak moves.
The vocal tract is a tube holding several resonances at once. Their peaks have a name you know: formants.
Each output sample is a weighted sum of recent input samples: the neighbour-averaging low-pass. Change the weights and the response curve changes shape.
No memory of its own output: a FIR filter (finite impulse response).
Different weights, different job: y[n] = (x[n] − x[n−1]) / 2 cancels slow trends and keeps fast wiggles: a high-pass.
a1 = 2r cos(2πF / fs) a2 = −r²
This filter feeds its own last two outputs back in alongside the input: that is feedback.
Input keeps circulating: after it stops, the output goes on oscillating at F, fading at a rate set by r. F places the peak; r (just below 1) makes it sharp.
Memory of its own output: an IIR filter (infinite impulse response).
Week 10 payoff: a Klatt formant synthesizer is a cascade of these resonators, one per formant; slide 5’s [ɑː] was built this way.
The glottal buzz proposes harmonics; the vocal tract's resonances decide which survive. Move your tongue → retune the filter → the peaks (formants) move. Swap the source (noise at a constriction, a stop burst) and the same filter idea covers fricatives and stops. F1 ↔ height (inverted), F2 ↔ backness: the articulation–acoustics dictionary you already own.
Reproduced from Jurafsky & Martin, SLP 3rd ed. (Aug 2026 draft), Fig. 15.24: tongue positions for three English vowels and the resulting smoothed spectra, F1 and F2 marked.
Same buzz in every case; only the filter changed. F1 falls as the tongue rises, F2 rises as it fronts: the tract shape is legible in the spectrum.
Measure F1 and F2 for hundreds of speakers, plot with both axes flipped, and the IPA vowel chart draws itself.
The ellipses matter: every vowel is a cloud, not a point. Speaker variability is the whole story of Weeks 5–9.
One thing to notice: this is 1952 American English. Where would Singapore English vowels sit? Your own hVd recordings answer that within the hour, and if they land off these clouds, that is data, not error.
Silence (closure) → vertical spike (burst) → rapid formant transitions into the vowel. Place of articulation hides in the transition direction.
High noise = /s/; lower, broader noise = /ʃ/; faint diffuse noise = /f θ/. Energy distribution is the cue.
Formant positions = vowel quality; formant movement = diphthongs, glides, coarticulation with neighbors.
Next week's punchline, planted now: the “features” a recognizer extracts (MFCCs) are an automated, compressed version of exactly this skill: a machine doing narrow phonetic transcription of the acoustics, 100 times per second.
Voice is quasi-biometric data, and this policy is your first case study: in Week 9 we audit how the industry samples voices, and in Week 12 we clone one, with consent, on camera.
Praat open. Colab tab open.
Mic check: New → Record mono Sound.
No mic, or opting out? The fallback recordings are in the course repo; everything works the same.
Praat block: one visit, then one upload
Colab block
Take-home: the formant round-trip (your vowels onto Week 1's chart, next to Peterson & Barney; SgE differences are findings, not errors) + annotated NB/WB screenshots. Stretch: KlattGrid: fix a 100 Hz source, morph the formants /ɑ/→/i/, and hear the filter alone change the vowel (Week 10 seed).
Check the ceiling first (wrong ceiling is the classic cause); avoid creaky stretches (move the cursor); and if your recording misbehaves, the fallback set in the repo is measured the same way. Write every F1/F2 pair down; the take-home needs them.
✓ eight F1/F2 pairs written down
✓ you heard your own /s/ die at 4 kHz
✓ a buzz you built from pure tones
The formant round-trip and NB/WB screenshots are take-home. Your recordings return next week as feature-extraction fodder.
“The sampling rate is the number of samples taken per second… to accurately measure a wave we must have at least two samples in each cycle.” The highest representable frequency, half the sampling rate, is the Nyquist frequency.
Jurafsky & Martin, SLP 3rd ed., ch. 15 (quoted; Aug 2026 draft).“The spectrum of a signal is a representation of each of its frequency components and their amplitudes.” A spectrogram is the spectrum unrolled over time.
Jurafsky & Martin, SLP 3rd ed., ch. 15 (quoted; Aug 2026 draft).“A formant is a frequency band that is particularly amplified by the vocal tract.” Different vowels = different tract shapes = different formant frequencies (F1, F2, …).
Jurafsky & Martin, SLP 3rd ed., ch. 15 (quoted; Aug 2026 draft).Explains any speech sound as a source shaped by a filter (the vocal tract’s resonances). The source varies: the glottal buzz for vowels and voiced sounds, noise at a constriction for fricatives, a burst for stops.
Fant (1960); Jurafsky & Martin, SLP 3rd ed., ch. 15 (Aug 2026 draft).(a) What is the highest frequency a 16 kHz recording can faithfully represent? (b) A fricative has strong energy at 7 kHz. Will a telephone recording (8 kHz sampling) capture it, and what does your answer predict for the /s/–/f/ contrast on the phone?
A voiced waveform repeats every 8 ms. (a) What is its F0? (b) Give the frequencies of its first three harmonics.
A pure 7 kHz tone is sampled at 10 kHz with no anti-aliasing filter (Nyquist = 5 kHz). What apparent frequency will the impostor have?
A 16-bit recording uses 2¹⁶ rungs. (a) Roughly what dynamic range, in dB, does that give? (b) A voice in a quiet room spans about 50 dB, so why do studios still record at 24-bit?
Worked answers in the appendix at the end of this deck.
Waveforms, sampling rate, bit depth, spectrograms, with the exact librosa calls from the practical. Skim the code on first read. huggingface.co/learn/audio-course/chapter1/introduction
Work through the sine-wave, sampling, and aliasing pages before the practical: the animations do in 30 seconds what a page of equations cannot. jackschaedler.github.io/circles-sines-signals
Waveforms, sampling, spectra, source–filter. Stop when it turns to feature extraction; that is next week's reading. web.stanford.edu/~jurafsky/slp3/15.pdf (chapter numbers follow the Aug 2026 draft)
The winding-machine visualization, for anyone who wants to open the black box. First 10 minutes are the payoff. 3blue1brown.com/lessons/fourier-transforms
Reference for the practical: the sections on spectrogram settings and formant measurement. wstyler.ucsd.edu/praat
The exact librosa calls next week’s practical uses. huggingface.co/learn/audio-course/chapter1/audio_data
The classic walkthrough of next week’s whole pipeline; focus on the figures. haythamfayek.com
The textbook version of the same pipeline, then the first page of the recognition story. (Chapter numbers follow the Aug 2026 draft.)
A gentler second pass at the same material, with worked numbers. practicalcryptography.com (archived copy linked from NTULearn if the site is down)
Next week: the spectrogram is compressed into thirteen numbers per frame: the features every recognizer actually hears.
The buzz proposes.
The tract disposes.
Source × filter: that's a vowel, and that's half of speech technology. Next week, “Thirteen Honest Numbers”: we compress the spectrogram into the 13 numbers every recognizer actually hears.
Jurafsky, D. & Martin, J. H. Speech and Language Processing, 3rd ed. (Aug 2026 draft), Ch 15. The acoustics sections; the WAV header figure redrawn after Fig. 15.12; PCM and μ-law from §15.4.2, eq. (15.5); definitions quoted on the formal slide.
Peterson, G. E. & Barney, H. L. (1952). “Control methods used in a study of the vowels.” JASA 24(2), 175–184. The F1×F2 vowel data, as in Week 1.
Fant, G. (1960). Acoustic Theory of Speech Production. Mouton. The source–filter model.
Shannon, C. E. (1949). “Communication in the presence of noise.” Proc. IRE 37(1), 10–21. The sampling theorem as taught here; the limit is named for Nyquist (1928).
Interactive resources: Schaedler, Seeing Circles, Sines, and Signals; 3Blue1Brown, “But what is the Fourier Transform?”; Hugging Face Audio Course, Unit 1.
Amplitude A = 0.4; frequency f = 250 Hz; period T = 1⁄250 s = 4 ms.
y(t) = 2 · sin(2π · 100 · t + π⁄2): a quarter-turn = π⁄2. It sounds identical (same loudness, same pitch); only the starting point changed. Phase moved, the ear shrugged.
(a) Half the sampling rate = 8 kHz. (b) Telephone Nyquist = 4 kHz, so the 7 kHz energy is lost. /s/ keeps much of its energy above 4 kHz while /f/ is flatter; the phone strips exactly the cue that separates them.
(a) F0 = 1 ⁄ 0.008 s = 125 Hz. (b) Harmonics at integer multiples: 125, 250, 375 Hz.
Aliased frequency = |7 kHz − 10 kHz| = 3 kHz: the impostor from the lecture figure. The 7 kHz tone masquerades as 3 kHz because the samples fit both. (One caution: this simple subtraction works only between Nyquist and the sample rate; the general rule is that frequencies mirror around Nyquist, folding back and forth, so a 17 kHz tone at 10 kHz sampling lands at 3 kHz too, not 7.)
(a) About 6 × 16 = 96 dB (2¹⁶ = 65,536 rungs). (b) The extra 8 bits are not for the listener: they are headroom, so you can record quietly and normalize later, and so the many processing steps that each round a little never reach the noise floor. On the finished voice, 16-bit already sits below the microphone's own hiss, so the steps stay inaudible.