Riiven Threads
Speech Recognition
Your Voice Was Statistics
The story you know
Speech recognition finally worked because computers got smart enough to understand language, like a digital ear that learned to listen.
What the record shows
The documents show recognition worked because statisticians modeled speech as hidden sequences decades before any system understood a word, and government funded corpora gave those models enough labeled sound to train on; deep learning in 2012 only replaced one probability estimator with a better one.
A phone wakes at "Hey Siri" and behaves as though it understood you. It did not. You say "Hey Siri" and expect understanding, but you get arithmetic: a running bet on which sound tends to follow which. The parts of that bet arrived separately. Statisticians had a way, by 1987, to find the most likely path through a stream of audio without knowing what a word is. A phonetics lab had measurements of what vowels actually do to a waveform. DARPA paid for the recordings every lab would be graded on. Neural networks turned up in 2012 to score the guesses better. So the question is not how machines learned to understand speech. It is why refusing to model meaning at all is what finally made voice interfaces work.
- 33.3% WER cut
- Error reduction the 1987 segment model won over a plain hidden Markov phonetic model.
- 6,300utterances
- TIMIT recordings from 630 American English speakers, funded and distributed by DARPA.
- 32.1% relative
- Accuracy gained on Punjabi children's speech by normalizing for shorter vocal tracts.
- 8.9points WER
- Switchboard penalty without deep neural acoustic models: 27.4% instead of 18.5%.
How each idea was handed down
Speech Recognition inherited its parts in sequence. Each field handed its result to the next.
The inheritance, in order
- 1987 Hidden Markov Models and Statistical Sequence Modelinghanded down ↓
- 1988 Government Funded Language Corpus Policyhanded down ↓
- 1991 Acoustic Phoneticshanded down ↓
- 2012 Artificial Neural Networks and Deep Learning
Follow the inheritance, link by link.
Each link hands its result to the one after it. The chart above places them by date.
Keystone
The decoder that guessed without knowing words
Play back a recorded sentence and the silences land in the wrong places. Words do not come with gaps.
Continuous speech is one smear of sound, and the pauses you think you hear are mostly not there. The 1987 stochastic segment model treated the smear as a chain of hidden states, states you never observe directly, and asked which sequence of them most likely produced the audio that arrived. Against a standard hidden Markov phonetic model, that approach cut word error rate by 33.3 percent on speaker-dependent continuous speech. The search stayed affordable through dynamic programming, a bookkeeping trick that reuses partial answers instead of retrying every possible path. Nothing here knows meaning. It knows that some sounds follow other sounds often, which is also what makes forced alignment possible: lining a transcript up against audio frame by frame.
Without this link
Without statistical sequence modeling, a recognizer has no principled way to turn variable-length audio into the most probable phone sequence. Forced alignment breaks, and continuous speech decoding loses the dynamic programming that maps audio frames to a transcript path.
Without the stochastic segment model, continuous speech recognition forfeits a 33.3 percent word error rate reduction over a hidden Markov phonetic model.
How we know
Forced alignment is the quiet workhorse. Once frames can be assigned to phone states by probability, the same alignment becomes training data for the next model, which is why statistical decoding bootstrapped everything built on top of it.
Source: A stochastic segment model for phoneme-based continuous speech recognition (1987) · tier1
A decoder that ranks probabilities is only as good as the recordings it was trained and tested on.
Somebody had to pay for the tapes
Strangers sat in quiet rooms at MIT and read scripted sentences into a microphone. That was federal money at work.
Nobody publishes a career on recordings they made in their own hallway, because no rival lab can check the result. DARPA funding turned collection into infrastructure: TIMIT, then Resource Management, ATIS, and the CSR Wall Street Journal sets, designed once and shipped to every institution in the field. TIMIT alone supplied 6300 recorded utterances from 630 American English speakers as a shared benchmark. That is what made word error rates comparable between groups instead of anecdotal. Progress on large-vocabulary continuous speech became a public scoreboard rather than a set of private claims.
Without this link
Without funded corpus policy, standardized sets like TIMIT, RM, ATIS and the CSR Wall Street Journal corpora would never have been designed, collected and distributed across institutions. Labs would tune on small ad hoc datasets, with no way to compare systems or measure real gains in accuracy and generalization.
Without DARPA sponsorship of TIMIT, 6300 utterances from 630 American English speakers would not exist as a standard benchmark.
How we know
Source: Speech database development at MIT: TIMIT and beyond (1990) · tier1
Shared tapes raised a harder question: which measurements in those waveforms actually correspond to a vowel.
Your throat is a tube with resonances
Say "ee" then "ah" and feel your tongue drop. The tube changed shape, so the sound changed loudness bands.
Those bands are formants: the frequencies a given mouth shape amplifies, visible as bright stripes on a spectrogram. Phonetics work going back to 1972 tied them, plus voice onset time (the lag between a consonant's release and the voice starting), to specific articulations. That gave decoders features grounded in anatomy rather than guessed spectral patterns. It also explained a failure mode. Children have shorter vocal tracts, and normalizing for tract length and prosody buys 32.1 percent relative accuracy on Punjabi children's speech.
Without this link
Without acoustic phonetics, systems have no empirical link between vocal tract resonances and measurable waveform features such as formant frequencies and voice onset time. Mapping continuous speech to vowels and consonants falls back on ad hoc spectral patterns, and robustness collapses across speakers, ages and speech pathologies.
Without vocal tract length and prosodic normalization, Punjabi children's speech recognition loses a 32.1 percent relative accuracy improvement.
How we know
Source: Formants in automatic speech recognition (1972) · tier1
Hand-measured features and shared corpora set a ceiling that better scoring, not better microphones, finally broke.
The referee that arrived twenty-five years late
Switchboard is telephone calls between strangers, badly recorded and fast. For two decades it was where recognizers went to look bad.
The 2012 work by Hinton, Dahl, Mohamed and Graves did not replace the decoder. It replaced the scoring layer, swapping Gaussian mixtures (each sound modeled as a blend of bell curves) for a deep network that learns its own features from the audio. On large-vocabulary Switchboard speech, that swap moved word error rate from 27.4% to 18.5%, a gap of 8.9 percentage points. The statistics stayed statistics. They just got a far better estimate of how likely each frame of sound was.
Without this link
Without deep learning, recognition stays capped at Gaussian mixture hidden Markov systems with markedly higher error rates and poor tolerance for noise and speaker variation. End-to-end acoustic-to-word systems, low-latency real-time transcription and near-human accuracy across many languages would all be out of reach.
Without deep neural acoustic models, Switchboard recognition stays 8.9 percentage points worse: 27.4% error instead of 18.5%.
How we know
Recurrent end-to-end models trained with CTC and attention pushed error rates into single digits on benchmark corpora, mapping audio straight to characters or words without a separate phone dictionary.
Source: Deep Neural Networks for Acoustic Modeling in Speech Recognition (2012) · tier1
Watch
A visual companion to the fields above.
Takeaway
The chain never contained a definition of a word. A statistical decoder ranked paths through audio, phonetics told it which measurable ridges in a waveform were worth ranking, DARPA's corpora decided whose ranking was better, and deep networks scored the frames more sharply after 2012. What that stack produced by 1992 was the first speech systems that could be dropped into a product and still work on a speaker they had never heard. The tension has not gone away. A recognizer that models only sound sequences is fragile exactly where sound is unusual: children, accents, disordered speech, anyone outside the recordings someone paid to collect. Every gain since has come from either better statistics or more voices in the dataset, never from teaching the machine what the sentence is about. When a transcript fails you, that is the seam to look at, not the sentence you spoke.
References
- A stochastic segment model for phoneme-based continuous speech recognition (1987) tier1
Mermelstein et al., IEEE, 1987
- Speech database development at MIT: TIMIT and beyond (1990) tier1
Lamel, Kassel, Seneff, Speech Communication, 1990
- Formants in automatic speech recognition (1972) tier1
Formants in automatic speech recognition, Journal of Phonetics, 1972
- Deep Neural Networks for Acoustic Modeling in Speech Recognition (2012) tier1
Hinton et al., IEEE/ICASSP tutorial paper on DNN-HMM speech recognition, 2012
