A video call ends. The transcript lands in the chat window. One speaker’s lines read as clean paragraphs. Another speaker, using the same kinds of sentences in the same meeting, comes out as half-sentences and words nobody said. The microphone was the same. The room was the same. What changed was whose voice the transcription software had been trained to expect.
Caption systems rarely fail loudly on accented speech. They fail politely, producing fluent-looking text that happens to be wrong, and the person reading that transcript has no way to spot the difference.
What the System Actually Computes
A captioning system does not hear words the way a person does.
Photo by It converts sound into short slices, then calculates which sequence of letters or word-fragments is most probable given those slices. That probability score, not the actual sound, is what the system optimizes during training.
The standard way to grade the result is word error rate (WER), which measures the share of words a system inserts, deletes, or gets wrong compared with what was actually said. A WER of 10% means roughly one word in ten is off.
Two things follow from that design. The model learns the pronunciations that showed up often in its training recordings, not the full range of ways a sentence can be spoken. And its language model favors word sequences that appeared frequently in text, so an unfamiliar vowel sound gets “corrected” into a plausible but wrong word rather than flagged as unclear.
For a general reader, this means a caption tool never admits uncertainty. It just hands you its best guess, dressed up as a finished sentence.
Where It Quietly Breaks
The pattern is simple: accuracy tracks how closely a speaker’s pronunciation matches the voices in the training data.
Photo by The further a speaker sits from that center, the wider the error band gets.
A 2024 study measured that gap directly. Across the systems tested, native English speakers saw word error rates from 1.3% to 29.8%. Non-native speakers saw a much wider range: 8.0% to 65.6%. The floor moves a little. The ceiling moves a lot, and that’s the part that matters. At a 65.6% error rate, roughly two out of every three words come out wrong, yet the transcript still reads like ordinary sentences.
Other research points the same way. One analysis found commercial systems misrecognizing speakers of African American Vernacular English, a dialect with its own grammar and pronunciation patterns, at nearly twice the rate of white speakers1. A separate comparison across models found minority dialects performing worse than Standard American English in every model tested. Evaluations of one widely used open model found it recognized American English more reliably than British or Australian English.
As AssemblyAI puts it: “Accents are a data and model-capacity problem, not a speaker problem.”
For a general reader, the takeaway is unsettling: the failure never comes with a warning light. A garbled caption looks exactly like an accurate one.
Why This Persists
These accent gaps aren’t a bug waiting for a patch.
Photo by They’re a boundary drawn by the available data, and that boundary is expensive to move.
Recordings pile up where they’re cheapest to collect: from the speakers, regions, and recording conditions already flowing through existing products. Adding more data for the well-covered middle of that distribution is nearly free, since more of the same keeps arriving on its own. Adding coverage for a dialect that was never sampled means recruiting speakers on purpose, recording them, and paying for careful transcription. Even the audit that would reveal the gap runs into the same wall: you can’t measure error on voices you never collected.
Downstream fixes show how much is recoverable, and how much isn’t. In one accessibility study, running raw captions through a language model cut WER from 23.07% to 9.75%, a real improvement of roughly 58%. That’s worth having. It’s also worth naming for what it is: a text-level cleanup guessing at what was probably meant, without ever getting a second listen to the audio that got misheard in the first place.
Accent rarely shows up alone, either. Automated captions get worse with background noise, overlapping talkers, and normal speech disfluencies, and those conditions cluster in exactly the settings where accented speech is common: crowded meetings, phone calls, noisy classrooms.
For a general reader, this means the model’s blind spots are really a record of whose voices were easy and cheap to record in the first place.
Go back to that transcript in the chat window. The clean paragraphs and the broken ones weren’t produced by two different levels of clarity. One system, working from one history of recordings, produced both, confident either way. The useful shift is to stop reading a caption as a record of what was said and start reading it as a record of what the system had heard before. When a transcript turns someone’s sentence into words nobody spoke, the honest explanation usually isn’t that the speaker was unclear. It’s that the system was working from a smaller library of human voices than the room actually contained.
Sources
- ASR misrecognition rates for African American Vernacular English speakers
- Word error rates for native and non-native English speakers
- Accent performance across ASR models
- ASR evaluation across native and non-native English accents
- DSAI 2024 captioning accessibility study
- Transcribing heavy accents
- Factors that downgrade automated captions
Try it yourself
- Open live captions or voice dictation on your phone and say one fixed sentence, such as "can you turn on the lights," five times in your natural accent.
- Record what the caption shows each time, word for word.
- Say the same sentence five more times while deliberately flattening your vowels toward a general American pronunciation.
- Compare how many of the ten transcripts are exact matches.
- What you should see
- The flattened-pronunciation attempts match more often, and the misses in the natural-accent set read as confident, well-formed sentences rather than as gaps or question marks.
- Why it happens
- The sentence and the microphone never changed, so the difference in accuracy comes from how close the pronunciation sits to the voices in the training data. What the system never sampled is what it gets wrong.
