On-device speech model accuracy: seven models, published numbers only

Brain Dump ships a model picker with seven on-device speech models, from a 32 MB file to a 574 MB file, all user selectable, all deletable, none of them calling a network. Two of them transcribe languages other than English, one of those across 25 languages from a 485 MB file on a phone. Choosing between them needs accuracy numbers, and accuracy numbers are where this category lies most often.

So the rule for this page: every number below was measured by somebody else, and it says who. We have not measured a word error rate. Not for one model, not on one device. Where nobody credible has published a number for the exact file the app downloads, the cell says UNKNOWN and stays UNKNOWN. A borrowed full-precision figure standing in for a quantized file is the specific dishonesty this page exists to avoid.

What word error rate is, and what it hides

Word error rate is substitutions plus deletions plus insertions, divided by the number of words in the reference transcript.

WER = (substitutions + deletions + insertions) / words in the reference

Lower is better. It also treats every mistake as equal. "There" for "their" and a mangled surname both score one error, and you do not experience those as the same mistake.

The seven files, and what has been published about each

The four transcribe.cpp models are converted and published by handy-computer, the team behind transcribe.cpp. They publish WER for each quantized file they ship, not for a full-precision proxy, on the full LibriSpeech test-clean split, 2,620 utterances, greedy decoding, no external language model, with an F32 run through the same harness for comparison. That is why those four rows exist at all.

Sizes are the byte counts the app expects, in base-10 MB. Each WER links to the measurement it came from. Every WER in that column is measured on English audio.

Four things belong with that table, and none of them are optional.

The three UNKNOWN rows are the Whisper models, and they are UNKNOWN on purpose. The app downloads quantized ggml builds (q5_1, q5_0). Whisper's own paper measures full-precision checkpoints. What quantization costs those particular ggml files has not been measured by anyone publishing their method, and the figures circulating are blog-grade. Whisper Small at full precision scores 3.4 on this split. That number is not in the table because it is not a number about the file on your phone.

Quantization looks nearly free here, on clean read speech only. The largest gap in the table is a tenth of a percentage point, on Parakeet 110M. On Moonshine and Canary the quantized file scores fractionally under F32, and the Moonshine card says outright that the gap sits inside the 95% bootstrap confidence interval and is noise, not an improvement. Nobody has measured what quantization costs under noise, under an unfamiliar accent, or on very short utterances, which is exactly where a small model is most fragile.

Two of these models are multilingual, on the device, with no network call. Parakeet 600M v3 transcribes 25 European languages and detects the language itself rather than being told which one it is hearing: Bulgarian, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hungarian, Italian, Latvian, Lithuanian, Maltese, Polish, Portuguese, Romanian, Russian, Slovak, Slovenian, Spanish, Swedish and Ukrainian, as listed on NVIDIA's card. Canary 180M Flash recognises English, German, Spanish and French out of a 139 MB file (NVIDIA). Parakeet 110M and Moonshine Base are English only. The three Whisper files are the multilingual checkpoints, not the English-only builds.

Supporting a language and being accurate in it are different things. Canary 180M Flash scores 1.87 on LibriSpeech test-clean and 9.53 on Common Voice 16.1 English, same model and same card: five times the error rate, from the speakers alone. Parakeet 600M v3 averages 11.97% across the FLEURS languages, ranging from 3.00% on Italian to 24.03% on Slovenian, on the same benchmark on the same day. Every WER in the table above is an English number, and it does not carry over.

Comparability has limits even inside the table. Canary's porter measures 1.93% and NVIDIA reports 1.87% on the same split, a difference of roughly one substitution in 27,000 words, caused by harness and normalizer rather than by the model. Different rows normalize text differently. Small gaps between rows are not real differences.

Why a single test-clean number tells you almost nothing

LibriSpeech test-clean is read audiobook speech: scripted, close mic, no background noise, no false starts, no proper noun the model has never seen. A voice note is none of those things. The model authors publish the evidence for how far apart those two worlds are.

One model, one row of one table, nine different corpora. From the Whisper paper's Table 8, greedy decoding, the multilingual Whisper Small checkpoint:

Same weights, same decoder, same day. The number moves from 3.4 to 29.3, an 8.6x spread, and accent alone accounts for the move from 3.4 to 18.1. Source: Radford et al., Table 8.

Noise is measurable and it is large. NVIDIA publishes a signal-to-noise sweep for Parakeet 600M v3, mixing MUSAN music and noise into the eight Open ASR Leaderboard sets:

Source: NVIDIA parakeet-tdt-0.6b-v3. A phone held at arm's length in a cafe is not the clean row.

Put those together and the honest summary is short. Accent, background noise, disfluency, domain vocabulary and quantization all move the real number, most of them by more than the entire spread between the models in the table above. A leaderboard figure on clean read speech predicts almost none of what you will get.

Two claims this page does not make

Nothing about speed. The four transcribe.cpp models currently run CPU only, with Metal switched off in our integration, and nobody has timed them on a phone. Any seconds-per-minute or time-to-first-word figure would be invented.

Nothing about which phone you have. Every supported device gets the same catalog. The picker shows all seven models on every device, and any of them can be downloaded and deleted again. If you want the 574 MB model on an old iPhone, the picker will let you.

What we will measure, and have not measured yet

The published table is somebody else's work on somebody else's corpus. Here is the benchmark we intend to run and publish, stated in advance so the result can be checked against the plan.

Corpus. 60 utterances, 20 to 60 seconds each, in three sets of 20. Set A is read reference text, the only set comparable to the published LibriSpeech figures. Set B is natural voice notes with real disfluency, self-correction and domain vocabulary. Set C is Set B re-recorded in a noisy room with the SNR measured and recorded, targeting 5 to 10 dB. At least three speakers, at least one a non-native English speaker. Recorded inside the app, so the audio passes through the capture path we actually ship.

Ground truth and scoring. One human verbatim transcript per utterance. The normalization spec written down before scoring, using the Whisper text normalizer, cited, so Set A stays comparable to the leaderboard. Scoring with jiwer, reporting WER plus the substitution, insertion and deletion split, plus a 95% bootstrap confidence interval. Twenty utterances is a small sample and that interval will be wide. Publishing it is the difference between a number and a claim.

Devices. Three: one older iPhone, one recent iPhone, one Apple Silicon Mac.

Also recorded, because nobody publishes it. Time to first text, wall-clock seconds per minute of audio, peak resident memory, and thermal state at the end of the run. On a phone those decide whether a model is usable more than WER does.

The repro script, per the standing rule that every number we publish ships with one. A scoring script taking a results directory and a corpus directory, a runner that drives the CLI at fixed printed decode settings and writes the model file's SHA-256 into every result, and a corpus manifest listing speaker, accent, set, duration, measured SNR and file hash for each utterance. If speaker consent does not allow publishing the audio, the manifest and hashes get published anyway and we say plainly that the audio is withheld.

Until that runs, this page has no number of our own on it, and it will not pretend otherwise.

Measuring your own dictation in ten minutes

You do not need any of the above to check the setup in your hand. Pick a 150-word passage with general vocabulary and no proper nouns you would have to spell. Read it once silently so your pace on the real run is natural. Dictate it into a blank note, copy the text out, and tally substitutions, deletions and insertions against the source. Punctuation the app inserts by itself does not count. Divide by 150.

Repeat three runs per condition and average. Change one variable at a time: quiet room against light background noise, phone at 15 to 25 cm against arm's length, same speaker and same passage throughout. Noise and mic distance are where the differences show up, and a result you produced yourself on your own voice beats every published table on this page.

Related: dictation tips and mic positioning.