What Is an ASR Model? How Automatic Speech Recognition Actually Works

You grab the standard open-source Whisper model because you want live captions. You wire it up, point a microphone at it, and wait for stable text to stream in as you talk. It does not work that way. The transcript shows up in chunks, and the reason is not a missing toggle; it is how the standard Whisper model is built.

That confusion happens often due to a misunderstanding about the architecture behind Whisper. Standard open-source Whisper is batch-oriented by design, and once you understand the main ASR architecture families, its behavior becomes easier to predict.

Automatic speech recognition sits under meeting transcription, voice interfaces, podcast search, call-center QA, and increasingly under the self-hosted AI people run next to their LLMs. Most builders have touched one ASR tool and been caught off guard by it. This article gives you the one mental model that makes the whole space legible: three architectural families, and how each family's structure produces its real-world behavior. Once you have it, “why does standard Whisper behave like a batch model?” and “is Vosk worth it?” become much easier to answer.

What Is an ASR Model?

An ASR (Automatic Speech Recognition) model converts spoken audio into text. It takes a numerical representation of the audio (a log-mel spectrogram), passes it through a neural encoder, and produces a text transcript via a decoder. Modern ASR models differ mainly in how the decoder aligns the audio to the words.

ASR and STT (speech-to-text) are the same thing; the HuggingFace ASR task page lists them as synonyms, so don't burn any energy trying to find a distinction. The interesting question isn't what the acronym stands for. You already know roughly what transcription does. The interesting question is why these models behave so differently from one another, and that's almost entirely a story about the decoder.

That's the thread to hold onto for the rest of this article. The encoder side is fairly standard across models. The decoder side is where the families split, and where every practical difference you'll hit comes from: how a model turns the encoded audio into words, and whether it can do that incrementally or only all at once.

How Does Audio Become Text?

In a Whisper-style pipeline, audio is resampled and converted into a log-mel spectrogram before being passed to the model. Whisper processes long audio through 30-second windows, and the exact spectrogram dimensions depend on the model version. The neural model reads this, not the raw waveform, and maps it to text tokens. (Those numbers are from the Whisper repo, which is a clean reference for the standard pipeline.)

Think of it as three stages. First, the raw waveform (amplitude over time) is converted into a spectrogram because frequency content carries phonetic information; raw amplitude alone is hard for a model to interpret. Second, the encoder turns the spectrogram into a sequence of embeddings: a compressed, learned representation of what's happening acoustically across the clip. Third, the decoder turns those embeddings into tokens, which assemble into words.

The reason any of this is hard, the reason ASR isn't a lookup table, is alignment. Speech is ambiguous at the acoustic level. The classic example: "recognize speech" and "wreck a nice beach" produce nearly identical audio. The model can't just match sounds to a dictionary; it has to figure out which sequence of words the audio most plausibly maps to, given context. How a model solves that alignment problem is exactly what separates the three families.

What Are the Three ASR Architectural Families?

Most ASR systems fall into a few practical architecture/decoding styles: encoder-decoder / attention models such as Whisper and Canary-Qwen, CTC-style models such as wav2vec 2.0, Kaldi/Vosk-style hybrid recognizers, and transducer / TDT systems such as NVIDIA Parakeet. They differ in how they align audio to text, and that design choice cascades into whether a system can stream, how fast it runs, and how much context it can use.

The three families are organized by how each one solves the alignment problem: matching a variable-length stretch of audio to a variable-length stretch of text. Each solution buys you something and costs you something, and the trade-off is the thing you'll feel in production, so it's worth understanding each on its own terms.

Encoder-Decoder (Attention): Whisper, Canary-Qwen

In an encoder-decoder model, the encoder processes an audio chunk into embeddings, and then the decoder generates text one token at a time while attending over the encoded input. For the standard open-source Whisper model, that means working over fixed audio windows rather than emitting stable transcript deltas from a continuous stream. That batch-oriented design is part of why Whisper is robust, but it is also why the standard model is not a native real-time captioning system.

Whisper is a Transformer encoder-decoder trained on a large, messy pile of audio, which is part of why it's robust across accents and noise. Speech-LLM systems such as Canary-Qwen are useful examples of the high-accuracy end of modern ASR. They pair a speech encoder with an LLM-style decoder and often rank near the top in benchmark accuracy, but they are usually slower than CTC or TDT systems in terms of throughput.

CTC and Kaldi/Vosk-Style Streaming Recognizers

CTC (Connectionist Temporal Classification) takes a different route: it outputs a probability distribution over labels for each slice of audio, and uses an alignment trick with blank tokens and repeated-label collapsing so the model does not need a precomputed frame-to-character alignment. Because CTC-style models can score frames incrementally, they are often well-suited to low-latency recognition.

Vosk is a different kind of practical streaming option. It is Kaldi-based, streams natively, ships small per-language models, and can run on lightweight hardware such as a Raspberry Pi or a phone. Its architecture is closer to the classic hybrid ASR stack: acoustic model, language model, phonetic dictionary, and graph-based decoding. The practical trade-off is still similar from a builder’s perspective: Vosk is lightweight and streamable, but it usually will not match the strongest encoder-decoder or speech-LLM systems in terms of transcription accuracy on difficult tasks.

Transducer / TDT: NVIDIA Parakeet

A transducer pairs an audio encoder with a separate prediction network, which lets it stream like CTC while still modeling how each token depends on the ones before it, closing much of the accuracy gap. NVIDIA's TDT (Token-and-Duration Transducer) pushes this further by predicting both the next token and the number of audio frames it spans, so the model can skip ahead over blank frames instead of emitting one prediction per frame.

The payoff is throughput. NVIDIA's Parakeet-TDT write-up describes the model making 8 predictions for an 8-word sentence, where a frame-by-frame approach would make 33 predictions, running 64% faster than the comparable RNN-T model, and processing 10 minutes of audio in a single second. You get streaming capability and strong accuracy at speeds the encoder-decoder family can't approach, which is why transducers dominate the high-throughput end of the leaderboard.

The family a model belongs to predicts its streaming ability, speed, and resource needs more reliably than its name does.

Why Doesn’t Standard Open-Source Whisper Stream Natively?

Standard open-source Whisper does not stream natively because it is an encoder-decoder model built around audio chunks rather than continuous transcript deltas. The decoder works after the model has encoded a chunk of audio, so the standard model is naturally batch-oriented.

That is why most “streaming Whisper” setups are workarounds. They feed Whisper short, rolling chunks, use voice-activity detection to trigger cuts on pauses, or wrap the model in logic that repeatedly reprocesses recent audio. This can feel near-realtime, but it is still not the same as a model designed to emit low-latency transcript deltas from live audio.

The current caveat is important: this applies to standard/open-source Whisper, not every OpenAI Whisper-branded transcription product. OpenAI’s GPT-Realtime-Whisper is a separate streaming speech-to-text model for real-time transcription. So the practical advice is not “Whisper can never stream.” The practical advice is: use standard Whisper for batch transcription, and use a real-time transcription model or another streaming-capable recognizer when live captions are the hard requirement.

What Is Word Error Rate and What's a Good WER Score?

Word Error Rate measures ASR accuracy as (substitutions + deletions + insertions) divided by the total number of words in the reference transcript. Lower is better, and 0 is perfect. A 7% WER means the system made about 7 word-level edit errors per 100 reference words. Strong modern ASR systems can achieve low-single-digit WER on some clean benchmark datasets, but leaderboard averages across multiple datasets are usually higher.

The formula counts three kinds of mistakes: substituting a wrong word, deleting a word that was said, and inserting a word that wasn't. Add those up, divide by how many words there were, and you get a single percentage. It's a clean metric, which is exactly why it's easy to over-trust.

The gap that catches people is between the lab figure and what you'll see in production. Benchmark audio is clean: read speech, decent microphones, little background noise, vocabulary the model has seen. Your audio has accents, crosstalk, jargon, a laptop fan, someone eating. A Hacker News thread on real-world ASR accuracy captures the frustration well: developers regularly report that models advertising above 95% accuracy in the lab drop into the mid-80s or worse once they hit messy production audio. The leaderboard score isn't lying; it's just answering a different question than the one you care about.

WER reality check: Treat a model's benchmark WER as a best-case reference point, not a promise. The only number that tells you whether a model works for your use case is the WER you measure on a sample of your own audio: your accents, your noise floor, your domain vocabulary. Measure before you commit.

A WER number only means something when it's attached to the audio it was measured on.

What Should You Know Before Trusting ASR Output?

Two things tend to surprise builders once an ASR model is in production. First, Whisper can hallucinate (produce fluent words that were never in the audio) in roughly 1% of segments, and the rate climbs as the audio gets noisier. Second, "faster-whisper" is not a different model; it's the same Whisper weights running through a faster inference engine. Both of these change how you should plan around the tool, and neither is obvious from a feature list.

The hallucination finding comes from the peer-reviewed ACM FAccT 2024 paper "Careless Whisper", which documents that about 1% of audio segments contained entire hallucinated phrases, and that hallucinations increased at low signal-to-noise conditions (-4 dB and -2 dB). In the tested conditions, the competing commercial systems didn't show a comparable problem. I read this the way I'd read any non-deterministic production failure mode: it's not a reason to avoid Whisper, it's a reason to design for it. If you're transcribing low-quality audio and acting on the output automatically, you want a confidence check or a human in the loop on the segments that matter, because the model can be confidently wrong on a quiet passage.

The hallucination caveat: Whisper's failure mode isn't garbled text you can spot. It's clean, plausible, invented text. Hallucinations cluster on low signal-to-noise audio (silence, heavy noise), so the riskiest segments are the ones a human skimming the transcript is least likely to double-check.

The faster-whisper confusion is more benign but trips up almost everyone self-hosting Whisper on GPU infrastructure. faster-whisper is a reimplementation of Whisper on top of CTranslate2, an inference engine tuned for Transformer models. Same weights, same accuracy, up to roughly 4x faster while using less memory, and its INT8 quantization mode trades a negligible amount of accuracy for further speed and a smaller memory footprint. So when you see "faster-whisper" in a setup guide, you're not choosing a different model with different transcription behavior; you're choosing a faster way to run the model you already understand.

How Do You Choose an ASR Approach?

Start from your constraint, not from the model. Need real-time captions? You need a streaming-capable system, such as a Kaldi/Vosk-style recognizer, a CTC-style model, or a transducer model. Want maximum accuracy on recorded files where latency does not matter? Use an encoder-decoder or speech-LLM model. Running on a CPU or edge hardware with no GPU? Vosk is a practical lightweight option. In every case, the architecture narrows the choice before the model name does.

That's the whole point of the mental model: you don't memorize which tool is "best," because there is no best. There's only best-for-a-constraint. Once you can name your hard requirement, the family falls out of it, and the field of candidate models shrinks from dozens to a handful. Pick the dominant constraint (latency, accuracy, hardware, or language coverage), match it to a family, then compare the two or three models inside that family on the details that matter to you.

Your next concrete step is to write down that one constraint before you open a single model card. If it is latency, you are in streaming-capable territory: Kaldi/Vosk-style recognizers, CTC-style models, or transducers. If it's accuracy on recorded audio, you're in encoder-decoder or speech-LLM territory, and you can ignore the streaming benchmarks. When you're ready to move from picking a family to standing up a model on your own hardware, that's where the deployment work begins.

ASR Model Decision Cheatsheet

The table below summarizes how four representative models map onto the three families and what that means for the properties you'll care about. The WER ranges and speed classes are drawn from the HuggingFace Open ASR Leaderboard and the vendor documentation cited throughout this article; treat them as benchmark figures, not guarantees for your audio.

Model	Family	Streams natively?	Typical WER range	Relative speed	Resource footprint
Standard open-source Whisper (large)	Encoder-decoder	No native streaming	~5–7% (depending on benchmark)	Slow (baseline)	Heavy (GPU-class model)
Canary-Qwen	Encoder-decoder / speech-LLM	No	~5.6% (leaderboard-leading)	Slower than CTC/TDT	Heavy (GPU-class)
Vosk	Kaldi-based hybrid / graph-decoding recognizer	Yes	~8–15% (depending on model and benchmark)	Fast on CPU	Light (runs on CPU, Pi, mobile)
Parakeet-TDT	Transducer / TDT	Yes	Low single digits (leaderboard)	Very fast	GPU, but extremely high throughput

FAQ

Frequently Asked Questions

01What Is the Difference Between Whisper and Vosk?

Standard open-source Whisper is an encoder-decoder model: strong accuracy, but batch-oriented and heavier on resources. Vosk is a Kaldi-based offline recognizer: it streams in real time, runs on CPU or edge devices, and ships small models, usually at the cost of lower accuracy than the strongest modern encoder-decoder systems. They do the same job, but their architectures are different, which is why they behave differently.

02Can Whisper Do Real-Time Transcription?

Standard open-source Whisper does not stream natively. It processes audio in chunks, so near-realtime Whisper apps usually approximate streaming by repeatedly transcribing short rolling windows. True low-latency transcription needs a system designed for streaming, such as GPT-Realtime-Whisper, a Kaldi/Vosk-style recognizer, or a transducer model like Parakeet.

03What Is WER (Word Error Rate) and What's a Good Score?

WER measures ASR accuracy as (substitutions + deletions + insertions) divided by the total number of reference words; lower is better and 0 is perfect. Below 5% on clean benchmark audio is strong for modern models. Real-world WER is usually higher (accuracy lower) because production audio has accents, noise, and domain-specific vocabulary that the benchmarks don't.

04What Is CTC in Speech Recognition?

CTC (Connectionist Temporal Classification) is a method that lets a model output a label for each slice of audio without knowing in advance how the audio frames line up with the characters. It uses blank tokens and a rule for collapsing repeated outputs to handle alignment, which is what allows CTC models to stream and run efficiently on a CPU.

05Is faster-whisper a Different Model From Whisper?

No. faster-whisper uses the same Whisper weights and runs them through the CTranslate2 inference engine. It's roughly 4x faster for the same accuracy while using less memory, and its INT8 quantization mode further reduces memory usage with negligible accuracy loss. It's a faster way to run Whisper, not a different model with different transcription behavior.