Scaling live transcription: 6× Whisper capacity from a single GPU

Our first honest answer to “how many live Whisper streams fit on one GPU” was 4–6. After a systematic tuning campaign the same class of hardware carried 24 usable live streams — and by the end of it we had established something more useful than the number: the accelerator had never been the constraint. This article is that campaign: how the baseline collapses, which levers moved it, the accuracy the resulting pipeline produces on a public test set, and the two optimizations that turned out to be worthless.

The definition of “usable”

One number governs the whole exercise: median caption delay must stay under 2.5 seconds. Above that, captions stop feeling live and people start reading instead of listening. Every “usable” claim below means median lag under 2.5 s with every stream still reporting — not peak throughput, not an average over a batch job.

Accuracy is measured against FLEURS — Google’s public multilingual speech corpus, human recordings with human reference transcripts. We use it precisely because it is not our audio: it keeps us honest and it lets anyone reproduce the comparison.

The naive setup, and why it stalls at 4–6 streams

The obvious architecture is one Whisper worker per call: audio comes off the media server, gets chunked, each chunk is decoded independently. It works, and it plateaus fast. Our baseline — large-v3 workers, float16, beam 5, no GPU sharing — produced this:

Usable ceiling: 4–6 streams. Note the shape — it does not degrade gracefully, it falls off a cliff between 4 and 8. That cliff is the whole story, and it is not made of matrix multiplications:

• Decoding overhead per stream. Every stream launches its own inference calls. Small, frequent, unbatched requests keep the GPU busy with kernel-launch overhead and idle gaps rather than useful compute.
• Per-stream CPU work. Resampling, voice-activity detection, chunk assembly and feature extraction all run on CPU, unbatched. At a handful of streams this is invisible; past ten, CPU contention delays feeds to the GPU and the GPU starves.
• Transport. Getting audio from the media pipeline to the ASR worker — depacketization, jitter handling, buffering, encryption — adds latency and CPU load that scales linearly with streams.

The uncomfortable finding from our benchmarks: the GPU was not the bottleneck. Utilization graphs showed the accelerator waiting on audio far more often than audio waiting on the accelerator. If you only profile the model, you will optimize the wrong thing.

The levers that actually moved the number

Getting from 4–6 to roughly 24 was not one change. It was model-variant selection, precision and quantization settings, batching and serving configuration, endpointing behaviour, and — decisively — the audio path in front of all of it. Four of those deserve their reasoning. We publish the reasoning rather than a configuration file, because the settings that suit one deployment’s latency budget and language mix are rarely the settings that suit another’s.

1. Turbo beats large-v3, and it isn’t close

large-v3-turbo keeps the full encoder and prunes the decoder to four layers (release notes). For live captioning — short utterances, continuous decoding — decoder depth is where your latency lives. In our runs the full large-v3 configuration blew the 2.5 s budget at a single stream, while turbo stayed comfortably inside it at eight. Turbo wins on latency and, because of the next point, gives up almost nothing on quality. If you are building live captions on large-v3 because it is the biggest, measure the pruned decoder before you buy hardware to compensate.

2. Beam 5 is nearly free — so stop apologizing for it

This is the finding that surprised us most. On a pruned decoder, beam search barely registers: measured against greedy decoding at fixed quality, beam 5 costs about 4% of decode throughput — because there are only four decoder layers to run it over.

And beam 1 is not free on the other side of the ledger. On FLEURS, dropping to greedy decoding cost us 2.1 WER points in Turkish and 1.5 points in Arabic. On large-v3 the same switch was catastrophic: Turkish went from 9.8% to 31.0% WER. Greedy decoding degrades exactly the segments that matter — proper nouns, numbers, code-switched phrases, morphologically rich languages. On a turbo-class decoder, use beam search; the received wisdom that it is a luxury comes from the era of deep decoders.

3. Quantization is not automatically faster — measure it on your card

Everyone repeats “quantize first, it’s free.” On one of the cards we tested, int8_float16 was slower than float16 for large-v3 while being a clear win for turbo. Quantization is a property of the model/kernel/card combination, not a law. Benchmark it on the silicon you will actually run. We nearly shipped a slower configuration on received wisdom.

4. Endpointing is a tuning surface, not preprocessing trivia

The voice-activity detector that decides when an utterance has ended shapes the reading experience more than the model does. We swept it systematically — thresholds, end-silence, minimum-speech and hangover variants — across a fixed clip set, thousands of runs.

The lesson generalizes even though the right values do not. End-silence does more for readability than any other setting in the pipeline. Set it short and you emit shredded fragments: many finals per minute, a handful of words each, and a large fraction of one- and two-word “stubs” that read as noise. Lengthen it and the same audio comes out as whole sentences with an order-of-magnitude drop in stub rate, in exchange for a fraction of a second of extra lag. In our sweep, moving from the shortest to the longest setting we tested cut the stub rate roughly eight-fold for under a quarter-second of added latency. Almost nobody tunes this, and everybody should — on their own audio, because conversational rhythm differs by language, domain and microphone.

One trap worth naming: there are two VAD stages in a faster-whisper-style pipeline and they are not the same knob — the endpointing VAD that decides when an utterance ended, and the decoder’s internal vad_filter that trims silence inside the audio it is given (Silero VAD is a common choice for the first). Conflating them produces confusing results for days.

The serving side is the same story. faster-whisper on a CTranslate2 build, multiple model copies sharing one card, and a batching scheduler with an explicit latency budget together change the throughput class of the same hardware. None of that is exotic; all of it has to be measured rather than assumed.

The finding that reframed the project: it was never the model

Here is what the campaign actually taught us. Same engine, same model, same card — the largest single improvement in caption latency at high concurrency came from changing how audio reached the model, not from anything inside it. Not one model parameter moved.

That generalizes into the rule we now apply on every AI-serving engagement: profile the whole path. Our ceilings were repeatedly in transport and in unbatched CPU preprocessing, and never in the accelerator. Measured purely as decode throughput, the same hardware sustains many times the number of streams it carries live; the entire gap is CPU and audio delivery. If you are sizing a deployment, size the media path and the client fleet, not just the accelerator — and be suspicious of any capacity plan that was derived from a single-stream benchmark.

Accuracy: what the resulting pipeline actually produces

Word error rate through the full live path — resample, endpointing, streaming decode, per-utterance finals — on FLEURS audio, against the same clips decoded in batch:

Two things a reader should take from this. First, live streaming is not free: isolating the causes across 172 FLEURS clips, the streaming and preprocessing path costs +2.5 WER points on average, while the model choice (turbo vs large-v3) costs only +0.8. The pipeline hurts you three times as much as the model does. If your live captions are worse than your batch transcripts, do not go shopping for a bigger model.

Second, Arabic is the outlier, and it is where the streaming penalty concentrates. Re-decoding the full utterance once it ends recovers 1.6 points (19.3% → 17.7%), about 38% of the streaming gap, at the cost of a second decode pass. Widening the rolling context window changed nothing at all — a useful negative result, since it is the first knob most teams reach for.

Note what this table is: our own measurement, on a public corpus, through a live path — reproducible in principle by anyone with the same open models. Client audio never enters our benchmarks, which is why these figures can be published at all.

Pitfalls we hit so you don’t have to

• Don’t normalize your way to regressions. We tried input loudness normalization to rescue quiet speakers. It helped synthetic quiet-audio tests and hurt clean production audio. Benchmark on production-realistic captures, not lab constructions.
• Watch the languages you don’t speak. Scoring pipelines mis-handle some scripts — Arabic normalization tripped ours and manufactured a regression that did not exist. A “regression” in your metrics may be a bug in your metric.
• Check VAD input shapes when you swap implementations. Dimensional mismatches fail quietly with garbage output rather than errors.
• Load-test the transport path, not the model. Our first concurrency ceiling was in audio delivery. Test the whole pipe — media server to caption on screen — under concurrent load, and generate that load from more than one machine or you will measure your load generator.

What this means for capacity planning

Tuned this way, a single dedicated GPU comfortably carries a bank’s video-advisory floor or a telehealth service — self-hosted, across 15 languages including Indian languages, with audio never leaving infrastructure the operator controls. That is the shape of the number that matters commercially: not peak throughput on a datasheet, but concurrent live streams at a latency people will accept.

Two planning consequences follow. Your accelerator is unlikely to be the first thing you run out of, so budget CPU and network capacity on the audio path with the same care. And the capacity figure you should plan against is one you measured on your own traffic shape — ours moved by a factor of four to six without the hardware changing at all, which is exactly how much a published number can mislead you.

The reliability side of running this in production — driver discipline, observability, failure modes — is in our guide to operating AI workloads, and the sovereignty argument is set out in self-hosted vs API AI for banks.

We run this stack as part of our AI & model engineering work, on infrastructure we operate ourselves — the operational side is in cloud, DevOps & managed operations and the AI infrastructure case study.

FAQ

Is the GPU the bottleneck for real-time Whisper transcription? Almost never. Across our benchmarking the accelerator sat waiting on audio far more often than the reverse, and the same hardware sustains many times more streams measured as raw decode throughput than it carries live. The limits are audio transport and unbatched per-stream CPU work. Profile the full path from media server to caption on screen before buying more GPUs.

Does beam search cost too much for live captioning? No — not with a pruned decoder. On large-v3-turbo beam 5 costs roughly 4% of decode throughput versus greedy, while greedy cost us 2.1 WER points in Turkish and 1.5 in Arabic on FLEURS. On full large-v3 the same switch took Turkish from 9.8% to 31.0% WER. Beam width is a cheap quality lever on turbo and an expensive one on large.

Why are my live captions worse than my batch transcripts of the same audio? Because streaming, not model size, is usually the cost. Isolating both on the same 172 FLEURS clips, our streaming and preprocessing path cost +2.5 WER on average while the model difference cost +0.8. Fix chunking, endpointing and resampling before changing models.

Rajan Jain is the CEO of Vaagmodo and leads our AI and infrastructure engineering, including the self-hosted transcription stack behind our live video platform work. Talk to us about your pipeline: contact or info@vaagmodo.com.