AI transcription for Indian languages: the state of the art

The best openly benchmarked Indian-language speech model transcribes Hindi at 10.3% word error rate and Malayalam at 34.8% — the same model, the same evaluation, a factor of three apart. That spread, not the headline average, is the number that should shape your product. A Hindi grievance-redressal bot and a Malayalam court transcript are not the same engineering problem, and anyone who quotes you one accuracy figure for “Indian languages” has not measured them.

This article collects the published measurements, says which test set each came from, and is explicit about which numbers are ours and which are not.

What the published numbers actually say

The broadest public per-language comparison remains Vistaar (AI4Bharat, INTERSPEECH 2023) — 59 benchmarks assembled from Kathbath, FLEURS, Common Voice, IndicTTS, MUCS and Gramvaani, used to evaluate three open and two commercial systems. Word error rates for IndicWhisper, the fine-tuned model released with it (full tables on GitHub):

Language	Kathbath	Common Voice	FLEURS	IndicTTS	Average
Hindi	10.3	11.4	15.0	7.6	13.6
Kannada	19.3	—	18.6	13.2	18.3
Marathi	19.9	22.8	20.5	11.4	18.2
Urdu	11.9	31.7	19.2	—	19.4
Bengali	16.6	24.7	20.9	18.8	20.1
Punjabi	16.9	22.4	23.1	—	20.5
Gujarati	17.8	—	23.5	19.1	22.8
Tamil	24.2	29.2	25.2	17.2	25.3
Odia	24.7	35.2	32.9	15.0	27.4
Telugu	25.0	—	25.4	33.8	28.8
Malayalam	34.8	44.5	22.6	21.4	32.3

Read the columns before the rows. Hindi on IndicTTS (studio-recorded, read speech) is 7.6%; Hindi on Gramvaani (spontaneous telephone speech from rural callers) is 26.8% — a 19-point swing within one language, larger than the gap between Hindi and Telugu on any single benchmark. Recording conditions dominate language choice. If your deployment is phone-quality audio from a village, the read-speech numbers on a vendor’s slide are not your numbers.

The second table is the one that answers “does specialization pay?” — same Hindi subset, six systems:

System	Kathbath	FLEURS	Common Voice	Gramvaani	Average
Google STT	14.3	19.4	20.8	59.9	23.9
IndicWav2Vec	12.2	18.3	20.2	42.1	21.0
Azure STT	13.6	24.3	14.6	42.3	20.0
NVIDIA Conformer-large	12.7	15.7	21.2	42.6	18.6
IndicWhisper	10.3	11.4	15.0	26.8	13.6

Across the whole Vistaar suite IndicWhisper had the lowest WER in 39 of 59 benchmarks, averaging 4.1 WER points better than the field. The gap is widest exactly where it matters — spontaneous telephone speech, where the commercial APIs lose 16 points to it.

These figures are from 2023, and the ceiling has moved since: AI4Bharat’s IndicVoices (2024) released 7,348 hours from 16,237 speakers across 145 districts and all 22 Eighth Schedule languages, and the models trained on it extend coverage well past Vistaar’s twelve. We cite the 2023 tables because they are the last broad, per-language, multi-system comparison published with reproducible test sets. If someone quotes you a newer number, ask which corpus.

Where Whisper fits

Whisper remains the workhorse for multilingual platforms, and the reason is operational rather than accuracy. Its Indic accuracy trails Indic-specialized models — that is what the table above shows, since IndicWhisper is Whisper fine-tuned on 10,700 hours of Indian audio. What Whisper gives you instead is one model covering roughly 100 languages, open weights you can self-host, robustness to messy audio, and a mature optimization ecosystem (quantization, batching, streaming servers) that specialized models are still building.

Option	Strengths	Limits
Whisper, self-hosted	One model for ~100 languages, open weights, robust to noise, strong tooling	Indic accuracy trails specialized models; hallucinates on long silence if VAD is weak
Indic-specialized open models	Best open-weights accuracy on target languages — 4.1 WER points on average, and far more on spontaneous speech	Narrower language and domain coverage; younger tooling
Commercial Indic APIs	Managed, no infrastructure	Data leaves your infrastructure; per-minute cost at scale; a hard sell under DPDP for government data

Our position after benchmarking: for a multilingual platform — a video system that must caption German, English and Hindi in one deployment — Whisper-class models win on operational simplicity, and you accept a few points of Indic WER for it. For a Hindi-first or Tamil-first high-volume service, the specialized models are worth the narrower stack, and on telephone-grade audio they are worth it decisively.

The real problem: nobody speaks one language

The defining feature of Indian speech is code-switching. Real meetings run in Hinglish — Hindi matrix, English technical vocabulary, mid-sentence switches — and none of the benchmarks above measure it. That is not a footnote; it is the gap between a benchmark and your product.

• Forced language tags hurt. Pinning Whisper to hi makes it render English fragments awkwardly or transliterate them into Devanagari; pinning to en mangles the Hindi. Per-segment auto-detection does better but can flap between languages mid-utterance. The fix is hysteresis rather than a bigger model — hold a detected language once confidence passes a threshold instead of re-deciding on every segment. Tune the holding behaviour on your own audio; the right amount of stickiness depends on how fast your speakers actually switch.
• Script choice is a product decision, and it is yours to make. Should “meeting kal hai” come out in Devanagari, Roman transliteration, or mixed script? Search indexes, translation engines and legal records all behave differently depending on the answer. Decide explicitly, write it down, and normalize consistently — retrofitting a script policy across a corpus of existing transcripts is miserable.
• Evaluate on code-switched test sets. A model’s monolingual Hindi WER tells you almost nothing about its Hinglish behaviour. Build test audio from your actual domain. This is the single highest-value week of work in an Indic ASR project and almost nobody does it.

Evaluation traps we have personally fallen into

• WER is script-sensitive, and your scorer is probably wrong. Word error rate over Indic scripts is fragile: tokenization, matras and normalization differences manufacture phantom errors. We once spent days chasing an “accuracy regression” that was a text-normalization bug in the scorer — ours involved Arabic script, and the lesson generalizes to every non-Latin script. Normalize hypothesis and reference aggressively, and report character error rate alongside WER for Indic scripts.
• Streaming costs you more than model choice does. Measuring both independently on 172 FLEURS clips in our own pipeline, the streaming and preprocessing path cost +2.5 WER points on average while switching between two model sizes cost +0.8. Live captions that underperform your batch transcripts are a chunking and endpointing problem three times out of four. Details in our Whisper scaling write-up.
• Clean benchmarks flatter everyone. See the 19-point Hindi swing between IndicTTS and Gramvaani above. Test on production-realistic captures — phone lines, ceiling mics in a panchayat hall, crosstalk. Preprocessing tricks that shine on synthetic degradation regress on clean speech; we have shipped that mistake and rolled it back.
• Latency is part of accuracy. For live captioning, a slightly worse transcript delivered in real time beats a better one that arrives late. Measure them together or you will optimize one into the ground.

How to read anyone’s accuracy numbers, including ours

Our production captioning deployment is configured for 15 languages, Indian languages among them, and our own end-to-end benchmark work runs on public corpora — German, English, Italian, Turkish, Arabic and Ukrainian among them — so the method is reproducible by anyone with the same open models. The Indic figures in the tables above are AI4Bharat’s, on named public test sets, and we have linked them so you can check them yourself rather than take a vendor’s word for it.

That is the standard worth applying to every vendor you evaluate. The question that separates them is not “what is your accuracy” — everyone has an answer — it is “on which corpus, recorded how, and can I re-run it.”

Deployment: why self-hosting changes the conversation

For Indian government and public-sector use — courts, assemblies, telemedicine, e-governance meeting records — the decisive question is rarely raw WER. It is where does the audio go?

That question now has statutory teeth. The Digital Personal Data Protection Act, 2023 is being operationalised by the DPDP Rules, 2025, notified by MeitY on 14 November 2025 with an eighteen-month phased rollout: the Data Protection Board from November 2025, the consent-manager registration framework from 14 November 2026, and the substantive obligations — consent notices, data-principal rights, breach notification, Significant Data Fiduciary duties — from 14 May 2027 (timeline summary). Citizen speech is personal data; a transcript is a derived record; routing either through a foreign API makes you a data fiduciary explaining a cross-border processing chain to a regulator that now has a board to hear the complaint. Government platforms additionally answer to the Guidelines for Indian Government Websites and Apps (GIGW) and to MeitY cloud-empanelment expectations for where workloads run.

Self-hosted multilingual ASR answers all of it structurally: open-weight models on GPUs in-country, on-premise where required, no external dependency, and an audit trail you own. The economics are not the obstacle people assume — a single dedicated GPU server carries an institutional deployment, and the operational discipline it demands is a known quantity, written up here. The sovereignty argument in full is in self-hosted vs API AI.

We build this as part of our AI & model engineering practice, with public-sector framing on our Public Sector page.

FAQ

Which Indian languages are hardest for AI transcription, and by how much? On AI4Bharat’s Vistaar benchmarks, Hindi averages 13.6% WER and Malayalam 32.3% for the same model — roughly a factor of three. Marathi, Kannada and Urdu sit in the high teens; Tamil, Telugu and Odia in the mid-to-high twenties. Quality tracks available training data, so validate per language and never accept a single “Indian languages” figure.

Is Whisper or an Indic-specialized model better for Indian languages? Specialized models win on accuracy — IndicWhisper had the lowest WER in 39 of 59 Vistaar benchmarks, averaging 4.1 points better, and beat commercial APIs by 15+ points on spontaneous telephone speech. Generic Whisper wins on operational breadth when one deployment must also handle non-Indian languages. Choose by whether your platform is Indic-only.

Does DPDP compliance require Indian-language ASR to run in India? The DPDP framework does not impose blanket localisation, but it makes every cross-border processing chain something you must document, justify and be able to unwind — with substantive obligations landing 14 May 2027. Self-hosting removes the question rather than answering it, which is why government and regulated buyers keep arriving at it.

---

Rajan Jain is the CEO of Vaagmodo and leads our AI engineering from Udaipur, including multilingual live transcription in production. Talk to us about your languages: contact or info@vaagmodo.com.