sajilck/whisper-small-malayalam

Whisper Small Malayalam

Fine-tuned version of openai/whisper-small on a multi-corpus Malayalam speech dataset.

A newer model is available: sajilck/whisper-small-malayalam-v2 scores 31.2% WER / 4.0% CER on the same leakage-free test set, against 43.5% / 7.3% for this model.

Correction (October 2026): earlier versions of this card reported 47.8% WER / 13.6% CER on the leakage-free set. That evaluation capped transcripts at 225 tokens, which cut off clips longer than about 9 seconds (Malayalam needs ~25 Whisper tokens per second). With the cap raised to Whisper's limit, the correct figures are 43.5% WER / 7.3% CER. All numbers below are corrected.

Model Description

• Base model: openai/whisper-small (244M parameters)
• Language: Malayalam (ml)
• Task: Automatic Speech Recognition (transcription)
• Training steps: 3500
• Best WER: 37.64% on CommonVoice 25 Malayalam test set
• Leakage-free multi-source test: 43.5% WER / 7.3% CER (231 clips across 6 sources, none with a transcript in the training split — see Multi-source evaluation)
• CPU speed (Transformers, FP32, 4 vCPU): RTF 1.96, i.e. slower than real time. The whisper.cpp Q8_0 build matches FP32 accuracy (43.2% WER / 7.4% CER) in about half the memory, but needs a one-line patch for clips longer than ~8 s

Training Data

The model was trained on an aggregated corpus of 6 Malayalam speech datasets, combined and published as sajilck/malayalam-asr-corpus.

Corpus	Source	Domain	Access
IMaSC	thennal/imasc	TTS / Read speech	HuggingFace
SMC Malayalam Speech Corpus	sajilck/smc-malayalam-speech-corpus	Read speech	Kaggle
IndicTTS Malayalam	kavyamanohar/indic-tts-malayalam-speech-corpus	TTS / Read speech	Kaggle
OpenSLR 63	sajilck/openslr63	Crowdsourced	Kaggle
CommonVoice 25 Malayalam	sajilck/common-voice-malayalam	Crowdsourced	Kaggle
Shrutilipi	amithm3/shrutilipi	Broadcast news (All India Radio)	HuggingFace

Total: ~86,900 training samples across TTS-recorded, read speech, crowdsourced and broadcast-news domains. Shrutilipi is the largest single source.

Benchmark Results

Model	Params	WER ↓	Notes
openai/whisper-small (base)	244M	~85%	No Malayalam fine-tuning
smcproject/Malwhisper-v1-medium	769M	61.84%	Single corpus (IMaSC only)
sajilck/whisper-small-malayalam	244M	37.64%	Multi-corpus fine-tuning

Key advantages over prior work:

• 3× smaller model than Malwhisper-v1-medium, better WER
• 6 corpora vs 1 — better speaker and domain diversity
• Multi-domain training — TTS, read speech, crowdsourced and broadcast-news audio

Multi-source evaluation

The CommonVoice figure above covers one domain. To see how the model does across all domains, it was scored on the corpus's leakage-free held-out set: the 2,684 test rows whose transcript does not appear, exactly or as a near-duplicate, in the train split (see Limitations). Up to 50 clips per source were drawn at random (231 clips, 18.6 min of audio; some sources have fewer clean rows than that) and decoded with FP32 greedy decoding, up to 444 new tokens per clip (no clip hit the limit). Evaluated October 2026 with transformers 4.46.3.

Metric	Result
WER, normalised (95% bootstrap CI)	43.5% (39.9% – 47.0%)
WER, raw	44.8%
CER, normalised	7.3%
Substitutions / deletions / insertions	33.3% / 5.1% / 5.1%
Utterances fully correct	15.2% (20.3% if word boundaries are ignored)
Empty outputs / repetition loops	0 / 0

Source	Clips	WER	CER
IndicTTS	50	33.1%	4.6%
IMaSC	44	39.5%	3.9%
OpenSLR 63	32	41.0%	6.1%
Shrutilipi	50	48.2%	9.4%
CommonVoice	50	56.0%	13.2%
SMC	5	35.0%	3.9%

SMC has only 5 clean test rows, so its score is not meaningful on its own.

Normalisation steps: Unicode NFC, old-style chillu (consonant + virama + ZWJ) converted to atomic chillu, zero-width joiners removed, and punctuation stripped.

How to read these numbers

• The headline is source-balanced. It weights each source roughly equally. The clean held-out set itself is 88% Shrutilipi; weighted by that mix, WER would be about 48% (an estimate from the per-source scores).
• Clean vs leaked rows. On 60 test rows whose transcripts do appear in training, WER was 32.5%, against 43.5% on clean rows. Part of that gap is memorisation, but part is source mix: the leaked rows are mostly IMaSC, which is easier read speech. The like-for-like comparison is IMaSC on its own, which went from 23.9% on the leaky split to 39.5% on clean rows.
• WER is high partly because Malayalam words are long. One wrong character makes a whole agglutinated word wrong, so CER (7.3%) is the better guide to how much of the speech is recognised. Word-boundary differences (മുഖ്യമന്ത്രി vs മുഖ്യ മന്ത്രി) account for only a small share: ignoring spaces raises fully-correct utterances from 15.2% to 20.3%. Most errors are real character-level mistakes, such as കുടുംബം → ഉടുംബം.
• Crowdsourced phone audio is the weakest domain. CommonVoice has the highest WER (56.0%) and CER (13.2%), with the most varied voices and microphones. Broadcast news (Shrutilipi) comes next at 48.2% / 9.4%.
• The 56.0% CommonVoice figure does not replace 37.64%. It comes from 50 CommonVoice-sourced rows of this corpus's own split, with a wide confidence interval and different normalisation. It is not the official CommonVoice 25 test set.

An earlier run (September 2026, 300 clips) drew from the full test split, including leaked rows, and scored 42.9% WER / 12.3% CER. That run also used the 225-token cap. Those numbers are superseded by the leakage-free results above.

Limitations

This section is maintained openly and updated as new evaluation work uncovers issues. Last updated after the leakage-free re-evaluation (October 2026).

Original benchmark was narrow. The initially reported 37.64% WER was measured only on the CommonVoice Malayalam test split — read sentences recorded by volunteers on their own phones and microphones. It was not evaluated against the other five source domains in the training corpus (IMaSC, SMC, IndicTTS, OpenSLR 63, Shrutilipi).

The corpus's official train/test split has confirmed leakage. An exact-transcript-hash and fuzzy near-duplicate check between this model's training corpus's train (86,911 rows) and test (4,828 rows) splits found:

• 41.6% of test rows have an exact-duplicate transcript in train
• 44.4% are flagged by fuzzy near-duplicate matching

Any WER computed on the raw test split partly reflects memorization, not generalization. On the leakage-free rows the model scores 43.5% WER; IMaSC alone moves from 23.9% on the leaky split to 39.5% on clean rows.

Leakage is concentrated in the smaller, fixed-script sources. Per-source kept rate after filtering out flagged rows:

Source	Kept %
IMaSC	2.7%
SMC	5.7%
OpenSLR 63	13.9%
CommonVoice	70.7%
IndicTTS	84.6%
Shrutilipi	93.0%

IMaSC and SMC — small corpora with a bounded set of scripted sentences read by a handful of speakers — are almost entirely duplicated across the split. Shrutilipi, despite being scraped at document scale, is the cleanest source by this measure.

The clean held-out set is source-imbalanced. After filtering, 2,684 clean rows remain, of which 87.6% are Shrutilipi. Only 44 IMaSC, 32 OpenSLR 63 and 5 SMC rows survive, so per-source scores for those sources have wide error bars. The headline evaluation samples sources evenly to avoid being a Shrutilipi-only number.

Stock whisper.cpp cuts off long Malayalam clips. whisper.cpp stops decoding each 30 s window after 220 tokens. Malayalam needs about 27 Whisper tokens per second, so anything longer than ~8 s is silently truncated. On the 231 leakage-free clips, stock whisper.cpp Q8_0 scored 47.9% WER / 14.0% CER; with the cap raised to 440 it scored 43.2% / 7.4%, the same as FP32 Transformers (43.5% / 7.3%). The 32 clips over 220 reference tokens account for the whole gap; shorter clips score the same either way. Earlier versions of this card reported 51.7%–54.4% WER for the GGML files on a 20-clip sample; those runs used the stock cap and are superseded. Quantization itself costs little: F16 and Q8_0 match FP32, and Q5_0 adds about 1.5 WER points. See GGML / whisper.cpp for the patch.

What's fixed: a leakage-free held-out set exists, and the multi-source evaluation has been re-run on it (43.5% WER / 7.3% CER), with the transcript-length cap corrected. v2 addresses the training-side issues. Still open: source imbalance in that set (too few clean read-speech rows), and checking the train split for internal duplicates.

General limitations

• Trained on read speech and crowdsourced audio, so it may perform worse on spontaneous, conversational Malayalam
• Foreign proper nouns (English names, place names) may be transcribed with Malayalam phonetic approximations
• Performance may vary across Malayalam dialects
• Not real-time on CPU with plain Transformers (see CPU inference benchmark)

Full writeups with methodology: Part 1: the leakage audit · Part 2: re-testing on clean data · Part 3: v2 and the evaluation bug

Usage

HuggingFace Transformers (Python)

from transformers import pipeline

pipe = pipeline(
    "automatic-speech-recognition",
    model="sajilck/whisper-small-malayalam",
    generate_kwargs={"language": "malayalam", "task": "transcribe"},
)

result = pipe("your_audio.wav")
print(result["text"])

For longer audio files:

pipe = pipeline(
    "automatic-speech-recognition",
    model="sajilck/whisper-small-malayalam",
    generate_kwargs={"language": "malayalam", "task": "transcribe"},
    chunk_length_s=30,
    stride_length_s=5,
)
result = pipe("long_audio.wav")
print(result["text"])

CPU Inference Benchmark (Transformers, no GPU)

This measures the model on CPU through plain HuggingFace Transformers with default settings. It is a baseline, not an optimised setup.

Setting	Value
Hardware	Kaggle CPU: Intel Xeon @ 2.20 GHz, 4 vCPU (2 physical cores), 33.7 GB RAM
Software	PyTorch 2.10 (CPU), Transformers 5.0
Decoding	FP32, greedy, batch size 1, max_new_tokens=225
Evaluation data	300 clips (average 4.8 s long) from the corpus test split, September 2026

Metric	Result
Real-time factor (RTF, all clips together)	1.96 (0.51× real time)
RTF median / p90 per clip	2.05 / 2.97
Latency per clip, p50 / p90 / max	9.1 s / 14.1 s / 15.9 s
Cold start (first clip, 3.5 s long)	4.5 s
Decoding speed	13 tokens/s (77 ms per token)
Model load time	13.6 s
Peak RAM	~3.9 GB

This run used max_new_tokens=225, which cut off clips longer than about 9 s, so times for those clips are slightly understated. Most clips are shorter than that.

RTF = processing time ÷ audio duration. Values above 1 mean slower than real time. The October leakage-free run on the same hardware gave consistent figures (RTF 2.17, p90 latency 15.3 s, 12.2 tokens/s).

Where the time goes. Feature extraction takes about 0.1% of the time; almost all of it is the decoder generating text one token at a time. Whisper's tokenizer splits Malayalam script into many small pieces: this run produced about 120 tokens per clip, roughly 25 tokens per second of audio. Speed on CPU is therefore limited mainly by transcript length in tokens, not by audio length.

Clip length	Clips	RTF	p90 latency
< 3 s	67	2.60	8.3 s
3–6 s	169	2.09	12.1 s
6–10 s	54	1.72	14.7 s
10–15 s	9	1.16	14.9 s

Settings for faster inference (24-clip subset; compare these rows with each other only)

Setting	WER	RTF	p90 latency	Size
FP32, greedy (default)	51.1%	1.80	14.5 s	967 MB
INT8 dynamic quantisation (torch.ao), greedy	56.7%	1.40	11.4 s	413 MB
FP32, beam search (5 beams)	51.7%	4.80	41.1 s	967 MB

CPU threads	RTF	Speed-up vs 1 thread
1	3.18	1.00×
2	1.83	1.73×
4	1.81	1.76×

• Use greedy decoding. Beam search with 5 beams was about 2.7× slower and no more accurate.
• Set threads to the number of physical cores. Hyperthreads added almost nothing (2 → 4 threads: +1%).
• Real-time streaming isn't feasible with this setup. With 6 s windows and a 4 s hop, p90 latency per window was 12.9 s, about 3.2× over budget. The whisper.cpp builds below are not faster on this CPU, but use about half the memory.

Evaluation notebooks: whisper-malayalam-cpu-eval.ipynb (speed, September 2026) and whisper-malayalam-clean-eval.ipynb (leakage-free accuracy, October 2026), both Kaggle CPU only.

GGML / whisper.cpp — CPU Inference (No GPU Required)

Quantized GGML variants are available for use with whisper.cpp, enabling Malayalam ASR on any laptop or edge device without a GPU.

Important: stock whisper.cpp stops after 220 tokens per 30 s window, which cuts off Malayalam speech longer than about 8 seconds. Apply the one-line patch in the Quick Start below. The PyPI pywhispercpp wheel has the same limit.

Scored on the same 231 leakage-free clips as the multi-source evaluation, with the token cap raised to 440:

Variant	File	Size	WER (95% CI)	CER	RTF
Transformers FP32 (reference)	—	967 MB	43.5% (39.9–47.0)	7.3%	2.17
F16	ggml-model-f16.bin	488 MB	43.5% (40.0–47.0)	7.3%	2.58
Q8_0 ✅ recommended	ggml-model-q8_0.bin	264 MB	43.2% (39.7–46.6)	7.4%	2.13
Q5_0	ggml-model-q5_0.bin	175 MB	45.0% (41.8–48.5)	7.6%	2.72
Q8_0, stock whisper.cpp (220-token cap)	ggml-model-q8_0.bin	264 MB	47.9% (44.1–51.6)	14.0%	2.09

• Q8_0 is the best choice. It matches FP32 accuracy and is the fastest of the three files. Q5_0 is the smallest but costs about 1.5 WER points and is slower here.
• Speed: on this CPU none of the builds runs in real time (RTF above 1). whisper.cpp encodes a full 30 s window even for a 5 s clip, so short clips are relatively slow. Peak memory was about 2.2 GB per process, against ~3.9 GB for Transformers.
• Older figures removed. Earlier versions of this card listed RTF 0.34 ("3× real time") for Q8_0 on 30 s audio and a 2.68% WER gap between F16 and Q8_0. Both came from stock whisper.cpp: 30 s of Malayalam needs roughly 800 tokens, so stopping at 220 skipped most of the decoding work.

Setup: Kaggle CPU, Intel Xeon @ 2.20 GHz, 2 physical cores / 4 threads; pywhispercpp 1.5.1 (whisper.cpp f24588a); greedy decoding, no timestamps, no temperature fallback; October 2026. Notebook: whisper-malayalam-v1-ggml-tokencap-fix.ipynb.

Quick Start

# 1. Build whisper.cpp with the token cap raised from 220 to 440
git clone https://github.com/ggml-org/whisper.cpp
cd whisper.cpp
sed -i 's|n_max = whisper_n_text_ctx(ctx)/2 - 4;|n_max = whisper_n_text_ctx(ctx) - 8;|' src/whisper.cpp
grep -c "n_max = whisper_n_text_ctx(ctx) - 8;" src/whisper.cpp   # must print 1
cmake -B build && cmake --build build --config Release

# 2. Download Q8_0 (recommended)
wget https://huggingface.co/sajilck/whisper-small-malayalam/resolve/main/ggml/ggml-model-q8_0.bin

# 3. Transcribe Malayalam audio (16kHz mono WAV)
./build/bin/whisper-cli -m ggml-model-q8_0.bin -l ml -f your_audio.wav

Audio Requirements

• Format: WAV (16-bit PCM)
• Sample rate: 16 kHz
• Channels: Mono

Convert any audio to the required format:

ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav

Training Details

Parameter	Value
Base model	openai/whisper-small
Training steps	3500
Effective batch size	16 (batch=4, grad_accum=4)
Learning rate	1e-5
Warmup steps	500
Precision	fp16
Hardware	NVIDIA Tesla P100 16GB
Framework	HuggingFace Transformers + Seq2SeqTrainer

Training Data Preprocessing

Audio from all 6 corpora was:

• Resampled to 16kHz mono
• Filtered to 0.5–30 second clips
• Converted to log-mel spectrograms (80 mel bins)
• Tokenized using Whisper's multilingual tokenizer with language token <|ml|>

Future Work

• v2: retrain with a higher learning rate, length filtering and de-duplicated training data Done: whisper-small-malayalam-v2 (lr 1e-4, following the Adalat AI Vividh-ASR paper), 31.2% WER / 4.0% CER on the same leakage-free set
• whisper-medium-malayalam and whisper-tiny-malayalam variants
• Evaluation on Vividh-ASR benchmark across all 4 speech difficulty tiers
• Expand the clean held-out set for read-speech sources (IMaSC, SMC, OpenSLR 63)
• Faster CPU inference for real-time streaming: benchmark CTranslate2 / faster-whisper INT8 and whisper.cpp on the same clips, targeting a 6 s window in under 4 s (v2's whisper.cpp build already runs in a CPU-only voice assistant demo)

Citation

@misc{sajilck2026whispermalayalam,
  author = {Sajil C.K.},
  title = {Whisper Small Malayalam: Multi-Corpus Fine-Tuning of Whisper for Malayalam ASR},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/sajilck/whisper-small-malayalam}
}

License

This model is released under the Apache 2.0 license, consistent with the base Whisper model. Training corpora retain their individual licenses — please refer to each source dataset for usage terms.