sajilck/whisper-small-malayalam

Whisper Small Malayalam

Fine-tuned version of openai/whisper-small on a multi-corpus Malayalam speech dataset. This is the first publicly available Whisper model fine-tuned specifically for Malayalam ASR.

Model Description

• Base model: openai/whisper-small (244M parameters)
• Language: Malayalam (ml)
• Task: Automatic Speech Recognition (transcription)
• Training steps: 3500
• Best WER: 37.64% on CommonVoice 25 Malayalam test set

Training Data

The model was trained on an aggregated corpus of 5 Malayalam speech datasets, combined and published as sajilck/malayalam-asr-corpus.

Corpus	Source	Domain	Access
IMaSC	thennal/imasc	TTS / Read speech	HuggingFace
SMC Malayalam Speech Corpus	sajilck/smc-malayalam-speech-corpus	Read speech	Kaggle
IndicTTS Malayalam	kavyamanohar/indic-tts-malayalam-speech-corpus	TTS / Read speech	Kaggle
OpenSLR 63	sajilck/openslr63	Crowdsourced	Kaggle
CommonVoice 25 Malayalam	sajilck/common-voice-malayalam	Crowdsourced	Kaggle

Total: ~86,000 samples across TTS-recorded, read speech, and crowdsourced domains.

Benchmark Results

Model	Params	WER ↓	Notes
openai/whisper-small (base)	244M	~85%	No Malayalam fine-tuning
smcproject/Malwhisper-v1-medium	769M	61.84%	Single corpus (IMaSC only)
sajilck/whisper-small-malayalam	244M	37.64%	Multi-corpus fine-tuning

Key advantages over prior work:

• 3× smaller model than Malwhisper-v1-medium, better WER
• 5 corpora vs 1 — better speaker and domain diversity
• Multi-domain training — TTS, read speech, and crowdsourced audio

Limitations

This section is maintained openly and updated as new evaluation work uncovers issues. Last updated after a systematic leakage audit of the training corpus.

Original benchmark was narrow. The initially reported 37.64% WER was measured only on the CommonVoice Malayalam test split — read, studio-quality speech. It was not evaluated against the other four source domains in the training corpus (IMaSC, SMC, IndicTTS, OpenSLR 63) or against broadcast/radio-news-style audio.

The corpus's official train/test split has confirmed leakage. An exact-transcript-hash and fuzzy near-duplicate check between this model's training corpus's train (86,911 rows) and test (4,828 rows) splits found:

• 41.6% of test rows have an exact-duplicate transcript in train
• 44.4% are flagged by fuzzy near-duplicate matching

Any WER computed on the raw test split partly reflects memorization, not generalization.

Leakage is concentrated in the smaller, fixed-script sources. Per-source kept rate after filtering out flagged rows:

Source	Kept %
IMaSC	2.7%
SMC	5.7%
OpenSLR 63	13.9%
CommonVoice	70.7%
IndicTTS	84.6%
Shrutilipi	93.0%

IMaSC and SMC — small corpora with a bounded set of scripted sentences read by a handful of speakers — are almost entirely duplicated across the split. Shrutilipi, despite being scraped at document scale, is the cleanest source by this measure.

The clean held-out set is source-imbalanced. After filtering, 2,684 clean rows remain, of which 87.6% are Shrutilipi. Current held-out evaluation is better powered to measure performance on radio-news-style audio than on read/studio speech.

Quantized (GGML/GGUF) variants show an unexplained WER gap. whisper.cpp f16/q5_0/q8_0 conversions evaluated at 51.7%–54.4% WER on a small sample — notably worse than the CommonVoice figure above. This has not yet been root-caused; contributing factors likely include sample composition (broader source mix) and the leakage above, but quantization-specific quality loss hasn't been ruled out.

What's fixed: a leakage-free held-out set is now used for ongoing evaluation. Still open: source imbalance in that set, and root-causing the GGML gap.

Full writeup with methodology: I Checked My Own ASR Dataset for Leakage — Here's What I Found

Usage

HuggingFace Transformers (Python)

from transformers import pipeline

pipe = pipeline(
    "automatic-speech-recognition",
    model="sajilck/whisper-small-malayalam",
    generate_kwargs={"language": "malayalam", "task": "transcribe"},
)

result = pipe("your_audio.wav")
print(result["text"])

For longer audio files:

pipe = pipeline(
    "automatic-speech-recognition",
    model="sajilck/whisper-small-malayalam",
    generate_kwargs={"language": "malayalam", "task": "transcribe"},
    chunk_length_s=30,
    stride_length_s=5,
)
result = pipe("long_audio.wav")
print(result["text"])

GGML / whisper.cpp — CPU Inference (No GPU Required)

Quantized GGML variants are available for use with whisper.cpp, enabling Malayalam ASR on any laptop or edge device without a GPU.

Variant	File	Size	RTF (30s audio)	Speed	Notes
FP16	ggml-model-f16.bin	487 MB	0.40	2.6× real-time	Best quality
Q5_0	ggml-model-q5_0.bin	175 MB	0.44	2.3× real-time	Smallest size
Q8_0	ggml-model-q8_0.bin	264 MB	0.34	3.0× real-time	✅ Recommended

Benchmarked on Kaggle CPU (4 cores, AVX2). Q8_0 outperforms F16 and Q5_0 due to efficient SIMD integer operations on AVX2 hardware. WER difference between F16 and Q8_0 is only 2.68%, making Q8_0 the best overall choice.

Note: RTF is high for short clips (<10s) due to fixed 30s mel spectrogram encoding overhead. All variants process 30s+ audio faster than real-time.

Quick Start

# 1. Build whisper.cpp
git clone https://github.com/ggml-org/whisper.cpp
cd whisper.cpp
cmake -B build && cmake --build build --config Release

# 2. Download Q8_0 (recommended)
wget https://huggingface.co/sajilck/whisper-small-malayalam/resolve/main/ggml/ggml-model-q8_0.bin

# 3. Transcribe Malayalam audio (16kHz mono WAV)
./build/bin/whisper-cli -m ggml-model-q8_0.bin -l ml -f your_audio.wav

Audio Requirements

Input audio must be 16kHz mono WAV. Convert using ffmpeg:

ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav

Audio Requirements

Input audio must be 16kHz mono WAV. Convert using ffmpeg:

ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav

Audio Requirements

• Format: WAV (16-bit PCM)
• Sample rate: 16 kHz
• Channels: Mono

Convert any audio to the required format:

ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav

Training Details

Parameter	Value
Base model	openai/whisper-small
Training steps	3500
Effective batch size	16 (batch=4, grad_accum=4)
Learning rate	1e-5
Warmup steps	500
Precision	fp16
Hardware	NVIDIA Tesla P100 16GB
Framework	HuggingFace Transformers + Seq2SeqTrainer

Training Data Preprocessing

Audio from all 5 corpora was:

• Resampled to 16kHz mono
• Filtered to 0.5–30 second clips
• Converted to log-mel spectrograms (80 mel bins)
• Tokenized using Whisper's multilingual tokenizer with language token <|ml|>

Limitations

• Trained on read speech and crowdsourced audio — may perform worse on spontaneous conversational Malayalam
• Foreign proper nouns (English names, place names) may be transcribed with Malayalam phonetic approximations
• Performance may vary across Malayalam dialects

Future Work

• v2: Adding Shrutilipi broadcast news corpus with higher learning rate (lr=2e-4) based on findings from Adalat AI Vividh-ASR paper
• whisper-medium-malayalam and whisper-tiny-malayalam variants
• Evaluation on Vividh-ASR benchmark across all 4 speech difficulty tiers

Citation

@misc{sajilck2026whispermalayalam,
  author = {Sajil C.K.},
  title = {Whisper Small Malayalam: Multi-Corpus Fine-Tuning of Whisper for Malayalam ASR},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/sajilck/whisper-small-malayalam}
}

License

This model is released under the Apache 2.0 license, consistent with the base Whisper model. Training corpora retain their individual licenses — please refer to each source dataset for usage terms.