sajilck/mms-tts-malayalam-sajil

MMS-TTS Malayalam: Sajil C.K.'s voice

A Malayalam text-to-speech model in the voice of Sajil C.K., created by fine-tuning facebook/mms-tts-mal (VITS, 16 kHz) on recordings from his Malayalam ML/Data Science YouTube lectures.

It needs no reference clip, runs about 4× faster than real time on 2 CPU threads, and keeps the base model's pronunciation while sounding like the speaker (speaker similarity 0.87 → 0.99).

Usage

import torch, scipy.io.wavfile
from transformers import VitsModel, AutoTokenizer

repo = "sajilck/mms-tts-malayalam-sajil"
model = VitsModel.from_pretrained(repo).eval()
tokenizer = AutoTokenizer.from_pretrained(repo)

model.speaking_rate = 1.2   # the frozen duration predictor speaks ~20% slower than the speaker; 1.2 matches his pace

text = "നമസ്കാരം, ഇന്നത്തെ വീഡിയോയിൽ നമ്മൾ ഗ്രേഡിയന്റ് ഡിസന്റ് പഠിക്കാം."
inputs = tokenizer(text, return_tensors="pt")
torch.manual_seed(0)        # VITS sampling is stochastic; fix the seed for repeatable output
with torch.no_grad():
    wav = model(**inputs).waveform[0].numpy()
scipy.io.wavfile.write("out.wav", model.config.sampling_rate, wav)

Normalise the text before synthesis. The tokenizer silently drops characters that aren't in its 84-symbol vocabulary, which shows up as skipped or garbled words.

• Use atomic chillu (ൻ ർ ൽ ൾ ൺ), not consonant + virama + ZWJ, and remove ZWJ/ZWNJ.
• Spell out numbers in Malayalam words. Digits 6–9 aren't in the vocabulary, and 0–5 were never trained.
• Write English terms in Malayalam script (for example ഗ്രേഡിയന്റ് ഡിസന്റ്). Latin letters aren't supported.

import re, unicodedata
CHILLU = {"ണ": "ൺ", "ന": "ൻ", "ര": "ർ", "ല": "ൽ", "ള": "ൾ"}
def normalize(t):
    t = unicodedata.normalize("NFC", t)
    for base, atomic in CHILLU.items():
        t = t.replace(base + "്‍", atomic)
    return re.sub(r"\s+", " ", t.replace("‍", "").replace("‌", "")).strip()

Training data

• Source: the author's own YouTube lecture audio (Malayalam, with English technical terms), cut into clips of 1–15 s. The transcripts come from YouTube captions.
• Cleaning:
  • Audio converted to mono 16 kHz, with leading and trailing silence trimmed.
  • Clips removed if their text had digits, Latin letters or characters outside the vocabulary, or if their speaking rate was an outlier.
  • This left 13.8 h in 6,813 training clips.
• Transcript filter: each clip was transcribed with sajilck/whisper-small-malayalam and kept only if the caption agreed with the ASR output (CER ≤ 0.3). This left 1,268 clips (2.7 h). The caption transcripts are noisy: the median clip scored CER 0.40 against the ASR. A first run trained on all 13.8 h learned the voice but lost pronunciation, with test CER going from 0.18 to 0.82.

Training procedure

• Base: facebook/mms-tts-mal, with the official MMS discriminator converted for fine-tuning using ylacombe/finetune-hf-vits.
• Frozen: the text encoder and the stochastic duration predictor. Only the flow, posterior encoder and HiFi-GAN decoder were trained (28.7M trainable parameters, 7.6M frozen). This keeps MMS's Malayalam pronunciation and timing while the model learns the speaker's voice.
• Hyperparameters:
  • 3,000 steps with batch size 16 and learning rate 2e-5
  • AdamW with betas (0.8, 0.99) and fp16
  • loss weights: mel 35, KL 1.5, duration 1, generator 1, feature matching 1, discriminator 3
• Compute: one Kaggle GPU (P100/T4) for about 3.3 h of training.

Evaluation

Both sets were scored with the same ASR model as judge. The first is 41 held-out clips, filtered the same way as the training data. The second is a 21-sentence challenge set covering loanwords, spelled-out numbers, long sentences, questions, short phrases and rare letters. CER and WER are corpus-level on normalised text, and the confidence interval is a 95% bootstrap. Speaker similarity is the WavLM x-vector cosine to the centroid of the speaker's real clips. RTF is measured on CPU with 2 threads.

system	test CER (95% CI)	test WER	challenge CER	speaker sim.	RTF (CPU)
speaker's real recordings	0.256 (0.244–0.269)	0.375	–	0.975*	–
base facebook/mms-tts-mal	0.184 (0.158–0.215)	0.262	0.144	0.866	0.32
this model (step 3000)	0.203 (0.179–0.226)	0.302	0.200	0.986	0.26

* leave-one-out similarity of real clips to each other.

Pronunciation stays close to the base model: the confidence intervals overlap on the test set. The voice is clearly the speaker's. The synthetic speech scores a lower CER than the real recordings, partly because the real clips are spontaneous lecture speech and partly because the captions are noisy.

Limitations

• 16 kHz output. Speech is less crisp than 22–24 kHz models.
• Coverage gaps. Five letters (ഃ ഊ ഘ ഛ ഢ) never occur in the training text, so the model falls back to the base model's rendering. Loanwords are the weakest category (challenge CER about 0.25–0.32).
• Very short inputs such as single words ("നന്ദി.") are unreliable, as they are in the base model.
• Pacing. The duration predictor is frozen, so rhythm follows MMS rather than the speaker. Use speaking_rate to adjust it.
• Evaluation caveats. The test set is small, and it was filtered by the same ASR model that judges it. Treat the numbers as indicative. The UTMOS score (2.3, versus 2.55 for real recordings) comes from an English-trained predictor that mostly reflects the recording conditions.

Intended use and ethics

This is the author's own voice, released for education, research and his own content, such as narration for his Malayalam ML lectures. Don't use it to impersonate the author or to make him appear to say things he didn't say. Label synthetic audio as synthetic where it could be mistaken for a real recording. The model inherits the CC-BY-NC 4.0 licence of the MMS weights, so commercial use is not permitted.

Related models

• sajilck/whisper-small-malayalam: Malayalam ASR, used here to filter data and score intelligibility
• sajilck/f5-tts-malayalam-v2: F5-TTS voice clone trained on the same channel audio

Citation

@article{pratap2023mms,
  title   = {Scaling Speech Technology to 1,000+ Languages},
  author  = {Pratap, Vineel and Tjandra, Andros and Shi, Bowen and others},
  journal = {arXiv preprint arXiv:2305.13516},
  year    = {2023}
}
@inproceedings{kim2021vits,
  title     = {Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech},
  author    = {Kim, Jaehyeon and Kong, Jungil and Son, Juhee},
  booktitle = {ICML},
  year      = {2021}
}

Author: Sajil C.K. (GitHub, intuitivetutorial.com)