Text-to-Speech
coqui

YourTTS trained on CML-TTS

A multilingual, multi-speaker YourTTS model for 8 languages, fine-tuned from the original YourTTS model (tts-hub/yourtts) by the authors of the CML-TTS dataset. It can clone a voice from a few seconds of reference audio (zero-shot) or use one of the speakers seen in training.

This is a copy of freds0/YourTTS-CML-Dataset for use with coqui-tts. The main change is a fix to config.json: the original lists an extra character (U+0329) in characters.phonemes that isn't in the checkpoint's vocabulary, so the model failed to load. Their language codes du, ge and sp for Dutch, German and Spanish are also replaced by the corresponding ISO 639-1 codes here.

Usage

Install coqui-tts:

pip install coqui-tts

Clone a voice from a reference recording:

from TTS.api import TTS

tts = TTS("tts_models/multilingual/cml-tts/your_tts")
tts.tts_to_file(
    text="Olá, este é um teste de síntese de voz.",
    speaker_wav="reference.wav",
    language="pt-br",
    file_path="output.wav",
)

Or use one of the speakers from the training data (see tts.speakers for all names):

tts.tts_to_file(text="...", speaker=tts.speakers[0], language="es", file_path="output.wav")

From the command line:

tts --model_name tts_models/multilingual/cml-tts/your_tts \
    --text "Dit is een test." --language_idx nl \
    --speaker_wav reference.wav --out_path output.wav

Languages

Language Code
Dutch nl
English en
French fr
German de
Italian it
Polish pl
Portuguese (Brazilian) pt-br
Spanish es

Model details

  • Architecture: YourTTS (VITS with language embeddings and external speaker embeddings), 24 kHz output.
  • Input: characters (no phonemizer), normalized with multilingual_cleaners.
  • Speaker conditioning: 512-dimensional d-vectors from the H/ASP speaker encoder included in this repository (model_se.pth, config_se.json). Embeddings of the training speakers are in samples_emb_all_langs.json.
  • Initialization: the original YourTTS checkpoint, trained on English, French and Brazilian Portuguese, then fine-tuned on the data below.

Training data

  • CML-TTS (Dutch, French, German, Italian, Polish, Portuguese, Spanish), derived from Multilingual LibriSpeech.
  • LibriTTS train-clean-360 (English).

Limitations

  • The model reads characters, not phonemes, so pronunciation of rare words, names, numbers and abbreviations can be wrong. Write numbers out as words.
  • The training data is read audiobook speech, so other speaking styles are reproduced less well.
  • Voice similarity depends strongly on the quality of the reference recording; clean speech of a few seconds without background noise works best.

License

The model is released under CC BY-NC-ND 4.0, as the original YourTTS model: non-commercial use only.

Citation

@inproceedings{oliveira2023cmltts,
  title = {{CML-TTS}: A Multilingual Dataset for Speech Synthesis in Low-Resource Languages},
  author = {Oliveira, Frederico S. and Casanova, Edresson and Candido Junior, Arnaldo and Soares, Anderson S. and Galv{\~a}o Filho, Arlindo R.},
  booktitle = {Text, Speech, and Dialogue (TSD)},
  year = {2023},
  eprint = {2306.10097},
  archivePrefix = {arXiv},
}

@inproceedings{casanova2022yourtts,
  title = {{YourTTS}: Towards Zero-Shot Multi-Speaker {TTS} and Zero-Shot Voice Conversion for Everyone},
  author = {Casanova, Edresson and Weber, Julian and Shulby, Christopher D. and Candido Junior, Arnaldo and G{\"o}lge, Eren and Ponti, Moacir A.},
  booktitle = {Proceedings of the 39th International Conference on Machine Learning},
  pages = {2709--2720},
  year = {2022},
}
Downloads last month
69
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for tts-hub/yourtts_cml

Finetuned
(2)
this model

Dataset used to train tts-hub/yourtts_cml

Papers for tts-hub/yourtts_cml