YourTTS trained on CML-TTS
A multilingual, multi-speaker YourTTS model
for 8 languages, fine-tuned from the original YourTTS model
(tts-hub/yourtts)
by the authors of the CML-TTS dataset. It
can clone a voice from a few seconds of reference audio (zero-shot) or use one
of the speakers seen in training.
This is a copy of
freds0/YourTTS-CML-Dataset
for use with coqui-tts. The main change
is a fix to config.json: the original lists an extra character (U+0329) in
characters.phonemes that isn't in the checkpoint's vocabulary, so the model
failed to load. Their language codes du, ge and sp for Dutch, German and Spanish
are also replaced by the corresponding ISO 639-1 codes here.
Usage
Install coqui-tts:
pip install coqui-tts
Clone a voice from a reference recording:
from TTS.api import TTS
tts = TTS("tts_models/multilingual/cml-tts/your_tts")
tts.tts_to_file(
text="Olá, este é um teste de síntese de voz.",
speaker_wav="reference.wav",
language="pt-br",
file_path="output.wav",
)
Or use one of the speakers from the training data (see tts.speakers for all
names):
tts.tts_to_file(text="...", speaker=tts.speakers[0], language="es", file_path="output.wav")
From the command line:
tts --model_name tts_models/multilingual/cml-tts/your_tts \
--text "Dit is een test." --language_idx nl \
--speaker_wav reference.wav --out_path output.wav
Languages
| Language | Code |
|---|---|
| Dutch | nl |
| English | en |
| French | fr |
| German | de |
| Italian | it |
| Polish | pl |
| Portuguese (Brazilian) | pt-br |
| Spanish | es |
Model details
- Architecture: YourTTS (VITS with language embeddings and external speaker embeddings), 24 kHz output.
- Input: characters (no phonemizer), normalized with
multilingual_cleaners. - Speaker conditioning: 512-dimensional d-vectors from the H/ASP speaker
encoder included in this repository (
model_se.pth,config_se.json). Embeddings of the training speakers are insamples_emb_all_langs.json. - Initialization: the original YourTTS checkpoint, trained on English, French and Brazilian Portuguese, then fine-tuned on the data below.
Training data
- CML-TTS (Dutch, French, German, Italian, Polish, Portuguese, Spanish), derived from Multilingual LibriSpeech.
- LibriTTS train-clean-360 (English).
Limitations
- The model reads characters, not phonemes, so pronunciation of rare words, names, numbers and abbreviations can be wrong. Write numbers out as words.
- The training data is read audiobook speech, so other speaking styles are reproduced less well.
- Voice similarity depends strongly on the quality of the reference recording; clean speech of a few seconds without background noise works best.
License
The model is released under CC BY-NC-ND 4.0, as the original YourTTS model: non-commercial use only.
Citation
@inproceedings{oliveira2023cmltts,
title = {{CML-TTS}: A Multilingual Dataset for Speech Synthesis in Low-Resource Languages},
author = {Oliveira, Frederico S. and Casanova, Edresson and Candido Junior, Arnaldo and Soares, Anderson S. and Galv{\~a}o Filho, Arlindo R.},
booktitle = {Text, Speech, and Dialogue (TSD)},
year = {2023},
eprint = {2306.10097},
archivePrefix = {arXiv},
}
@inproceedings{casanova2022yourtts,
title = {{YourTTS}: Towards Zero-Shot Multi-Speaker {TTS} and Zero-Shot Voice Conversion for Everyone},
author = {Casanova, Edresson and Weber, Julian and Shulby, Christopher D. and Candido Junior, Arnaldo and G{\"o}lge, Eren and Ponti, Moacir A.},
booktitle = {Proceedings of the 39th International Conference on Machine Learning},
pages = {2709--2720},
year = {2022},
}
- Downloads last month
- 69
Model tree for tts-hub/yourtts_cml
Base model
freds0/YourTTS-CML-Dataset