| | --- |
| | license: mit |
| | datasets: |
| | - styletts2-community/multilingual-phonemes-10k-alpha |
| | language: |
| | - fr |
| | - en |
| | - es |
| | - ca |
| | - de |
| | - el |
| | - fa |
| | - fi |
| | - pt |
| | - pl |
| | - ru |
| | - sv |
| | - uk |
| | - zh |
| | --- |
| | |
| | # Multilingual PL-BERT checkpoint |
| |
|
| | The checkpoint open-sourced here is trained by Papercup using the open-source PL-BERT model found here https://github.com/yl4579/PL-BERT. It is trained to be supported by StyleTTS2, which can be found here: https://github.com/yl4579/StyleTTS2. You can see in the model card the languages that it has been trained on (the languages correspond to the crowdsourced dataset found here https://huggingface.co/datasets/styletts2-community/multilingual-phonemes-10k-alpha). |
| |
|
| | Notable differences compared to the default PL-BERT checkpoint and config available [here](https://github.com/yl4579/StyleTTS2/tree/main/Utils/PLBERT): |
| | * Because we are working with many languages, we are using a different tokenizer now: `bert-base-multilingual-cased`. |
| | * The PL-BERT model was trained on the data obtained from `styletts2-community/multilingual-phonemes-10k-alpha` for 1.1M iterations. |
| | * The `token_maps.pkl` file has changed (also open-sourced here). |
| | * We have changed the `util.py` file to deal with an error when loading `new_state_dict["embeddings.position_ids"]`. |
| |
|
| | ## How do I train StyleTTS2 with this new PL-BERT checkpoint? |
| |
|
| | * Simply create a new folder under `Utils` in your StyleTTS2 repository. Call it, for example, `PLBERT_all_languages`. |
| | * Copy paste into it `config.yml`, `step_1100000.t7` and `util.py`. |
| | * Then, in your StyleTTS2 config file, change `PLBERT_dir` to `Utils/PLBERT_all_languages`. You will also need to change your import as such: |
| | * Change `from Utils.PLBERT.util import load_plbert` |
| | * To `from Utils.PLBERT_all_languages.util import load_plbert` |
| | * Alternatively, you can just replace the relevant files in `Utils/PLBERT` and not have to change any code. |
| | * Now, you need to create train and validation files. You will need to use `espeak` to create a file in the same format as the ones that exist in the `Data` folder of the StyleTTS2 repository. Careful! You will need to change the `language` argument to phonemise your text if it's not in English. You can find the correct language codes [here](https://github.com/espeak-ng/espeak-ng/blob/master/docs/languages.md). For example, Latin American Spanish is `es-419` |
| |
|
| | Voila, you can now train a multilingual StyleTTS2 model! |
| |
|
| | Thank you to Aaron (Yinghao) Li for these contributions. |
