Capacitron

[a-froghyar]

---

Disclaimer

@Edresson and @WeberJulian have asked me to open a PR with my newly implemented model, Capacitron. There are still some things to clean up and some fixes to do, however I'm opening this PR already for the discussion around it. Contributions for the list below are welcome, however I'll be slow in the next 10 days with checking them. I have just merged the new dev branch into this branch with the new Coqpit refactor among others and I haven't had the chance to try to run the training yet. I've implemented this model as my master thesis and I'm currently still training different models for the thesis, so I won't have too much capacity (hehe) to experiment in the next month.

---

This PR implements a new model into 🐸 TTS based on the Capacitron model from Google. It's a partial implementation of the models detailed in the paper, hierarchical latent embeddings are still to be done - this is a TODO for later. If you'd like to get an idea what the model does and how it works, here's a post I did a few weeks ago. Audio samples to follow shortly.

Phonemes

I haven't followed the entire discussion about the phoneme issue, however I'm aware that because of license issues, espeak needed to be dropped. @Edresson mentioned that I could try to train the model with graphemes as well, however I haven't had time to do this yet. According to the paper, however, Capacitron does need phoneme inputs to work.

TODOs (List is still to be updated):

• Implement Coqpit config instead of capacitron_blizzad.json
• Fix changes coming from formatting
• Fix capacitron multispeaker training -- (HInt: tacotron_abstract.py)
• Refactor the parameter splitting in train_tacotron.py and training.py
• Create test for capacitron_layers.py
• Fix tests
• Refactor capacitron_eval.py into a notebook
• Fix Continue Training yields broken model training because of capacitron_beta a-froghyar/Capacitron#8
• Fix LR Scheduler Bug when continuing training a-froghyar/Capacitron#4
• Fix Eval Test during training not recognising reference wav/reference text a-froghyar/Capacitron#5

To Discuss

• Some of the losses that are normally activated in Tacotron are deactivated in Capacitron - during my experiments I stripped down the loss calculation to the bare minimum (which is already complicated enough) and I haven't experimented with turning them back on again since then
• DDC + Capacitron

[Edresson]

Disclaimer

@Edresson and @WeberJulian have asked me to open a PR with my newly implemented model, Capacitron. There are still some things to clean up and some fixes to do, however I'm opening this PR already for the discussion around it. Contributions for the list below are welcome, however I'll be slow in the next 10 days with checking them. I have just merged the new dev branch into this branch with the new Coqpit refactor among others and I haven't had the chance to try to run the training yet. I've implemented this model as my master thesis and I'm currently still training different models for the thesis, so I won't have too much capacity (hehe) to experiment in the next month.

This PR implements a new model into TTS based on the Capacitron model from Google. It's a partial implementation of the models detailed in the paper, hierarchical latent embeddings are still to be done - this is a TODO for later. If you'd like to get an idea what the model does and how it works, here's a post I did a few weeks ago. Audio samples to follow shortly.

Phonemes

I haven't followed the entire discussion about the phoneme issue, however I'm aware that because of license issues, espeak needed to be dropped. @Edresson mentioned that I could try to train the model with graphemes as well, however I haven't had time to do this yet. According to the paper, however, Capacitron does need phoneme inputs to work.

TODOs (List is still to be updated):

• Implement Coqpit config instead of capacitron_blizzad.json
• Fix changes coming from formatting
• Fix capacitron multispeaker training -- (HInt: tacotron_abstract.py)
• Refactor the parameter splitting in train_tacotron.py and training.py
• Create test for capacitron_layers.py
• Fix tests
• Refactor capacitron_eval.py into a notebook
• Fix a-froghyar#4

To Discuss

• Some of the losses that are normally activated in Tacotron are deactivated in Capacitron - during my experiments I stripped down the loss calculation to the bare minimum (which is already complicated enough) and I haven't experimented with turning them back on again since then
• DDC + Capacitron

Thanks, @a-froghyar an excellent PR congratulations again :). I will help and fix the compatibility issues. I also intend to leave the VAE generic and implement support in Tacotron 2. I intend to add it as was done for GST.

[erogol]

Also be aware of the upcoming Trainer API. Maybe it is better for you to check it as you design your PR ✨ .

[a-froghyar]

Update:

@Edresson was kind enough to open a PR on the branch - a-froghyar#6 -, this is being reviewed this week and will be pushed hopefully soon. The list on the top will be updated by the end of this week and the new Trainer API will also be taken into account.

[CLAassistant]

All committers have signed the CLA.

[a-froghyar]

Little update: I'm finishing up my thesis in the next 2 months so activity here won't be too much in the meantime, however there is active development on the model/PR :)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. You might also look our discussion channels.

[a-froghyar]

I'm submitting and presenting my thesis in the next weeks, the PR is still active.

[erogol]

@a-froghyar good luck!

[a-froghyar]

I'm done with the masters and I'll be working on this PR in the next week. @erogol @Edresson @WeberJulian could I post Qs here later? There seemed to have been loads of changes in the past 3 months so it'll take some time to reorganise things. :)

[a-froghyar]

OK, there's way too many conflicts - I'm actually going to restart the PR from a fresh dev branch. I'm gonna post the new PR here for follow-up!