Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[TTS] Add Mixed Representation Training #3473

Merged
merged 35 commits into from
Feb 11, 2022
Merged

[TTS] Add Mixed Representation Training #3473

merged 35 commits into from
Feb 11, 2022

Conversation

blisc
Copy link
Collaborator

@blisc blisc commented Jan 20, 2022 •
edited
Loading

  • Adds phoneme probability to our current g2p module.
  • Updates FastPitch yaml to use this functionality.

redoctopus and others added 11 commits January 14, 2022 14:25
@redoctopus
Update CMUdict with ADLR version pronunciations
25e7dff 
Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>
@blisc
minor updates for finetuning
bf21276 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
update conf
6bb0f0b 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
merge
8db6e5c 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
Merge remote-tracking branch 'nvidia/cmudict_update' into tts_finetun…
ed6121f 
…ing_updates
@blisc
update
725e15c 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
update
d0f7edc 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
bug fixes
eacd2ba 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
update config
3793c13 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
bf16 support
729fc4c 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
bf16 support
539a4ab 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc blisc changed the title Add Mixed Representation Training [TTS] Add Mixed Representation Training Jan 20, 2022
blisc added 3 commits January 21, 2022 09:19
@blisc
bugfix
05309f1 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
update
b479e7a 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
finalize changes
3869f79 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc blisc changed the base branch from r1.6.0 to main February 1, 2022 14:54
blisc added 3 commits February 1, 2022 06:57
@blisc
merge with main
cdc1f6c 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
undo notebook 1.6.0 pins
23aeb2c 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
more 1.6.0 undos
0d33f71 
Signed-off-by: Jason <jasoli@nvidia.com>
@Oktai15
Copy link
Contributor

Oktai15 commented Feb 3, 2022

See this bug: #3593

@Oktai15
Copy link
Contributor

Oktai15 commented Feb 3, 2022

@blisc

blisc added 7 commits February 3, 2022 08:13
@blisc
wip
780828c 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
update num_workers
b54f390 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
update hypers
b09dee9 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
revert to main _align yamls
a4124e3 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
update yamls
7898a40 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
cleanup
1342c23 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
merge with main
832f431 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc blisc requested a review from Oktai15 February 4, 2022 16:37
@blisc blisc marked this pull request as ready for review February 4, 2022 16:38
Oktai15
Oktai15 suggested changes Feb 5, 2022
View reviewed changes
Copy link
Contributor

@Oktai15 Oktai15 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great PR, thanks for this feature. Need to fix comments.

examples/tts/conf/fastpitch_align_v1.05.yaml Outdated
@@ -10,8 +10,8 @@ sup_data_path: ???
sup_data_types: [ "align_prior_matrix", "pitch" ]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some important notes about FastPitch + Aligner in mixed representation training overall:

  1. Did you try to train this model? According to FastPitch paper, "However, the mixed approach of training on phonemes and graphemes introduced unpleasant artifacts", have you checked that the artifacts do not appear? If so, it is very-very big result!
  2. Despite the fact that this config is very cool, I would prefer to create separate config for mixer representation training, because it is not ordinary case. Moreover, beta_binomial_interpolator is not so accurate. What do you think about it? I suggest to keep it unchanged and create conf/fastpitch_align_mixed_representation_v1.05.yaml (we will rename it to conf/fastpitch_align_mixed_representation.yaml in 1.8.0).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this will be the default fastpitch config moving forward as it is what will be used in riva.

examples/tts/conf/fastpitch_align_v1.05.yaml Outdated Show resolved Hide resolved
examples/tts/conf/fastpitch_align_v1.05.yaml Outdated Show resolved Hide resolved
nemo/collections/tts/torch/data.py Outdated Show resolved Hide resolved
nemo/collections/tts/torch/data.py Show resolved Hide resolved
nemo/collections/tts/torch/data.py Outdated Show resolved Hide resolved
nemo/collections/tts/torch/g2ps.py Outdated Show resolved Hide resolved
nemo/collections/tts/torch/g2ps.py Outdated Show resolved Hide resolved
nemo/collections/tts/torch/tts_tokenizers.py Show resolved Hide resolved
nemo/collections/tts/torch/tts_tokenizers.py Show resolved Hide resolved
blisc added 3 commits February 7, 2022 09:11
@blisc
remove unnecessary line
d8f796e 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
address comments
a31e824 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
merge
4f09e27 
Signed-off-by: Jason <jasoli@nvidia.com>
@lgtm-com
Copy link

lgtm-com bot commented Feb 7, 2022

This pull request introduces 2 alerts when merging 4f09e27 into 8313b01 - view on LGTM.com

new alerts:

  • 1 for Non-callable called
  • 1 for Unused local variable

@blisc
update vocoder mel uploading; add contextmanager to mixed g2p
341deb2 
Signed-off-by: Jason <jasoli@nvidia.com>
Oktai15
Oktai15 suggested changes Feb 11, 2022
View reviewed changes
Jenkinsfile Outdated Show resolved Hide resolved
nemo/collections/tts/models/fastpitch.py Show resolved Hide resolved
nemo/collections/tts/models/fastpitch.py Outdated Show resolved Hide resolved
nemo/collections/tts/models/fastpitch.py Show resolved Hide resolved
nemo/collections/tts/models/mixer_tts.py Show resolved Hide resolved
nemo/collections/tts/torch/g2ps.py Show resolved Hide resolved
nemo/collections/tts/torch/tts_tokenizers.py Show resolved Hide resolved
blisc added 3 commits February 11, 2022 07:09
@blisc
update comments; make prob required argument
1201136 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
added val check
bc218ce 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
update message
0f9386e 
Signed-off-by: Jason <jasoli@nvidia.com>
Oktai15
Oktai15 suggested changes Feb 11, 2022
View reviewed changes
nemo/collections/tts/models/fastpitch.py Outdated Show resolved Hide resolved
blisc added 3 commits February 11, 2022 07:53
@blisc
update
8d0baaa 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
Merge remote-tracking branch 'nvidia/main' into tts_finetuning_updates
2a34a70 
Signed-off-by: Jason <jasoli@nvidia.com>
@blisc
revert num workers
28d1fd5 
Signed-off-by: Jason <jasoli@nvidia.com>
@Oktai15 Oktai15 self-requested a review February 11, 2022 20:58
Oktai15
Oktai15 approved these changes Feb 11, 2022
View reviewed changes
Copy link
Contributor

@Oktai15 Oktai15 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great PR, thanks.

We need to think more about accuracy of beta binomial interpolator and also think about potential bug in TTSDataset

@blisc blisc merged commit c645c4c into NVIDIA:main Feb 11, 2022
@blisc blisc deleted the tts_finetuning_updates branch February 11, 2022 22:26
fayejf pushed a commit that referenced this pull request Mar 2, 2022
@blisc @redoctopus @fayejf
[TTS] Add Mixed Representation Training (#3473)
26144f9 
* Update CMUdict with ADLR version pronunciations

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

* minor updates for finetuning

Signed-off-by: Jason <jasoli@nvidia.com>

* update conf

Signed-off-by: Jason <jasoli@nvidia.com>

* merge

Signed-off-by: Jason <jasoli@nvidia.com>

* update

Signed-off-by: Jason <jasoli@nvidia.com>

* update

Signed-off-by: Jason <jasoli@nvidia.com>

* bug fixes

Signed-off-by: Jason <jasoli@nvidia.com>

* update config

Signed-off-by: Jason <jasoli@nvidia.com>

* bf16 support

Signed-off-by: Jason <jasoli@nvidia.com>

* bf16 support

Signed-off-by: Jason <jasoli@nvidia.com>

* bugfix

Signed-off-by: Jason <jasoli@nvidia.com>

* update

Signed-off-by: Jason <jasoli@nvidia.com>

* finalize changes

Signed-off-by: Jason <jasoli@nvidia.com>

* undo notebook 1.6.0 pins

Signed-off-by: Jason <jasoli@nvidia.com>

* more 1.6.0 undos

Signed-off-by: Jason <jasoli@nvidia.com>

* wip

Signed-off-by: Jason <jasoli@nvidia.com>

* update num_workers

Signed-off-by: Jason <jasoli@nvidia.com>

* update hypers

Signed-off-by: Jason <jasoli@nvidia.com>

* revert to main _align yamls

Signed-off-by: Jason <jasoli@nvidia.com>

* update yamls

Signed-off-by: Jason <jasoli@nvidia.com>

* cleanup

Signed-off-by: Jason <jasoli@nvidia.com>

* remove unnecessary line

Signed-off-by: Jason <jasoli@nvidia.com>

* address comments

Signed-off-by: Jason <jasoli@nvidia.com>

* update vocoder mel uploading; add contextmanager to mixed g2p

Signed-off-by: Jason <jasoli@nvidia.com>

* update comments; make prob required argument

Signed-off-by: Jason <jasoli@nvidia.com>

* added val check

Signed-off-by: Jason <jasoli@nvidia.com>

* update message

Signed-off-by: Jason <jasoli@nvidia.com>

* update

Signed-off-by: Jason <jasoli@nvidia.com>

* revert num workers

Signed-off-by: Jason <jasoli@nvidia.com>

Co-authored-by: Jocelyn Huang <jocelynh@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Oktai15 Oktai15 Oktai15 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@blisc @Oktai15 @redoctopus