[TTS] FastPitch speaker interpolation (NVIDIA#4997)

* Adding speaker interpolation tutorial Signed-off-by: subhankar-ghosh <subhankar2321@gmail.com> * Added interpolate_speaker method to FastPitch Signed-off-by: subhankar-ghosh <subhankar2321@gmail.com> * Adding speaker interpolation tutorial Signed-off-by: subhankar-ghosh <subhankar2321@gmail.com> * Added interpolate_speaker method to FastPitch Signed-off-by: subhankar-ghosh <subhankar2321@gmail.com> * Adding speaker interpolation tutorial Signed-off-by: subhankar-ghosh <subhankar2321@gmail.com> * Added interpolate_speaker method to FastPitch Signed-off-by: subhankar-ghosh <subhankar2321@gmail.com> * Making style related changes Signed-off-by: subhankar-ghosh <subhankar2321@gmail.com> * Adding installation instructions to tutorial Signed-off-by: subhankar-ghosh <subhankar2321@gmail.com> * Formatting changes Signed-off-by: subhankar-ghosh <subhankar2321@gmail.com> * More Formatting changes Signed-off-by: subhankar-ghosh <subhankar2321@gmail.com> * Adding speaker interpolation tutorial Signed-off-by: subhankar-ghosh <subhankar2321@gmail.com> * Added interpolate_speaker method to FastPitch Signed-off-by: subhankar-ghosh <subhankar2321@gmail.com> * Adding speaker interpolation tutorial Signed-off-by: subhankar-ghosh <subhankar2321@gmail.com> * Making style related changes Signed-off-by: subhankar-ghosh <subhankar2321@gmail.com> * Formatting changes Signed-off-by: subhankar-ghosh <subhankar2321@gmail.com> * More Formatting changes Signed-off-by: subhankar-ghosh <subhankar2321@gmail.com> * Merge conflicts Signed-off-by: subhankar-ghosh <subhankar2321@gmail.com> Signed-off-by: subhankar-ghosh <subhankar2321@gmail.com> Signed-off-by: Hainan Xu <hainanx@nvidia.com>
hainan-xv · Nov 29, 2022 · e2b1246 · e2b1246
1 parent bed608a
commit e2b1246
Show file tree

Hide file tree

Showing 2 changed files with 438 additions and 0 deletions.
diff --git a/nemo/collections/tts/models/fastpitch.py b/nemo/collections/tts/models/fastpitch.py
@@ -666,6 +666,38 @@ def forward_for_export(self, text, pitch, pace, volume=None, batch_lengths=None,
                 volume = volume_tensor
         return self.fastpitch.infer(text=text, pitch=pitch, pace=pace, volume=volume, speaker=speaker)
 
+    def interpolate_speaker(
+        self, original_speaker_1, original_speaker_2, weight_speaker_1, weight_speaker_2, new_speaker_id
+    ):
+        """
+        This method performs speaker interpolation between two original speakers the model is trained on.
+        Inputs:
+            original_speaker_1: Integer speaker ID of first existing speaker in the model
+            original_speaker_2: Integer speaker ID of second existing speaker in the model
+            weight_speaker_1: Floating point weight associated in to first speaker during weight combination
+            weight_speaker_2: Floating point weight associated in to second speaker during weight combination
+            new_speaker_id: Integer speaker ID of new interpolated speaker in the model
+        """
+        if self.fastpitch.speaker_emb is None:
+            raise Exception(
+                "Current FastPitch model is not a multi-speaker FastPitch model. Speaker interpolation can only \
+                be performed with a multi-speaker model"
+            )
+        n_speakers = self.fastpitch.speaker_emb.weight.data.size()[0]
+        if original_speaker_1 >= n_speakers or original_speaker_2 >= n_speakers or new_speaker_id >= n_speakers:
+            raise Exception(
+                f"Parameters original_speaker_1, original_speaker_2, new_speaker_id should be less than the total \
+                total number of speakers FastPitch was trained on (n_speakers = {n_speakers})."
+            )
+        speaker_emb_1 = (
+            self.fastpitch.speaker_emb(torch.tensor(original_speaker_1, dtype=torch.int32).cuda()).clone().detach()
+        )
+        speaker_emb_2 = (
+            self.fastpitch.speaker_emb(torch.tensor(original_speaker_2, dtype=torch.int32).cuda()).clone().detach()
+        )
+        new_speaker_emb = weight_speaker_1 * speaker_emb_1 + weight_speaker_2 * speaker_emb_2
+        self.fastpitch.speaker_emb.weight.data[new_speaker_id] = new_speaker_emb
+
 
 @torch.jit.script
 def create_batch(