Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix replace_bos_with_pad not found #6450

Merged

Conversation

github-actions[bot]
Copy link
Contributor

What does this PR do ?

minor fix for replace_bos_with_pad not found error in CI

Collection: [Note which collection this PR will affect]

Changelog

  • Add specific line by line info of high level changes in this PR.

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this 

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

  • New Feature
  • Bugfix
  • Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

  • Related to # (issue)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
@aklife97 aklife97 merged commit daa9744 into main Apr 22, 2023
@aklife97 aklife97 deleted the cherry-pick-main-cda68c7c3d1603ef0d52fb633fb739e387e6cb2e branch April 22, 2023 00:09
yzhang123 pushed a commit that referenced this pull request Apr 24, 2023
* [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

[TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [TTS] whitelist broken path fix. (#6412)

* [TTS] whitelist broken path fix.

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [TTS] FastPitch speaker encoder (#6417)

* Add initial codes

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove wemb

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix import

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Restore aligner loss

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add ConditionalInput

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix error and support pre-trained config

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Follow comments

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Rename config

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Change copyright and random weight test

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add initial codes

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix import error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add initial codes

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix dataset error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove reference speaker embedding

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove SV encoder

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Follow comments

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix length type

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix append

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Move error msg

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add look-up into speaker encoder

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add valueerror msg

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Move lookup

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove unused

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Rebase and Fix error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix spk encoder

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Rename n_speakers

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Follow comments

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix n_speakers None error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Sharded manifests for tarred datasets (#6395)

* testing sharded manifests

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* compatibility

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* proper fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* adding flag tot convert_to_tarred_audio_dataset

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* shard_manifests conf param

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* propagating the shard_manifests param

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* propagating the shard_manifests param

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* distributed checks

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* typo

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* typo

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes based on PR comments and tests

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes to convert_to_tarred_audio_dataset.py

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* reversing manifest shards flag

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* excluding manifests from webdataset url expansion

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* expand manifest paths before attempting to cache from datastore

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* explicit use of UTF-8 for manifest i/o

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

---------

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update wfst_text_normalization.rst (#6374)

Add Hungarian (incoming in NeMo-text-processing)

Signed-off-by: Jim O’Regan <jaoregan@tcd.ie>

* Support Swiglu in TP PP Conversion (#6437) (#6451)

* Support Swiglu in TP PP Conversion



* Guard activation



* Guard activation



---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>

* Update NeMo_TTS_Primer.ipynb (#6436)

* Update NeMo_TTS_Primer.ipynb

Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch.

Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>

* Update NeMo_TTS_Primer.ipynb

Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point.

Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>

---------

Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>

* add rampup batch size support for Megatron GPT (#6424)

* added rampup batch size support

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* added tests for rampup batch size

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* fixed the typos

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* added assertions

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* changed assertion rules

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* deleted unused imports

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* changed tests for rampup batch size

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* updated rampup batch size tests

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed styling

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* rampup batch size tests changes

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

---------

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461)

* 1. Meagtron encoder decoder fix for empty validation outputs.



* 1. Debugging.

---------

Signed-off-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: Micha Livne <michalivne@users.noreply.github.com>
Co-authored-by: Micha Livne <mlivne@nvidia.com>

* Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448)

* added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

* updated README with the new agg_tokenizer_manifest flag

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

* fixed typo in scripts/speech_recognition/code_switching/README.md

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

* changed agg_tokenizer_manifest to is_lid_manifest

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

---------

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Co-authored-by: Dima Rekesh <bmwshop@gmail.com>

* Added/updated new Conformer configs (#6426) (#6467)

* Update script for ngram rnnt and hat beam search decoding (#6370)

* add rnnt ngram beamsearch script

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add return encoding embedding option

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* update script

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add rnnt and hat ngram decoding script

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add some parameters

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add return_encoder_embeddings parameter to RNNTDecodingConfig

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* replace return_encoder_embeddings parameter

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* generalization of scipt behavior

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove return_encoder_embeddings parameter

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* remove return_encoder_embeddings parameter

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add manual encoder_embeddings calculation

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix beam_width value to 8

Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>

* fix rescoring description

Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>

---------

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>

* BERT pre-training mp fork to spawn (#6442) (#6454)

* change bert fork to spawn



* num_workers=0 fix



---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* fix replace_bos_with_pad not found (#6443) (#6450)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* reduce workers on NMT CI (#6472) (#6474)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* 1. Added KERPLE positional embeddings to encoder-decoder.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Added a missing file.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Fixing commits.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Signed-off-by: Jim O’Regan <jaoregan@tcd.ie>
Signed-off-by: smajumdar <titu1994@gmail.com>
Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Signed-off-by: Micha Livne <mlivne@nvidia.com>
Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Dima Rekesh <bmwshop@gmail.com>
Co-authored-by: Jim O’Regan <jaoregan@tcd.ie>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
github-actions bot added a commit that referenced this pull request Apr 24, 2023
* [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

[TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [TTS] whitelist broken path fix. (#6412)

* [TTS] whitelist broken path fix.

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [TTS] FastPitch speaker encoder (#6417)

* Add initial codes

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove wemb

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix import

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Restore aligner loss

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add ConditionalInput

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix error and support pre-trained config

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Follow comments

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Rename config

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Change copyright and random weight test

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add initial codes

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix import error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add initial codes

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix dataset error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove reference speaker embedding

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove SV encoder

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Follow comments

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix length type

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix append

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Move error msg

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add look-up into speaker encoder

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add valueerror msg

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Move lookup

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove unused

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Rebase and Fix error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix spk encoder

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Rename n_speakers

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Follow comments

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix n_speakers None error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Sharded manifests for tarred datasets (#6395)

* testing sharded manifests

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* compatibility

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* proper fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* adding flag tot convert_to_tarred_audio_dataset

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* shard_manifests conf param

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* propagating the shard_manifests param

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* propagating the shard_manifests param

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* distributed checks

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* typo

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* typo

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes based on PR comments and tests

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes to convert_to_tarred_audio_dataset.py

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* reversing manifest shards flag

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* excluding manifests from webdataset url expansion

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* expand manifest paths before attempting to cache from datastore

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* explicit use of UTF-8 for manifest i/o

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

---------

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update wfst_text_normalization.rst (#6374)

Add Hungarian (incoming in NeMo-text-processing)

Signed-off-by: Jim O’Regan <jaoregan@tcd.ie>

* Support Swiglu in TP PP Conversion (#6437) (#6451)

* Support Swiglu in TP PP Conversion



* Guard activation



* Guard activation



---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>

* Update NeMo_TTS_Primer.ipynb (#6436)

* Update NeMo_TTS_Primer.ipynb

Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch.

Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>

* Update NeMo_TTS_Primer.ipynb

Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point.

Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>

---------

Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>

* add rampup batch size support for Megatron GPT (#6424)

* added rampup batch size support

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* added tests for rampup batch size

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* fixed the typos

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* added assertions

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* changed assertion rules

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* deleted unused imports

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* changed tests for rampup batch size

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* updated rampup batch size tests

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed styling

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* rampup batch size tests changes

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

---------

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461)

* 1. Meagtron encoder decoder fix for empty validation outputs.



* 1. Debugging.

---------

Signed-off-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: Micha Livne <michalivne@users.noreply.github.com>
Co-authored-by: Micha Livne <mlivne@nvidia.com>

* Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448)

* added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

* updated README with the new agg_tokenizer_manifest flag

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

* fixed typo in scripts/speech_recognition/code_switching/README.md

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

* changed agg_tokenizer_manifest to is_lid_manifest

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

---------

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Co-authored-by: Dima Rekesh <bmwshop@gmail.com>

* Added/updated new Conformer configs (#6426) (#6467)

* Update script for ngram rnnt and hat beam search decoding (#6370)

* add rnnt ngram beamsearch script

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add return encoding embedding option

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* update script

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add rnnt and hat ngram decoding script

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add some parameters

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add return_encoder_embeddings parameter to RNNTDecodingConfig

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* replace return_encoder_embeddings parameter

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* generalization of scipt behavior

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove return_encoder_embeddings parameter

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* remove return_encoder_embeddings parameter

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add manual encoder_embeddings calculation

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix beam_width value to 8

Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>

* fix rescoring description

Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>

---------

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>

* BERT pre-training mp fork to spawn (#6442) (#6454)

* change bert fork to spawn



* num_workers=0 fix



---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* fix replace_bos_with_pad not found (#6443) (#6450)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* reduce workers on NMT CI (#6472) (#6474)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* 1. Added KERPLE positional embeddings to encoder-decoder.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Added a missing file.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Fixing commits.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Signed-off-by: Jim O’Regan <jaoregan@tcd.ie>
Signed-off-by: smajumdar <titu1994@gmail.com>
Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Signed-off-by: Micha Livne <mlivne@nvidia.com>
Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Dima Rekesh <bmwshop@gmail.com>
Co-authored-by: Jim O’Regan <jaoregan@tcd.ie>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
michalivne added a commit that referenced this pull request Apr 24, 2023
* [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

[TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

---------




* [TTS] whitelist broken path fix. (#6412)

* [TTS] whitelist broken path fix.



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------




* [TTS] FastPitch speaker encoder (#6417)

* Add initial codes



* Remove wemb



* Fix import



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Restore aligner loss



* Add ConditionalInput



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix error and support pre-trained config



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Follow comments



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Rename config



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Change copyright and random weight test



* Add initial codes



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix import error



* Add initial codes



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix dataset error



* Remove reference speaker embedding



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Remove SV encoder



* Follow comments



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix length type



* Fix append



* Move error msg



* Add look-up into speaker encoder



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Add valueerror msg



* Move lookup



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Remove unused



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix error



* Rebase and Fix error



* Fix spk encoder



* Rename n_speakers



* Follow comments



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix n_speakers None error



---------




* Sharded manifests for tarred datasets (#6395)

* testing sharded manifests



* compatibility



* proper fixes



* adding flag tot convert_to_tarred_audio_dataset



* shard_manifests conf param



* propagating the shard_manifests param



* propagating the shard_manifests param



* distributed checks



* typo



* typo



* fixes



* fixes



* fixes



* fixes



* fixes



* fixes



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes based on PR comments and tests



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes to convert_to_tarred_audio_dataset.py



* reversing manifest shards flag



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests



* excluding manifests from webdataset url expansion



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* expand manifest paths before attempting to cache from datastore



* explicit use of UTF-8 for manifest i/o



---------




* Update wfst_text_normalization.rst (#6374)

Add Hungarian (incoming in NeMo-text-processing)



* Support Swiglu in TP PP Conversion (#6437) (#6451)

* Support Swiglu in TP PP Conversion



* Guard activation



* Guard activation



---------




* Update NeMo_TTS_Primer.ipynb (#6436)

* Update NeMo_TTS_Primer.ipynb

Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch.



* Update NeMo_TTS_Primer.ipynb

Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point.



---------



* add rampup batch size support for Megatron GPT (#6424)

* added rampup batch size support



* added tests for rampup batch size



* fixed the typos



* added assertions



* changed assertion rules



* deleted unused imports



* changed tests for rampup batch size



* updated rampup batch size tests



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed styling



* rampup batch size tests changes



---------







* Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461)

* 1. Meagtron encoder decoder fix for empty validation outputs.



* 1. Debugging.

---------





* Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448)

* added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default



* updated README with the new agg_tokenizer_manifest flag



* fixed typo in scripts/speech_recognition/code_switching/README.md



* changed agg_tokenizer_manifest to is_lid_manifest



---------




* Added/updated new Conformer configs (#6426) (#6467)

* Update script for ngram rnnt and hat beam search decoding (#6370)

* add rnnt ngram beamsearch script



* add return encoding embedding option



* update script



* add rnnt and hat ngram decoding script



* add some parameters



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add return_encoder_embeddings parameter to RNNTDecodingConfig



* replace return_encoder_embeddings parameter



* generalization of scipt behavior



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove return_encoder_embeddings parameter



* remove return_encoder_embeddings parameter



* add manual encoder_embeddings calculation



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix beam_width value to 8



* fix rescoring description



---------






* BERT pre-training mp fork to spawn (#6442) (#6454)

* change bert fork to spawn



* num_workers=0 fix



---------




* fix replace_bos_with_pad not found (#6443) (#6450)




* reduce workers on NMT CI (#6472) (#6474)




* 1. Added KERPLE positional embeddings to encoder-decoder.



* 1. Added a missing file.



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Fixing commits.



* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Signed-off-by: Jim O’Regan <jaoregan@tcd.ie>
Signed-off-by: smajumdar <titu1994@gmail.com>
Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Signed-off-by: Micha Livne <mlivne@nvidia.com>
Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Micha Livne <michalivne@users.noreply.github.com>
Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Dima Rekesh <bmwshop@gmail.com>
Co-authored-by: Jim O’Regan <jaoregan@tcd.ie>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
hsiehjackson pushed a commit to hsiehjackson/NeMo that referenced this pull request Jun 2, 2023
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
hsiehjackson added a commit to hsiehjackson/NeMo that referenced this pull request Jun 2, 2023
* [TTS] FastPitch adapter fine-tune and conditional layer normalization (NVIDIA#6416)

[TTS] FastPitch adapter fine-tune and conditional layer normalization (NVIDIA#6416)

---------




* [TTS] whitelist broken path fix. (NVIDIA#6412)

* [TTS] whitelist broken path fix.



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------




* [TTS] FastPitch speaker encoder (NVIDIA#6417)

* Add initial codes



* Remove wemb



* Fix import



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Restore aligner loss



* Add ConditionalInput



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix error and support pre-trained config



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Follow comments



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Rename config



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Change copyright and random weight test



* Add initial codes



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix import error



* Add initial codes



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix dataset error



* Remove reference speaker embedding



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Remove SV encoder



* Follow comments



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix length type



* Fix append



* Move error msg



* Add look-up into speaker encoder



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Add valueerror msg



* Move lookup



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Remove unused



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix error



* Rebase and Fix error



* Fix spk encoder



* Rename n_speakers



* Follow comments



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix n_speakers None error



---------




* Sharded manifests for tarred datasets (NVIDIA#6395)

* testing sharded manifests



* compatibility



* proper fixes



* adding flag tot convert_to_tarred_audio_dataset



* shard_manifests conf param



* propagating the shard_manifests param



* propagating the shard_manifests param



* distributed checks



* typo



* typo



* fixes



* fixes



* fixes



* fixes



* fixes



* fixes



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes based on PR comments and tests



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes to convert_to_tarred_audio_dataset.py



* reversing manifest shards flag



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests



* excluding manifests from webdataset url expansion



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* expand manifest paths before attempting to cache from datastore



* explicit use of UTF-8 for manifest i/o



---------




* Update wfst_text_normalization.rst (NVIDIA#6374)

Add Hungarian (incoming in NeMo-text-processing)



* Support Swiglu in TP PP Conversion (NVIDIA#6437) (NVIDIA#6451)

* Support Swiglu in TP PP Conversion



* Guard activation



* Guard activation



---------




* Update NeMo_TTS_Primer.ipynb (NVIDIA#6436)

* Update NeMo_TTS_Primer.ipynb

Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch.



* Update NeMo_TTS_Primer.ipynb

Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point.



---------



* add rampup batch size support for Megatron GPT (NVIDIA#6424)

* added rampup batch size support



* added tests for rampup batch size



* fixed the typos



* added assertions



* changed assertion rules



* deleted unused imports



* changed tests for rampup batch size



* updated rampup batch size tests



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed styling



* rampup batch size tests changes



---------







* Meagtron encoder decoder fix for empty validation outputs (NVIDIA#6459) (NVIDIA#6461)

* 1. Meagtron encoder decoder fix for empty validation outputs.



* 1. Debugging.

---------





* Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (NVIDIA#6448)

* added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default



* updated README with the new agg_tokenizer_manifest flag



* fixed typo in scripts/speech_recognition/code_switching/README.md



* changed agg_tokenizer_manifest to is_lid_manifest



---------




* Added/updated new Conformer configs (NVIDIA#6426) (NVIDIA#6467)

* Update script for ngram rnnt and hat beam search decoding (NVIDIA#6370)

* add rnnt ngram beamsearch script



* add return encoding embedding option



* update script



* add rnnt and hat ngram decoding script



* add some parameters



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add return_encoder_embeddings parameter to RNNTDecodingConfig



* replace return_encoder_embeddings parameter



* generalization of scipt behavior



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove return_encoder_embeddings parameter



* remove return_encoder_embeddings parameter



* add manual encoder_embeddings calculation



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix beam_width value to 8



* fix rescoring description



---------






* BERT pre-training mp fork to spawn (NVIDIA#6442) (NVIDIA#6454)

* change bert fork to spawn



* num_workers=0 fix



---------




* fix replace_bos_with_pad not found (NVIDIA#6443) (NVIDIA#6450)




* reduce workers on NMT CI (NVIDIA#6472) (NVIDIA#6474)




* 1. Added KERPLE positional embeddings to encoder-decoder.



* 1. Added a missing file.



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Fixing commits.



* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Signed-off-by: Jim O’Regan <jaoregan@tcd.ie>
Signed-off-by: smajumdar <titu1994@gmail.com>
Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Signed-off-by: Micha Livne <mlivne@nvidia.com>
Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Micha Livne <michalivne@users.noreply.github.com>
Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Dima Rekesh <bmwshop@gmail.com>
Co-authored-by: Jim O’Regan <jaoregan@tcd.ie>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
ekmb added a commit that referenced this pull request Jun 12, 2023
…d Flash Attention (#6666)

* move to nvidia megatron repo (#6465) (#6475)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Megatron KERPLE positional embeddings (#6478) (#6480)

* [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

[TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

---------




* [TTS] whitelist broken path fix. (#6412)

* [TTS] whitelist broken path fix.



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------




* [TTS] FastPitch speaker encoder (#6417)

* Add initial codes



* Remove wemb



* Fix import



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Restore aligner loss



* Add ConditionalInput



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix error and support pre-trained config



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Follow comments



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Rename config



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Change copyright and random weight test



* Add initial codes



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix import error



* Add initial codes



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix dataset error



* Remove reference speaker embedding



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Remove SV encoder



* Follow comments



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix length type



* Fix append



* Move error msg



* Add look-up into speaker encoder



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Add valueerror msg



* Move lookup



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Remove unused



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix error



* Rebase and Fix error



* Fix spk encoder



* Rename n_speakers



* Follow comments



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix n_speakers None error



---------




* Sharded manifests for tarred datasets (#6395)

* testing sharded manifests



* compatibility



* proper fixes



* adding flag tot convert_to_tarred_audio_dataset



* shard_manifests conf param



* propagating the shard_manifests param



* propagating the shard_manifests param



* distributed checks



* typo



* typo



* fixes



* fixes



* fixes



* fixes



* fixes



* fixes



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes based on PR comments and tests



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes to convert_to_tarred_audio_dataset.py



* reversing manifest shards flag



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests



* excluding manifests from webdataset url expansion



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* expand manifest paths before attempting to cache from datastore



* explicit use of UTF-8 for manifest i/o



---------




* Update wfst_text_normalization.rst (#6374)

Add Hungarian (incoming in NeMo-text-processing)



* Support Swiglu in TP PP Conversion (#6437) (#6451)

* Support Swiglu in TP PP Conversion



* Guard activation



* Guard activation



---------




* Update NeMo_TTS_Primer.ipynb (#6436)

* Update NeMo_TTS_Primer.ipynb

Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch.



* Update NeMo_TTS_Primer.ipynb

Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point.



---------



* add rampup batch size support for Megatron GPT (#6424)

* added rampup batch size support



* added tests for rampup batch size



* fixed the typos



* added assertions



* changed assertion rules



* deleted unused imports



* changed tests for rampup batch size



* updated rampup batch size tests



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed styling



* rampup batch size tests changes



---------







* Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461)

* 1. Meagtron encoder decoder fix for empty validation outputs.



* 1. Debugging.

---------





* Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448)

* added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default



* updated README with the new agg_tokenizer_manifest flag



* fixed typo in scripts/speech_recognition/code_switching/README.md



* changed agg_tokenizer_manifest to is_lid_manifest



---------




* Added/updated new Conformer configs (#6426) (#6467)

* Update script for ngram rnnt and hat beam search decoding (#6370)

* add rnnt ngram beamsearch script



* add return encoding embedding option



* update script



* add rnnt and hat ngram decoding script



* add some parameters



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add return_encoder_embeddings parameter to RNNTDecodingConfig



* replace return_encoder_embeddings parameter



* generalization of scipt behavior



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove return_encoder_embeddings parameter



* remove return_encoder_embeddings parameter



* add manual encoder_embeddings calculation



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix beam_width value to 8



* fix rescoring description



---------






* BERT pre-training mp fork to spawn (#6442) (#6454)

* change bert fork to spawn



* num_workers=0 fix



---------




* fix replace_bos_with_pad not found (#6443) (#6450)




* reduce workers on NMT CI (#6472) (#6474)




* 1. Added KERPLE positional embeddings to encoder-decoder.



* 1. Added a missing file.



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Fixing commits.



* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Signed-off-by: Jim O’Regan <jaoregan@tcd.ie>
Signed-off-by: smajumdar <titu1994@gmail.com>
Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Signed-off-by: Micha Livne <mlivne@nvidia.com>
Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Micha Livne <michalivne@users.noreply.github.com>
Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Dima Rekesh <bmwshop@gmail.com>
Co-authored-by: Jim O’Regan <jaoregan@tcd.ie>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix an invalid link in get_data.py of ljspeech (#6456)

Usage of the link in line 63 leads to downloading a html file not a tsv file, so we need to change it to a raw link.

Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* 1. Added external index sample. (#6462) (#6483)

Signed-off-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: Micha Livne <michalivne@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Update README to add core installation (#6488) (#6489)

* update README for megatron-core



* fix



---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix cache aware hybrid bugs (#6466) (#6484)

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix typos (#6494) (#6495)

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add disclaimer about dataset for ASR (#6496)

Signed-off-by: smajumdar <titu1994@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* fix (#6502)

datastore_path_to_webdataset_url(p) if is_datastore_path(p) and is_tarred_path(p) else p
NameError: name 'is_tarred_path' is not defined

Co-authored-by: George <gzelenfroind@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* fix broken links r1.18.0 (#6501) (#6504)

* fix broken links



* fix broken links



---------

Signed-off-by: Evelina <ebakhturina@nvidia.com>
Co-authored-by: Evelina <10428420+ekmb@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [TTS] Create functions for TTS preprocessing without dataloader (#6317)

* [TTS] Create functions for TTS preprocessing without dataloader

Signed-off-by: Ryan <rlangman@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Cache aware streaming nfa (#6209)

* add cache aware streaming to nemo aligner

Signed-off-by: Slyne Deng <slyned@nvidia.com>

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [BugFix] Force _get_batch_preds() to keep logits in decoder timestamps generator (#6499)

* [BugFix] _get_batch_preds() is forced to keep logits in  decoder timestamps generators

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Ingnore keep_logits boolean in FrameASRBatchLogits

Signed-off-by: Taejin Park <tango4j@gmail.com>

---------

Signed-off-by: Taejin Park <tango4j@gmail.com>
Co-authored-by: Jagadeesh Balam <4916480+jbalam-nv@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [TTS] Fix FastPitch energy code (#6511)

Signed-off-by: Ryan <rlangman@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* fix custom forward_torch_softmax (#6512) (#6517)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [TTS] fixed broken path. (#6514) (#6518)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix normalization of impulse response in ImpulsePerturbation (#6505)

Signed-off-by: Ante Jukić <ajukic@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add interleaved pp support (#6498)

* Add support for Virtual Pipeline Parallel conversion

Signed-off-by: smajumdar <titu1994@gmail.com>

* Add support for Virtual Pipeline Parallel conversion

Signed-off-by: smajumdar <titu1994@gmail.com>

* Switch to megatron core

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix typos (#6523)

* Fix typos

Signed-off-by: smajumdar <titu1994@gmail.com>

* Fix typos

Signed-off-by: smajumdar <titu1994@gmail.com>

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* New noise_norm perturbation based on Riva work (#6445)

* Initial commit for new noise_norm perturbation

Signed-off-by: Daniel Egert <degert@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Minor fix to random seed in perturb

Signed-off-by: Daniel Egert <degert@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated code to reflect feedback

Signed-off-by: Daniel Egert <degert@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updates for feedback given by code reviewers

Signed-off-by: Daniel Egert <degert@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updates in response to PR feedback

Signed-off-by: Daniel Egert <degert@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added comment about ref_mic being None

Signed-off-by: Daniel Egert <degert@nvidia.com>

* Updated perturb to use inspect module

Signed-off-by: Daniel Egert <degert@nvidia.com>

---------

Signed-off-by: Daniel Egert <degert@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [TTS] Add script for computing feature stats (#6508)

* [TTS] Add script for computing feature stats

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Add overwrite config

Signed-off-by: Ryan <rlangman@nvidia.com>

---------

Signed-off-by: Ryan <rlangman@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add Frame-VAD model and datasets (#6441)

* add model, dataset, necessary utils and tests

Signed-off-by: stevehuang52 <heh@nvidia.com>

* fix tarred data

Signed-off-by: stevehuang52 <heh@nvidia.com>

* fix typo

Signed-off-by: stevehuang52 <heh@nvidia.com>

* update docstring

Signed-off-by: stevehuang52 <heh@nvidia.com>

* update doc

Signed-off-by: stevehuang52 <heh@nvidia.com>

* update doc

Signed-off-by: stevehuang52 <heh@nvidia.com>

* update pretrained model info

Signed-off-by: stevehuang52 <heh@nvidia.com>

---------

Signed-off-by: stevehuang52 <heh@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Support dynamic length batches with GPT SFT (#6510)

* Support synamic length with GPT SFT

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* make branch functional

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* added back the fast emit section to the configs. (#6540) (#6542)

* added back the fast emit section to the configs.



* added back the fast emit section to the configs.



---------

Signed-off-by: Vahid <vnoroozi@nvidia.com>
Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* removing unnessary avoid_bfloat16_autocast_context (#6481)

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* FC models in menu (#6473)

* FC models in menu

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [TTS] Add tutorials for FastPitch TTS speaker adaptation with adapters (#6431)

* Add tts adapter tutorial

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Update main tutorial

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add tts adapter tutorial

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Update main tutorial

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Update tutorial

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Follow comments

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Follow comments

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix load .nemo error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Support multi-speaker fine-tune

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Follow comments

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Use .nemo

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Follow Comments

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix bug

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix bug

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix bug

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add precomputed speaker emb

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix space

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove repeated argument

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* optional batch size

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix comments in notebook

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [TTS] Create initial TTS dataset feature processors (#6507)

Signed-off-by: Ryan <rlangman@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* fix (#6529) (#6546)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add FastConformer Hybrid ASR models for EN, ES, IT, DE, PL, HR, UA, BY (#6549) (#6553)

* Added fastconfomer hybrid asr models for en, es, it, de, pl, hr, ua, by



* updated ASR docs with the fastconformer hybrid checkpoints



* added the fastconformer RNNT and CTC models



---------

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>
Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add scores for FastConformer models (#6557) (#6558)

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix fp16 (#6543) (#6544)

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Patch transcribe and support offline transcribe for hybrid model (#6550) (#6559)

Signed-off-by: fayejf <fayejf07@gmail.com>
Co-authored-by: fayejf <36722593+fayejf@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix notebook bad json (#6561)

Signed-off-by: smajumdar <titu1994@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Change Megatron Enc Dec model to use persistent_workers (#6548) (#6552)

* persistent workers



* fix



---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Make KenLM with PC for AggregateTokenizer and merge it (#6081)

* do_lowercase, rm_punctuation

Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>

* support beam_strategy = beam

Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>

* black

Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix config and^Cunctuation capitalization

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rm math

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* update kenlm

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* black

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add opengrm

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* mv install_beamsearch_decoders

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* punctuation_to_preserve

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Only tikenizer opion

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* Black

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* DEFAULT_TOKEN_OFFSET

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* aggregate_tokenizer

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* install kenlm with more than 5gram

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* install_beamsearch_decoders

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* ngram_bin_path kenlm_bin_path

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* black

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* fix greedy PC bug

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* move global params

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* fix description and perplexity

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* fix description

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* NEMO_PATH

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* nemo:23.01

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* License

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* description

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* isinstance

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* refactor kenlm stdin

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* black

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* add cmd arg

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* use new iter_files

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* EncDecHybridRNNTCTCModel

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* punctuation

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* train_kenlm args

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* add docstrings

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add ngram_merge docs

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* ngram_prune

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* rename to ngram_merge

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* rename to ngram

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* add comments

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* Ngram

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* nemo_model_file

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* install_opengrm_ngram

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* install opengrm

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* rename to install_opengrm.sh

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* rm extra import

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* train_paths

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* text_processing

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* fix ngram_bin_path

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* DECODERS_PATH

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* farcompile

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* rm text processing

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* text_processing

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* AggregateTokenizer.DummyTokenizer

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* comments

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* TextProcessingConfig

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* typo

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* doc

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* types

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* nemo_model_file

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* rm assert

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* import kenlm_utils

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* return None

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* Copyright

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* 2022

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

* 2023

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>

---------

Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>
Signed-off-by: Nikolay Karpov <karpnv@gmail.com>
Co-authored-by: Nikolay Karpov <nkarpov@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* fix for running on 1 GPU.

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* temp rtd fix (#6568) (#6569)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [TTS] Add script for mapping speaker names to indices (#6509)

Signed-off-by: Ryan <rlangman@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* whitespace (#6574)

Signed-off-by: Nikolay Karpov <karpnv@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Update manifest.py for speedup (#6565) (#6573)

* Update manifest.py

Re-order the checks for faster processing audio filepaths that are already absolute paths



* Update manifest.py



---------

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Co-authored-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* More streaming conformer export fixes (#6567) (#6578)

Signed-off-by: Greg Clark <grclark@nvidia.com>
Co-authored-by: Greg Clark <grclark@nvidia.com>
Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* user selected max_seq_len should be less than model's max_seq_len (#6333) (#6386)

* user selection should not break model max limit



* eval max seq length



---------

Signed-off-by: arendu <adithya.r@gmail.com>
Signed-off-by: Adi Renduchintala <108822655+arendu@users.noreply.github.com>
Co-authored-by: Adi Renduchintala <108822655+arendu@users.noreply.github.com>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>
Co-authored-by: Eric Harper <complex451@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Framework for PEFT via mixins  (#6391)

* init commit ptuning via mixin

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updates

Signed-off-by: arendu <adithya.r@gmail.com>

* gpt ptuning places virtual tokens on the left only

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* encoder input modified when pre_process is true

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* optimizer group and state dict updates

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* adapter ptuning working for pp>1

Signed-off-by: arendu <adithya.r@gmail.com>

* adapter defaults

Signed-off-by: arendu <adithya.r@gmail.com>

* adapter ptuining config defaults

Signed-off-by: arendu <adithya.r@gmail.com>

* training works

Signed-off-by: arendu <adithya.r@gmail.com>

* loading and saving adapter only params during training

Signed-off-by: arendu <adithya.r@gmail.com>

* added checks and comments

Signed-off-by: arendu <adithya.r@gmail.com>

* clean up

Signed-off-by: arendu <adithya.r@gmail.com>

* checks for grad is None before calling all_reduce

Signed-off-by: arendu <adithya.r@gmail.com>

* load adapter .nemo file working

Signed-off-by: arendu <adithya.r@gmail.com>

* resume training for adapters

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* peft tuning

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* minor

Signed-off-by: arendu <adithya.r@gmail.com>

* file not needed

Signed-off-by: arendu <adithya.r@gmail.com>

* undo prompt learning dataset changes

Signed-off-by: arendu <adithya.r@gmail.com>

* undo updates to gpt prompt learning model

Signed-off-by: arendu <adithya.r@gmail.com>

* naming updates

Signed-off-by: arendu <adithya.r@gmail.com>

* decoding

Signed-off-by: arendu <adithya.r@gmail.com>

* predict_step in gpt_sft_model

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updates

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* removed inference from tuning config

Signed-off-by: arendu <adithya.r@gmail.com>

* no test in peft training

Signed-off-by: arendu <adithya.r@gmail.com>

* answer only loss and correct defaults for val_loss

Signed-off-by: arendu <adithya.r@gmail.com>

* hybrid adapters and ptuning

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* eval working..

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* prepending tokens for ptuning

Signed-off-by: arendu <adithya.r@gmail.com>

* cleaned up eval config

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* clean up

Signed-off-by: arendu <adithya.r@gmail.com>

* update

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* default prompt template

Signed-off-by: arendu <adithya.r@gmail.com>

* Lora added

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Support synamic length with GPT SFT

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* make branch functional

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* defaults to max_pad_length=False in GPT SFT dataset

Signed-off-by: arendu <adithya.r@gmail.com>

* adapter parallel_adapters to support Lora

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* added early stopping by default

Signed-off-by: arendu <adithya.r@gmail.com>

* eval script for peft and eval config. bug fixes in predict step and added out_features to t5 adapter config

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updates

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updates

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* docs

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* better defaults

Signed-off-by: arendu <adithya.r@gmail.com>

* updates

Signed-off-by: arendu <adithya.r@gmail.com>

* update

Signed-off-by: arendu <adithya.r@gmail.com>

* docs

Signed-off-by: arendu <adithya.r@gmail.com>

---------

Signed-off-by: arendu <adithya.r@gmail.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: Adi Renduchintala <108822655+arendu@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* cache and reuse inputs (#6422) (#6452)

Co-authored-by: Sangkug Lym <slym@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add patches for Virtual Parallel conversion (#6589)

* Add patches for Virtual Parllel conversion

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Pass `.scale` instead of scaler object to core (#6551)

* pass .scale instead of scaler object to core (#6545)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Update megatron_gpt_model.py

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* scale changes for main

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Documentation for ASR-TTS models (#6594) (#6595)

* Add docs about hybrid ASR-TTS models



* Add docs about text-only datasets



* Add docs about ASR-TTS checkpoints



* Add docs about ASR-TTS configs and training



* Clean up



* ASR-TTS docs: add to api, fix imports



* Clean up



* Wrap optional import



* Revert general ASR import



---------

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
Co-authored-by: Vladimir Bataev <vbataev@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [TTS] Fix aligner nan loss in fp32 (#6435)

* Fix nan loss in fp32

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Update SDP docs (#6485) (#6596)

* add info about SDP e.g. processor classes in docs



* add link to SDP docs in README



* address code review comments and add SDP overview diagram



* Fix spelling typo



---------

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>
Co-authored-by: Elena Rastorgueva <80532067+erastorgueva-nv@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Bug/typo fixes (#6599)

Signed-off-by: Igor Gitman <igitman@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Manual garbage collection with an interval (#6469) (#6482)

* Manual garbage collection with an interval



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* use trainer.global_step for tracking the interval of GC



---------

Signed-off-by: Sangkug Lym <slym@nvidia.com>
Co-authored-by: Sangkug Lym <slym@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Make tensor split contiguous (#6580) (#6593)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [ASR] Fix for old models in change_attention_model (#6608)

* fixes

Signed-off-by: sam1373 <samuelkriman@gmail.com>

* done already

Signed-off-by: sam1373 <samuelkriman@gmail.com>

---------

Signed-off-by: sam1373 <samuelkriman@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Update manifest.py to use os.path for get_full_path (#6598)

* Update manifest.py to use os.path for get_full_path

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update manifest.py to get rid of pathlib

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update manifest.py

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* Update manifest.py

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Cherry pick commits in #6601 to main (#6611)

* fix write

Signed-off-by: fayejf <fayejf07@gmail.com>

* decoding ctc

Signed-off-by: fayejf <fayejf07@gmail.com>

* temp set rnnt decoding return_best_hypothesis to true

Signed-off-by: fayejf <fayejf07@gmail.com>

* add wer cal back to transcribe_speech as requested

Signed-off-by: fayejf <fayejf07@gmail.com>

* add wer cal back to speech_to_text_buffered_infer_rnnt  as requested

Signed-off-by: fayejf <fayejf07@gmail.com>

* add wer cal back to speech_to_text_buffered_infer_ctc as requested

Signed-off-by: fayejf <fayejf07@gmail.com>

* style fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* reflect change in asr_evaluator

Signed-off-by: fayejf <fayejf07@gmail.com>

* reflect som and vahid comment

Signed-off-by: fayejf <fayejf07@gmail.com>

* remove return_best_hy=true in transcribe_speech

Signed-off-by: fayejf <fayejf07@gmail.com>

* no text skip

Signed-off-by: fayejf <fayejf07@gmail.com>

* revert partial

Signed-off-by: fayejf <fayejf07@gmail.com>

---------

Signed-off-by: fayejf <fayejf07@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Create dummy iters to satisy len checks (#6600) (#6603)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* add GPT eval mode fix for interleaved to main (#6610)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix batch size reconf for T5 FT for multi-validation (#6582) (#6588)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Not doing CastToFloat by default (#6524) (#6563)

* Not doing CastToFloat by default



* Added docustring



* Dummy commit



---------

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
Co-authored-by: Boris Fomitchev <borisfom@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Turn autocast off when precision is fp32 (#6576)

* Turn autocast off when precision is fp32 (#6554)

* Turn autocast off when precision is fp32

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* address review

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* merge

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* correct auto-merge

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* correct auto-merge

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* add to GPT SFT

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* update core commit hash in readme (#6622) (#6623)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* add hat image to docs (#6619) (#6621)

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Allow indices exchange via distributed (#6618) (#6624)

Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com>
Co-authored-by: mikolajblaz <mikolajblaz@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Offline and streaming inference support for hybrid model (#6570)

* streaming buffered for hybrid + ctc

Signed-off-by: fayejf <fayejf07@gmail.com>

* change default model_stride in eval.yaml

Signed-off-by: fayejf <fayejf07@gmail.com>

* add fc model_stride

Signed-off-by: fayejf <fayejf07@gmail.com>

* small fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* check whether model and decoding match

Signed-off-by: fayejf <fayejf07@gmail.com>

* small fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* streaming buffered for hybrid + rnnt

Signed-off-by: fayejf <fayejf07@gmail.com>

* style fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* fix yaml

Signed-off-by: fayejf <fayejf07@gmail.com>

* reflect comment wip

Signed-off-by: fayejf <fayejf07@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* refactor and verified

Signed-off-by: fayejf <fayejf07@gmail.com>

* add get_full_path to buffered

Signed-off-by: fayejf <fayejf07@gmail.com>

* small fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* add RNNTDecodingConfig

Signed-off-by: fayejf <fayejf07@gmail.com>

* model name & instruction of changing decoding

Signed-off-by: fayejf <fayejf07@gmail.com>

---------

Signed-off-by: fayejf <fayejf07@gmail.com>
Signed-off-by: fayejf <36722593+fayejf@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Patch decoding for PC models (#6630) (#6631)

* Patch decoding logic for PC models



* Patch decoding logic for PC models



---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix wer.py where 'errors' variable was not set (#6633) (#6634)

Fix wer.py where 'errors' variable was not set when both reference and hypothesis are empty strings

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Co-authored-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Restore GPT support for interleaved pipeline parallelism (#6528) (#6613)

* Restore logic for data-parallel communication with pipeline parallelism in GPT



* Support dynamic attention masks in GPT



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Debug typos



* Debug data iterator caching with interleaved pipeline parallelism

Each model chunk accesses the data iterator multiple times, so we need to cache multiple samples.



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update Megatron-LM commit



* Distinguish between list of data iterators and data iterator that is a list



* Create dummy iters to satisy len checks



* Kludge while waiting for Megatron-LM update



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* set transformers offline to avoid rate limiting



---------

Signed-off-by: Tim Moon <tmoon@nvidia.com>
Signed-off-by: Eric Harper <complex451@gmail.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: ericharper <complex451@gmail.com>
Signed-off-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add FA

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix XPOS

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add warning

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix bugs

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix attention

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix comment

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix cast dtype

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Undo xpos

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* bugfix (#6636)

Signed-off-by: fayejf <fayejf07@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Disable interctc tests (#6638)

Signed-off-by: Igor Gitman <igitman@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add megatron_core to requirements (#6639) (#6640)

* add megatron_core to requirements



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: ericharper <complex451@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove from jenkins (#6642)

* Remove from jenkins (#6641)

* add megatron_core to requirements

Signed-off-by: ericharper <complex451@gmail.com>

* remove from jenkins

Signed-off-by: ericharper <complex451@gmail.com>

---------

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove dup

Signed-off-by: ericharper <complex451@gmail.com>

---------

Signed-off-by: ericharper <complex451@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* sft model can use this script for eval (#6637)

* sft model can use this script for eval

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* please fix me

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* minor

Signed-off-by: arendu <adithya.r@gmail.com>

---------

Signed-off-by: arendu <adithya.r@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [TTS] Fix TTS audio preprocessing bugs (#6628)

Signed-off-by: Ryan <rlangman@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Move black parameters to pyproject.toml (#6647)

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* ASR-TTS Models: Support hybrid RNNT-CTC, improve docs. (#6620)

* ASR-TTS: support hybrid RNNT-CTC models
* Do not warn on optional import
* Explain adding options to config
* Fix import guard docs
* Add docs for ConcatDataset
* Add explanation for sampling parameters
* Initial docs for the enhancer model
* Fix use_start_end_token parameter usage

---------

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* fix conversion and eval (#6648)

* fix conversion and eval

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: arendu <adithya.r@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Confidence ensembles implementation (#6614)

* Working version to train conf model + save ensemble class

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Working version

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Remove copy of transcribe_speech.py

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Move models parameter to config

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Add explicit parameters to transcribe

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Small cleanups

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Add temperature and integration tests

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Add more tests

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Add pc removal config

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Cleanup

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Fix typo

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Address review comments

Signed-off-by: Igor Gitman <igitman@nvidia.com>

---------

Signed-off-by: Igor Gitman <igitman@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Patch memory used for NeMo Megatron models (#6615)

* Patch memory used for NeMo Megatron models

Signed-off-by: smajumdar <titu1994@gmail.com>

* Cleanup the dtype of embeddings

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Refactor util function for parsing precision

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Refactor util function for parsing precision

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Try patch for Megatron O2

Signed-off-by: smajumdar <titu1994@gmail.com>

* Refactor to incorporate megatron amp 02 state

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Refactor to incorporate megatron amp 02 state

Signed-off-by: smajumdar <titu1994@gmail.com>

* Correct indent

Signed-off-by: smajumdar <titu1994@gmail.com>

* Correct utils import

Signed-off-by: smajumdar <titu1994@gmail.com>

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* handle artifacts when path is dir (#6658)

Signed-off-by: arendu <adithya.r@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* remove upgrading setuptools in reinstall.sh (#6659)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: fayejf <36722593+fayejf@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* merge lora weights into base model (#6597)

* merge lora weights into base model

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* typo fix

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* minor update

Signed-off-by: arendu <adithya.r@gmail.com>

* update copyright

Signed-off-by: arendu <adithya.r@gmail.com>

* eval needs to know the PEFT class

Signed-off-by: arendu <adithya.r@gmail.com>

* add target class in training script so that we can use it in eval

Signed-off-by: arendu <adithya.r@gmail.com>

* update

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update to work for tp1

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* set restore model path

Signed-off-by: arendu <adithya.r@gmail.com>

* peft can be none

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updated merge script so that eval works easily

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* eval with peft or sft model

Signed-off-by: arendu <adithya.r@gmail.com>

* keep sentences in jsonl format

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* convert sft using correct classpath

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updated to force sft yaml to have the correct target

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updated docs

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix conversion and eval

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: arendu <adithya.r@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* upgrade to 23.04 (#6660)

Signed-off-by: ericharper <complex451@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Merge r1.18.0 bugfixes and doc updates to main (#6655)

* update branch

Signed-off-by: ericharper <complex451@gmail.com>

* Remove from jenkins (#6641)

* add megatron_core to requirements

Signed-off-by: ericharper <complex451@gmail.com>

* remove from jenkins

Signed-off-by: ericharper <complex451@gmail.com>

---------

Signed-off-by: ericharper <complex451@gmail.com>

* remove dup

Signed-off-by: ericharper <complex451@gmail.com>

* update branch

Signed-off-by: ericharper <complex451@gmail.com>

* [TTS] reformat NeMo versions in the tts logging messages to avoid batch process them when upgrading NeMo versions.

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

---------

Signed-off-by: ericharper <complex451@gmail.com>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Confidence ensembles: fix issues and add tuning functionality (#6657)

* Implement compute confidence to properly handle blanks

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Implement proper confidence for transducers

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Implement tuning logic

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Add tests for confidence tuning

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Remove unused imports

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Add types/docs

Signed-off-by: Igor Gitman <igitman@nvidia.com>

* Add comment about the main conf compute loop

Signed-off-by: Igor Gitman <igitman@nvidia.com>

---------

Signed-off-by: Igor Gitman <igitman@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [TTS] Implement new TextToSpeech dataset (#6575)

* [TTS] Implement new TextToSpeech dataset

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Add unit tests

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Fix defaulting of use_log_energy

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Fix TTS export test

Signed-off-by: Ryan <rlangman@nvidia.com>

---------

Signed-off-by: Ryan <rlangman@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Dialogue dataset  (#6654)

* chatbot interface

Signed-off-by: Yi Dong <yidong@nvidia.com>

* latest gradio

Signed-off-by: Yi Dong <yidong@nvidia.com>

* default greedy

Signed-off-by: Yi Dong <yidong@nvidia.com>

* better chatbot

Signed-off-by: Yi Dong <yidong@nvidia.com>

* handle preamble

Signed-off-by: Yi Dong <yidong@nvidia.com>

* added chatbot training capablity

Signed-off-by: Yi Dong <yidong@nvidia.com>

* added chatbot ui

Signed-off-by: Yi Dong <yidong@nvidia.com>

* remove debug code

Signed-off-by: Yi Dong <yidong@nvidia.com>

* default human

Signed-off-by: Yi Dong <yidong@nvidia.com>

* use special token for roles

Signed-off-by: Yi Dong <yidong@nvidia.com>

* special tokens

Signed-off-by: Yi Dong <yidong@nvidia.com>

* fix name

Signed-off-by: Yi Dong <yidong@nvidia.com>

* new chat dataset

Signed-off-by: Yi Dong <yidong@nvidia.com>

* fix the system token

Signed-off-by: Yi Dong <yidong@nvidia.com>

* upgrade gradio

Signed-off-by: Yi Dong <yidong@nvidia.com>

* save the chat history

Signed-off-by: Yi Dong <yidong@nvidia.com>

* update ui

Signed-off-by: root <you@example.com>

* update chat interface

Signed-off-by: Yi Dong <yidong@nvidia.com>

* handles canonical form

Signed-off-by: Yi Dong <yidong@nvidia.com>

* new sft chatbot

Signed-off-by: Yi Dong <yidong@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* change format

Signed-off-by: Yi Dong <yidong@nvidia.com>

* check extra_id in the tokenizer

Signed-off-by: Yi Dong <yidong@nvidia.com>

* added vocab property check

Signed-off-by: Yi Dong <yidong@nvidia.com>

* added missing file

Signed-off-by: Yi Dong <yidong@nvidia.com>

---------

Signed-off-by: Yi Dong <yidong@nvidia.com>
Signed-off-by: root <you@example.com>
Co-authored-by: root <you@example.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add support for RNNT/hybrid models to partial transcribe (#6609)

* Add support for RNNT/hybrid models to partial transcribe

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* Update transcribe_utils.py

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* Update transcribe_speech.py

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* Update transcr…
KunalDhawan added a commit that referenced this pull request Jul 20, 2023
* peft eval directly from ckpt (#6785)

* update to load from ckpt

Signed-off-by: arendu <adithya.r@gmail.com>

* update

Signed-off-by: arendu <adithya.r@gmail.com>

* load ckpt peft model

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update style

Signed-off-by: arendu <adithya.r@gmail.com>

---------

Signed-off-by: arendu <adithya.r@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Add Frame-VAD examples and utils (#6463)

* add model, dataset, necessary utils and tests

Signed-off-by: stevehuang52 <heh@nvidia.com>

* fix tarred data

Signed-off-by: stevehuang52 <heh@nvidia.com>

* fix typo

Signed-off-by: stevehuang52 <heh@nvidia.com>

* add fvad examples and update utils

Signed-off-by: stevehuang52 <heh@nvidia.com>

* add copyright

Signed-off-by: stevehuang52 <heh@nvidia.com>

* refactor and add tests

Signed-off-by: stevehuang52 <heh@nvidia.com>

* update dataset

Signed-off-by: stevehuang52 <heh@nvidia.com>

* update test

Signed-off-by: stevehuang52 <heh@nvidia.com>

* refactor

Signed-off-by: stevehuang52 <heh@nvidia.com>

* refactor

Signed-off-by: stevehuang52 <heh@nvidia.com>

* fix typos

Signed-off-by: stevehuang52 <heh@nvidia.com>

---------

Signed-off-by: stevehuang52 <heh@nvidia.com>
Co-authored-by: fayejf <36722593+fayejf@users.noreply.github.com>
Co-authored-by: Taejin Park <tango4j@gmail.com>

* [TTS][zh] refine hardcoded lowercase for ASCII letters. (#6781)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* Spellchecking ASR customization model (#6179)

* bug fixes

Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com>

* fix bugs, add preparation and evaluation scripts, add readme

Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com>

* small fixes

Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add real coverage calculation, small fixes, more debug information

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add option to pass a filelist and output folder - to handle inference from multiple input files

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* added preprocessing for yago wikipedia articles - finding yago entities and their subphrases

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* yago wiki preprocessing, sampling, pseudonormalization

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* more scripts for preparation of training examples

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* bug fixes

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add some alphabet checks

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add bert on subwords, concatenate it to bert on characters

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add calculation of character_pos_to_subword_pos

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* bug fix

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* bug fix

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* pdb

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* tensor join bug fix

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* double hidden_size in classifier

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* pdb

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* default index value 0 instead of -1 because index cannot be negative

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* pad index value 0 instead of -1 because index cannot be negative

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* remove pdb

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix bugs, add creation of tarred dataset

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add possibility to change sequence len at inference

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* change sampling of dummy candidates at inference, add candidate info file

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix import

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix bug

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* update transcription now uses info

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* write path

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* 1. add tarred dataset support(untested). 2. fix bug with ban_ngrams in indexing

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* skip short_sent if no real candidates

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix import

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add braceexpand

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fixes

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix bug

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix bug

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix bug in np.ones

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix bug in collate

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* change tensor type to long because of error in torch.gather

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix for empty spans tensor

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* same fixes in _collate_fn for tarred dataset

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix bug from previous commit

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* change int types to be shorter to minimize tar size

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* refactoring of datasets and inference

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* bug fix

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* bug fix

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* bug fix

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* tar by 100k examples, small fixes

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* small fixes, add analytics script

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* Add functions for dynamic programming comparison to get best path by ngrams

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fixes

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* small fix

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fixes to support testing on SPGISpeech

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add preprocessing for userlibri

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* some refactoring

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* some refactoring

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* move some functions to utils to reuse from other project

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* move some functions to utils to reuse from other project

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* move some functions to utils to reuse from other project

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* small refactoring before pr. Add bash-scripts reproducing evaluation

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* style fix

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* small fixes in inference

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* bug fix - didn't move window on last symbol

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix bug - shuffle was before truncation of sorted candidates

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* refactoring, fix some bugs

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* variour fixes. Add word_indices at inference

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add candidate positions

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Move data preparation and evaluation to other repo

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add infer_reproduce_paper. Refactoring

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* refactor inference using fragment indices

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add some helper functions

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix bug with parameters order

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix bugs

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* refactoring, fix bug

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add multiple variants of adjusting start/end positions

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* more fixes

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add unit tests, other fixes

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix CodeQl warnings

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* bug fixes

Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com>

* fix bugs, add preparation and evaluation scripts, add readme

Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com>

* small fixes

Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add real coverage calculation, small fixes, more debug information

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add option to pass a filelist and output folder - to handle inference from multiple input files

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* added preprocessing for yago wikipedia articles - finding yago entities and their subphrases

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* yago wiki preprocessing, sampling, pseudonormalization

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* more scripts for preparation of training examples

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* bug fixes

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add some alphabet checks

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add bert on subwords, concatenate it to bert on characters

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add calculation of character_pos_to_subword_pos

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* bug fix

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* bug fix

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* pdb

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* tensor join bug fix

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* double hidden_size in classifier

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* pdb

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* default index value 0 instead of -1 because index cannot be negative

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* pad index value 0 instead of -1 because index cannot be negative

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* remove pdb

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix bugs, add creation of tarred dataset

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add possibility to change sequence len at inference

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* change sampling of dummy candidates at inference, add candidate info file

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix import

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix bug

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* update transcription now uses info

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* write path

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* 1. add tarred dataset support(untested). 2. fix bug with ban_ngrams in indexing

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* skip short_sent if no real candidates

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix import

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add braceexpand

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fixes

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix bug

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix bug

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix bug in np.ones

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix bug in collate

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* change tensor type to long because of error in torch.gather

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix for empty spans tensor

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* same fixes in _collate_fn for tarred dataset

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix bug from previous commit

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* change int types to be shorter to minimize tar size

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* refactoring of datasets and inference

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* bug fix

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* bug fix

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* bug fix

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* tar by 100k examples, small fixes

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* small fixes, add analytics script

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* Add functions for dynamic programming comparison to get best path by ngrams

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fixes

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* small fix

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fixes to support testing on SPGISpeech

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add preprocessing for userlibri

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* some refactoring

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* some refactoring

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* move some functions to utils to reuse from other project

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* move some functions to utils to reuse from other project

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* move some functions to utils to reuse from other project

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* small refactoring before pr. Add bash-scripts reproducing evaluation

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* style fix

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* small fixes in inference

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* bug fix - didn't move window on last symbol

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix bug - shuffle was before truncation of sorted candidates

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* refactoring, fix some bugs

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* variour fixes. Add word_indices at inference

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add candidate positions

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Move data preparation and evaluation to other repo

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add infer_reproduce_paper. Refactoring

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* refactor inference using fragment indices

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add some helper functions

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix bug with parameters order

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix bugs

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* refactoring, fix bug

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add multiple variants of adjusting start/end positions

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* more fixes

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add unit tests, other fixes

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix CodeQl warnings

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add script for full inference pipeline, refactoring

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add tutorial

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* take example data from HuggingFace

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add docs

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix comment

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* fix bug

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* small fixes for PR

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add some more tests

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* try to fix tests adding with_downloads

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* skip tests with tokenizer download

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

---------

Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com>
Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>
Co-authored-by: Alexandra Antonova <aleksandraa@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [TTS] Implement new vocoder dataset (#6670)

* [TTS] Implement new vocoder dataset

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Redo config structure, minor fixes

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Fix alignment logging

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Fix script usage example

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Fixed epoch LR scheduling

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Support .nemo checkpoint in FP callback

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Remove align interpolator

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Remove HiFi-GAN defaults list interpolation

Signed-off-by: Ryan <rlangman@nvidia.com>

* [TTS] Rename weighted_sample_steps to weighted_sampling_steps_per_epoch

Signed-off-by: Ryan <rlangman@nvidia.com>

---------

Signed-off-by: Ryan <rlangman@nvidia.com>

* GPT inference long context (#6687)

* deb infer

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* deb infer

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* clean up

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* dont do maxlen trunc for non abs pos emb

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* dont do maxlen trunc for non abs pos emb

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* convert for training only

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add eval test, add save .nemo for sft model

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* jenkins format fix

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* update jenkins

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* update jenkins

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* fix jenkins

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* remove test, ci timeout

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* fix for m_gpt_eval.py

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* jenkins test

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* fix gpt_eval with sft model

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* revert jenkins

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* keep float conversion for model.generate()

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix inference dtype

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Evelina <ebakhturina@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* TDT model pull request (#6536)

* TDT model pull request, initial draft

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* TDT PR WIP

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* TDT PR WIP

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* TDT PR WIP

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* TDT WIP

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* TDT WIP

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* TDT WIP

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* TDT WIP

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* TDT WIP

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* TDT WIP

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* TDT WIP

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* TDT WIP

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* TDT WIP

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* TDT WIP

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* TDT WIP

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* addressed some review comments, part1

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* addressed some review comments, part1, one line fix

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add tests for comparing TDT alphas with pytorch VS kernel computation

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add tests for comparing multiblank alphas with pytorch VS kernel computation

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add tests for fixed case computation for TDT

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add more comments for greedy-batch decoding for TDT

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* include config for TDT model with stateless decoders

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* add reference to TDT in Readme

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* slight modification of config file comments

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* addressed more comments

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* more detailed comments for tdt kernel

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* one line fix

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* fixed small bug that results in test fails for rnnt_decoding

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* fixed small bug that results in test fails for rnnt_decoding

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed small bug that results in test fails for rnnt_decoding

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* remove unused import

Signed-off-by: Hainan Xu <hainanx@nvidia.com>

---------

Signed-off-by: Hainan Xu <hainanx@nvidia.com>
Co-authored-by: Hainan Xu <hainanx@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Fix get_parameters when using main params optimizer (#6764) (#6787)

* fix get param



* change name



---------

Signed-off-by: ericharper <complex451@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Lddl bert (#6761) (#6790)

* initial POC for LDDL Bert

* Finish LDDL POC

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* address comments

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix merge head

* resolving merge

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add support for  val/test loaders

* change to new LDDL class + add winding

* fix logging level

* fix winding

* test fix

* fixes to winding

* add file system

* add prepemption optimizations

* more logging

* more prints

* better logging

* asfsf

* add barrier

* removing prints

* working with mb lddl loader

* final changes

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update requirements file with LDDL



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* revert adding to requirements

---------

Signed-off-by: wdykas <wdykas@nvidia.com>
Co-authored-by: wdykas <73254672+wdykas@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Fix check (#6798) (#6800)

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>

* Fix validation with drop_last=False (#6704)

Signed-off-by: Mikołaj Błaż <mblaz@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* SDE unt lvl comparison (#6669)

Added a visual utterance-level comparison of two ASR models

Signed-off-by: George <gzelenfroind@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Debug Transformer Engine FP8 support with Megatron-core infrastructure (#6791)

* Construct FP8 amax reduction group

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Update Megatron-core version in CI

Signed-off-by: Tim Moon <tmoon@nvidia.com>

---------

Signed-off-by: Tim Moon <tmoon@nvidia.com>
Co-authored-by: Tim Moon <tmoon@nvidia.com>
Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>

* Lora/PEFT training script CI test (#6664)

* new lora test

Signed-off-by: arendu <adithya.r@gmail.com>

* updates

Signed-off-by: arendu <adithya.r@gmail.com>

* check for chat

Signed-off-by: arendu <adithya.r@gmail.com>

* update

Signed-off-by: arendu <adithya.r@gmail.com>

* update

Signed-off-by: arendu <adithya.r@gmail.com>

* small train set

Signed-off-by: arendu <adithya.r@gmail.com>

* update

Signed-off-by: arendu <adithya.r@gmail.com>

* precision change

Signed-off-by: arendu <adithya.r@gmail.com>

* fixed typo in paths

Signed-off-by: arendu <adithya.r@gmail.com>

* full data with limit val batches

Signed-off-by: arendu <adithya.r@gmail.com>

* tp2 instead of pp2

Signed-off-by: arendu <adithya.r@gmail.com>

* tp2 instead of pp2

Signed-off-by: arendu <adithya.r@gmail.com>

---------

Signed-off-by: arendu <adithya.r@gmail.com>
Signed-off-by: Adi Renduchintala <adithya.r@gmail.com>

* change branch to main, small fix (#6803)

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add call to p2p overlap (#6779) (#6786)

* add call to p2p overlap



* update Jenkins for test



---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* fixed  decor to show messages only when the wrapped object is called. (#6793)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* Bug fix for reset_sequence_parallel_args (#6802) (#6805)

Signed-off-by: Markel Sanz Ausin <markelsanz14@gmail.com>
Co-authored-by: Markel Sanz Ausin <markelsanz14@gmail.com>

* text_generation_utils memory reduction if no logprob needed (#6773)

* repro for gpt eval mp mem issue

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* add print statements for memory allocation

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* adjusted hot fix that prevents softmax on the entire output embedding,now memory bottlenecked by attention softmax which needs to be solved with FA or long attention

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* using compute_logprob to configure inference

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* enable compute logprob for peft

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* remove print statements

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix ci

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* added docstrings

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add missing config

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* remove truncate prompt length feature

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tensor before all gather needs to be contiguous

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

---------

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Evelina <10428420+ekmb@users.noreply.github.com>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>

* Fixed bug in MaskedSpecAug that overestimates samples. (#6775)

Signed-off-by: tbartley94 <tbartley@nvidia.com>

* update core version (#6817) (#6819)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* lora pp2 (#6818)

Signed-off-by: arendu <adithya.r@gmail.com>

* Add optional index mapping dir in mmap text datasets (#6683)

If datasets are stored on a read-only medium, index files
cannot be created into adjacent files and an
alternative directory must be specified for index
mapping files.

This commit adds an optional `index_mapping_dir` to
the constructors.
Unit tests are also added.



[pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Update path formatting for relative paths

Signed-off-by: Greg Heinrich <gheinrich@nvidia.com>

* Add inference kv cache support for transformer TE path (#6627)

* Add kv cache support for transformer TE path

Signed-off-by: Yen-Shi Wang <yenshiw@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Mark get_data_parallel_group as WAR

Signed-off-by: Yen-Shi Wang <yenshiw@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Initialize process group for FP8 training

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Update Megatron GPT eval script for non-FP8 path

Signed-off-by: Yen-Shi Wang <yenshiw@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Yen-Shi Wang <yenshiw@nvidia.com>
Signed-off-by: Tim Moon <tmoon@nvidia.com>
Signed-off-by: Yen-Shi Wang <6960565+yen-shi@users.noreply.github.com>
Co-authored-by: Yen-Shi Wang <yenshiw@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Tim Moon <tmoon@nvidia.com>
Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Support large inputs to Conformer and Fast Conformer (#6556)

* initial commit

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* typos

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* tweaks to padding

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* comments

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* attempt at first working version

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* typos and fixed p calculation

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* removing merge artifacts

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* typo

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* removing unnessary imports

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* if batch split succeeded no need to conv again

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* adding channel wise split

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* adding reference to pytorch issue 80020

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* removing time chunking methods

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* accounting for the actual self._stride value

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* limiting the fix to dw_striding subsampling

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* renamed methods

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* one more accounting for the actual self._stride value

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* support for causal convs

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* option to set conv chunking size manually

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixing imports

* subsampling test

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rename variable

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* imports in test

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* more runtime checks

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* a more careful test

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* bug in causal

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix in causal

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* change_conv_chunking_factor methods

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* renamed methods

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* disabling chunking by default

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* typo

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* changing default chunking to auto

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* only split if needed

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* only split if needed

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

---------

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* sharded_manifests updated docs (#6833)

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* added fc-xl, xxl and titanet-s models (#6832)

Signed-off-by: Nithin Rao Koluguri <nithinraok>
Co-authored-by: Nithin Rao Koluguri <nithinraok>

* add reference to our paper (#6821)

* add reference to our paper

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* add paper reference to docs

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

---------

Signed-off-by: Alexandra Antonova <antonova_sasha@list.ru>

* Upperbound Numpy to < 1.24 (#6829)

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Multi-lookahead cache-aware streaming models (#6711)

* added methods.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added methods.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added initial code.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added initial code.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added initial code.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added config files.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed bugs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated confs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated confs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated confs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated confs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* improved f.conv1d

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* pulled from main.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* pulled from main.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added postpostnorm.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed the target continiouse bug.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added dw_striding causal.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added print for debugging.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added print for debugging.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed causal convolutions.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added _midnorm.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed transcribe.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* cleaned code.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* moved back configs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* moved back configs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated fast emit for FC models.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated fast emit for FC models.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed bug.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed bug and addressed comments.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed configs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed configs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* dropped the test.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

---------

Signed-off-by: Vahid <vnoroozi@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* added changes to ramp up bs  (#6799)

* rampup bs changes

Signed-off-by: dimapihtar <dpykhtar@nvidia.com>

* rampup bs changes

Signed-off-by: dimapihtar <dpykhtar@nvidia.com>

* fixed styling

Signed-off-by: dimapihtar <dpykhtar@nvidia.com>

* fix bug

Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>

---------

Signed-off-by: dimapihtar <dpykhtar@nvidia.com>
Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Fix typo in core.rst (#6838)

Signed-off-by: Dounx <imdounx@gmail.com>

* add back ptuning pp2 test (#6394)

Signed-off-by: arendu <adithya.r@gmail.com>

* t5 lora tuning (#6612)

* t5 lora

Signed-off-by: arendu <adithya.r@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* eval lora t5

Signed-off-by: arendu <adithya.r@gmail.com>

* adjust differernt lora dims

Signed-off-by: arendu <adithya.r@gmail.com>

* minor changes

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* bugfix for state_dict

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

---------

Signed-off-by: arendu <adithya.r@gmail.com>
Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: David Mosallanezhad <dmosallanezh@nvidia.com>
Co-authored-by: David <amosalla@asu.edu>

* NFA updates (#6695)

* update V_NEGATIVE_NUM constant to make better use of torch.float32 range

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* adjust backpointers dtype if U_max too large

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* Remove print statements

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* Remove need for user to specify model_downsample_factor

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* change model.cfg.sample_rate to model.cfg.preprocessor.sample_rate

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* add check to make sure that window_stride is in model.cfg.preprocessor

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* reduce memory consumption of backpointers by making them relative instead of absolute

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* update librosa.get_duration() 'filename' param to 'path'

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* Do not throw error if 'text' or 'pred_text' are empty and make sure CTM filepaths in the output manifest are null

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* preprocess input text by removing any duplicate spaces and converting any newlines to spaces

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* Use Utterance dataclass instead of dictionaries for keeping track of token/word/segment alignments

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* refactor so can save alignments as ctm and ass format files

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* fix bugs for saving character based ASS files and for using pred_text to do alignment

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* Make token level .ass file use tokens with recovered capitalization

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* Do not try to generate alignment files if text or pred text is empty, or if number of tokens is too large for T

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* rename output manifest file to say '_with_output_file_paths.json'

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* add flag to resegment ass subtitle file to fill available text space

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* Fix bug in resegmentation code

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* Fix bug which skipped some utterances if batch_size more than 1

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* reduce memory requirements by doing torch.gather on a slice of the log probs when they are needed

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* reduce memory requirements by not saving whole v_matrix

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* remove any extra spaces in pred_text

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove unused list pred_text_all_lines

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* support using hybrid Transducer-CTC models for alignment

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* fix typo - add brackets to torch.cuda.is_available()

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* make sure token case restoration will work if superscript or subscript num is in text

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* remove any BOM from input text

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* pick out 1st hypotheses if there is a tuple of them

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* Remove print statement

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* add detail to error message if fail to recover capitalization of tokens

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* add flag use_local_attention

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* rename additional_ctm_grouping_separator -> additional_segment_grouping_separator

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* update description of additional_segment_grouping_separator

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* add simple docstring to get_utt_obj function

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* Make docstring for add_t_start_end_to_utt_obj

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* update docstrings for add_t_start_end_to_utt_obj and get_batch_variables

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* update README and comments in align.py

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* change 'ground truth' -> 'reference text' in documentation

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* add header

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* add comments to get_utt_obj function

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* move constants so they are after imports

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* add file description for make_ass_files

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* get rid of Utterance object's S attribute, and correct tests so they pass now

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* remove some unused variables

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* remove unused variable model from functions saving output files

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* remove unused var minimum_timestamp_duration from make_ass_files functions and return utt_obj

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* move minimum_timestamp_duration param to CTMFileConfig

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* remove unused enumerate and unused import

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* switch reading duration from librosa to soundfile to avoid filename/path deprecation message

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

---------

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>
Signed-off-by: Elena Rastorgueva <80532067+erastorgueva-nv@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Added rouge monitoring support for T5 (#6737)

* Added rouge monitoring support for t5

Signed-off-by: Matvei Novikov <mattyson.so@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Matvei Novikov <mattyson.so@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* GPT extrapolatable position embedding (xpos/sandwich/alibi/kerple) and Flash Attention (#6666)

* move to nvidia megatron repo (#6465) (#6475)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Megatron KERPLE positional embeddings (#6478) (#6480)

* [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

[TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

---------




* [TTS] whitelist broken path fix. (#6412)

* [TTS] whitelist broken path fix.



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------




* [TTS] FastPitch speaker encoder (#6417)

* Add initial codes



* Remove wemb



* Fix import



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Restore aligner loss



* Add ConditionalInput



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix error and support pre-trained config



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Follow comments



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Rename config



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Change copyright and random weight test



* Add initial codes



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix import error



* Add initial codes



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix dataset error



* Remove reference speaker embedding



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Remove SV encoder



* Follow comments



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix length type



* Fix append



* Move error msg



* Add look-up into speaker encoder



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Add valueerror msg



* Move lookup



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Remove unused



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix error



* Rebase and Fix error



* Fix spk encoder



* Rename n_speakers



* Follow comments



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix n_speakers None error



---------




* Sharded manifests for tarred datasets (#6395)

* testing sharded manifests



* compatibility



* proper fixes



* adding flag tot convert_to_tarred_audio_dataset



* shard_manifests conf param



* propagating the shard_manifests param



* propagating the shard_manifests param



* distributed checks



* typo



* typo



* fixes



* fixes



* fixes



* fixes



* fixes



* fixes



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes based on PR comments and tests



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes to convert_to_tarred_audio_dataset.py



* reversing manifest shards flag



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests



* excluding manifests from webdataset url expansion



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* expand manifest paths before attempting to cache from datastore



* explicit use of UTF-8 for manifest i/o



---------




* Update wfst_text_normalization.rst (#6374)

Add Hungarian (incoming in NeMo-text-processing)



* Support Swiglu in TP PP Conversion (#6437) (#6451)

* Support Swiglu in TP PP Conversion



* Guard activation



* Guard activation



---------




* Update NeMo_TTS_Primer.ipynb (#6436)

* Update NeMo_TTS_Primer.ipynb

Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch.



* Update NeMo_TTS_Primer.ipynb

Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point.



---------



* add rampup batch size support for Megatron GPT (#6424)

* added rampup batch size support



* added tests for rampup batch size



* fixed the typos



* added assertions



* changed assertion rules



* deleted unused imports



* changed tests for rampup batch size



* updated rampup batch size tests



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed styling



* rampup batch size tests changes



---------







* Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461)

* 1. Meagtron encoder decoder fix for empty validation outputs.



* 1. Debugging.

---------





* Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448)

* added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default



* updated README with the new agg_tokenizer_manifest flag



* fixed typo in scripts/speech_recognition/code_switching/README.md



* changed agg_tokenizer_manifest to is_lid_manifest



---------




* Added/updated new Conformer configs (#6426) (#6467)

* Update script for ngram rnnt and hat beam search decoding (#6370)

* add rnnt ngram beamsearch script



* add return encoding embedding option



* update script



* add rnnt and hat ngram decoding script



* add some parameters



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add return_encoder_embeddings parameter to RNNTDecodingConfig



* replace return_encoder_embeddings parameter



* generalization of scipt behavior



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove return_encoder_embeddings parameter



* remove return_encoder_embeddings parameter



* add manual encoder_embeddings calculation



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix beam_width value to 8



* fix rescoring description



---------






* BERT pre-training mp fork to spawn (#6442) (#6454)

* change bert fork to spawn



* num_workers=0 fix



---------




* fix replace_bos_with_pad not found (#6443) (#6450)




* reduce workers on NMT CI (#6472) (#6474)




* 1. Added KERPLE positional embeddings to encoder-decoder.



* 1. Added a missing file.



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Fixing commits.



* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Signed-off-by: Jim O’Regan <jaoregan@tcd.ie>
Signed-off-by: smajumdar <titu1994@gmail.com>
Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Signed-off-by: Micha Livne <mlivne@nvidia.com>
Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Micha Livne <michalivne@users.noreply.github.com>
Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Dima Rekesh <bmwshop@gmail.com>
Co-authored-by: Jim O’Regan <jaoregan@tcd.ie>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix an invalid link in get_data.py of ljspeech (#6456)

Usage of the link in line 63 leads to downloading a html file not a tsv file, so we need to change it to a raw link.

Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* 1. Added external index sample. (#6462) (#6483)

Signed-off-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: Micha Livne <michalivne@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Update README to add core installation (#6488) (#6489)

* update README for megatron-core



* fix



---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix cache aware hybrid bugs (#6466) (#6484)

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix typos (#6494) (#6495)

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add disclaimer about dataset for ASR (#6496)

Signed-off-by: smajumdar <titu1994@gmail.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* fix (#6502)

datastore_path_to_webdataset_url(p) if is_datastore_path(p) and is_tarred_path(p) else p
NameError: name 'is_tarred_path' is not defined

Co-authored-by: George <gzelenfroind@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* fix broken links r1.18.0 (#6501) (#6504)

* fix broken links



* fix broken links



---------

Signed-off-by: Evelina <ebakhturina@nvidia.com>
Co-authored-by: Evelina <10428420+ekmb@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [TTS] Create functions for TTS preprocessing without dataloader (#6317)

* [TTS] Create functions for TTS preprocessing without dataloader

Signed-off-by: Ryan <rlangman@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Cache aware streaming nfa (#6209)

* add cache aware streaming to nemo aligner

Signed-off-by: Slyne Deng <slyned@nvidia.com>

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [BugFix] Force _get_batch_preds() to keep logits in decoder timestamps generator (#6499)

* [BugFix] _get_batch_preds() is forced to keep logits in  decoder timestamps generators

Signed-off-by: Taejin Park <tango4j@gmail.com>

* Ingnore keep_logits boolean in FrameASRBatchLogits

Signed-off-by: Taejin Park <tango4j@gmail.com>

---------

Signed-off-by: Taejin Park <tango4j@gmail.com>
Co-authored-by: Jagadeesh Balam <4916480+jbalam-nv@users.noreply.github.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

…
ericharper added a commit that referenced this pull request Aug 11, 2023
* [TTS] bugfix for missing configs. (#4725)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* docs typo fix

Signed-off-by: Oleksii Kuchaiev <okuchaiev@nvidia.com>

* Fix pynini install in TTS tutorials (#4729)

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

* Fix ASR notebooks (#4738)

Signed-off-by: smajumdar <smajumdar@nvidia.com>

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Multilingual VAD model (#4734)

* add ngc link

Signed-off-by: fayejf <fayejf07@gmail.com>

* add tuned VAD config on ASR data

Signed-off-by: fayejf <fayejf07@gmail.com>

* yaml note

Signed-off-by: fayejf <fayejf07@gmail.com>

* update vad asr notebook with mVAD

Signed-off-by: fayejf <fayejf07@gmail.com>

* update vad infer config comment

Signed-off-by: fayejf <fayejf07@gmail.com>

* fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* mvad sd config for ch109

Signed-off-by: fayejf <fayejf07@gmail.com>

* update sd readme

Signed-off-by: fayejf <fayejf07@gmail.com>

* add new mVAD model to doc

Signed-off-by: fayejf <fayejf07@gmail.com>

* style fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* update sd tutorial with mVAD

Signed-off-by: fayejf <fayejf07@gmail.com>

* typo fix

Signed-off-by: fayejf <fayejf07@gmail.com>

Signed-off-by: fayejf <fayejf07@gmail.com>

* publish pretrained itn t5 model for English (#4748)

Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com>

Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com>
Co-authored-by: Alexandra Antonova <aleksandraa@nvidia.com>

* Updated docs and doc paths (#4754)

* Updated docs and doc paths

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Update Multitask_Prompt_and_PTuning.ipynb

* Update README.rst

* Changed branch name to use single quotes

Signed-off-by: Virginia Adams <vadams@nvidia.com>

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* fix bug relating to ddp strategy in joint intent slot classification tutorial (#4762)

* [TTS] updated config with a German IPA phoneme tokenizer (#4756)

* [TTS] added a German IPA phoneme tokenizer
* [TTS][ASR] enabled customized arguments for trimming the leading and trailing silence.
* [TTS] disabled spline interpolation for beta-binomial distribution. Let it generate align prior and save to disks. Use a new phoneme tokenizer.
* [TTS] use consistent spline interpolation with fastpitch checkpoint when generating mel-spectrograms for hifigan finetune.

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* Update r1.11 to new heteronyms list (#4745)

* Update configs to new heteronyms list
* Remove old heteronyms list, add alt 'merchandise' pron to CMUdict
* Update remaining references to old heteronyms list

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* [TTS] Add multi-speaker German FastPitch and HiFiGAN NGC checkpoints (#4763)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* [TTS] Add single male speaker German FastPitch and HiFiGAN NGC checkpoints (#4770)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* Update CMUdict with more recent 0.7b entries (#4768)

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* Install pynini in docker container (#4733)

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Fix tutorial formatting (#4778)

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

* [TTS] deprecated old scripts for ljspeech. (#4780)

* deprecated old scripts for ljspeech.
* removed relevent function calls in TTS docs.

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* update branch and typos (#4788)

Signed-off-by: ericharper <complex451@gmail.com>

Signed-off-by: ericharper <complex451@gmail.com>

* Adding support for models trained with full context for cache-aware streaming. (#4687)

* added support for models trained with full context.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* dropped seq_range

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed indexing in caching methods.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated docs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* addressed comments.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* change frame-wise to cache-aware.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* change frame-wise to cache-aware.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* change frame-wise to cache-aware.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* Update megatron encoder decoder model to support py37 for colab (#4791)

* [ASR] Add pretrained ASR models for Croatian (#4682)

* [ASR] Add pretrained ASR models for Croatian

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

* Fix style for import

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

Signed-off-by: Ante Jukić <ajukic@nvidia.com>
Co-authored-by: Ante Jukić <ajukic@nvidia.com>
Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>

* added/fixed export for Megatron models (#4712)

* added/fixed export for Megatron models

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

* fixed style

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

* fixed FusedScaleMaskSoftmax in BioMegatron

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

* included comments

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>
Co-authored-by: David Mosallanezhad <dmosallanezh@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* update branch for qa notebook

Signed-off-by: ericharper <complex451@gmail.com>

* Fix initializing weights from ptl ckpt with exclude (#4807)

Signed-off-by: sam1373 <samuelkriman@gmail.com>

Signed-off-by: sam1373 <samuelkriman@gmail.com>

* Fix index error from addition of voiced_mask and p_voiced (#4811)

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

* T5 prompt learning fixes (#4771)

* RPE, hidden size and config fixes

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update to reflect new config names

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Sentencepiece fixes

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix finetuning

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Add encoder seq len to gpt

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Add finetune eval script

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix name

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update Jenkinsfile

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update config

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix CI test

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update check

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Backward compat

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update CI test

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Split rank for Enc-Dec models

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Address comments

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Co-authored-by: Virginia Adams <78445382+vadam5@users.noreply.github.com>

* G2P docs (#4841)

* g2p docs added

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* fix references

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* address review feedback

Signed-off-by: ekmb <ebakhturina@nvidia.com>

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* Fix providing glue in seq2seq eval (#4843)

* Fix providing glue in seq2seq eval

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Updated inference code and squad scripts (#4835)

* Updated inference code and squad scripts

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Reverted GPT & T5 inference files back to use NLPDDPlugin

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Overwrite frozen LM to use fused adam

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Added padded vocab size

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Fixed val check interval value

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Python format fix

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Make t5 prompt learning preds write to file

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Added back dp=1 check

Signed-off-by: Virginia Adams <vadams@nvidia.com>

Signed-off-by: Virginia Adams <vadams@nvidia.com>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>

* Update README.rst

* Fix uppercasing mismatch for IPA heteronyms (#4860)

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

* Set the number of workers to 0 for validation and test sets in all enc-dec models (#4790)

* Set workers to 0 for validation and test

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Revert pin memory

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Co-authored-by: Sean Naren <sean.narenthiran@gmail.com>

* Fix mha (#4866)

* fix bug in mha forward function related to cache update return type

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* fix lgtm

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>
Co-authored-by: Sean Naren <snarenthiran@nvidia.com>

* ipa bug fix (#4871)

Signed-off-by: ekmb <ebakhturina@nvidia.com>

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* Fix Megatron NMT consumed samples and ckpt_to_nemo split rank (#4884)

* Fix nmt and ckpt_to_nemo

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* added utf8 encoding (#4892)

Signed-off-by: Virginia Adams <vadams@nvidia.com>

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* 1. Applying the same patch to r1.11.0 (#4894)

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

* Update tutorials.rst (#4897)

* update readme with apex commit

Signed-off-by: ericharper <complex451@gmail.com>

* Add support for Apex distributed Adam optimizer with GPT-3 (#4487)

* Add support for Apex distributed Adam optimizer with GPT-3

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Fix bug in grad clipping with dist Adam

Grad norm was computed over all params, not respecting model parallelism.

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Fix bug with DDP initialization

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Make distopt dependent on megatron_amp_o2

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Fix code formatting

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Handle dist Adam in optimizer unit tests

Signed-off-by: Tim Moon <tmoon@nvidia.com>

Signed-off-by: Tim Moon <tmoon@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* update readme

Signed-off-by: ericharper <complex451@gmail.com>

* update readme

Signed-off-by: ericharper <complex451@gmail.com>

* latent model support

* 1. Debugging.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging.

* update branch

Signed-off-by: ericharper <complex451@gmail.com>

* fix replace_bos_with_pad not found (#6443)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Support Swiglu in TP PP Conversion (#6437)

* Support Swiglu in TP PP Conversion

Signed-off-by: smajumdar <titu1994@gmail.com>

* Guard activation

Signed-off-by: smajumdar <titu1994@gmail.com>

* Guard activation

Signed-off-by: smajumdar <titu1994@gmail.com>

---------

Signed-off-by: smajumdar <titu1994@gmail.com>

* BERT pre-training mp fork to spawn (#6442)

* change bert fork to spawn

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* num_workers=0 fix

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Meagtron encoder decoder fix for empty validation outputs (#6459)

* 1. Meagtron encoder decoder fix for empty validation outputs.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging.

---------

Signed-off-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: Micha Livne <mlivne@nvidia.com>

* Added/updated new Conformer configs (#6426)

* updated conf files.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added confs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* moved longconformer confs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated readme.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated readme.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated batch sizes and added fastconformer ctc streaming configs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated batch sizes.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added hybrid support.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added hybrid support.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

---------

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* reduce workers on NMT CI (#6472)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* move to nvidia megatron repo (#6465)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Megatron KERPLE positional embeddings (#6478)

* [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

[TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [TTS] whitelist broken path fix. (#6412)

* [TTS] whitelist broken path fix.

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [TTS] FastPitch speaker encoder (#6417)

* Add initial codes

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove wemb

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix import

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Restore aligner loss

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add ConditionalInput

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix error and support pre-trained config

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Follow comments

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Rename config

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Change copyright and random weight test

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add initial codes

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix import error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add initial codes

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix dataset error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove reference speaker embedding

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove SV encoder

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Follow comments

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix length type

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix append

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Move error msg

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add look-up into speaker encoder

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add valueerror msg

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Move lookup

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove unused

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Rebase and Fix error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix spk encoder

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Rename n_speakers

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Follow comments

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix n_speakers None error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Sharded manifests for tarred datasets (#6395)

* testing sharded manifests

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* compatibility

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* proper fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* adding flag tot convert_to_tarred_audio_dataset

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* shard_manifests conf param

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* propagating the shard_manifests param

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* propagating the shard_manifests param

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* distributed checks

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* typo

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* typo

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes based on PR comments and tests

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes to convert_to_tarred_audio_dataset.py

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* reversing manifest shards flag

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* excluding manifests from webdataset url expansion

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* expand manifest paths before attempting to cache from datastore

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* explicit use of UTF-8 for manifest i/o

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

---------

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update wfst_text_normalization.rst (#6374)

Add Hungarian (incoming in NeMo-text-processing)

Signed-off-by: Jim O’Regan <jaoregan@tcd.ie>

* Support Swiglu in TP PP Conversion (#6437) (#6451)

* Support Swiglu in TP PP Conversion



* Guard activation



* Guard activation



---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>

* Update NeMo_TTS_Primer.ipynb (#6436)

* Update NeMo_TTS_Primer.ipynb

Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch.

Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>

* Update NeMo_TTS_Primer.ipynb

Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point.

Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>

---------

Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>

* add rampup batch size support for Megatron GPT (#6424)

* added rampup batch size support

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* added tests for rampup batch size

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* fixed the typos

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* added assertions

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* changed assertion rules

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* deleted unused imports

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* changed tests for rampup batch size

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* updated rampup batch size tests

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed styling

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* rampup batch size tests changes

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

---------

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461)

* 1. Meagtron encoder decoder fix for empty validation outputs.



* 1. Debugging.

---------

Signed-off-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: Micha Livne <michalivne@users.noreply.github.com>
Co-authored-by: Micha Livne <mlivne@nvidia.com>

* Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448)

* added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

* updated README with the new agg_tokenizer_manifest flag

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

* fixed typo in scripts/speech_recognition/code_switching/README.md

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

* changed agg_tokenizer_manifest to is_lid_manifest

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

---------

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Co-authored-by: Dima Rekesh <bmwshop@gmail.com>

* Added/updated new Conformer configs (#6426) (#6467)

* Update script for ngram rnnt and hat beam search decoding (#6370)

* add rnnt ngram beamsearch script

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add return encoding embedding option

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* update script

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add rnnt and hat ngram decoding script

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add some parameters

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add return_encoder_embeddings parameter to RNNTDecodingConfig

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* replace return_encoder_embeddings parameter

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* generalization of scipt behavior

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove return_encoder_embeddings parameter

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* remove return_encoder_embeddings parameter

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add manual encoder_embeddings calculation

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix beam_width value to 8

Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>

* fix rescoring description

Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>

---------

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>

* BERT pre-training mp fork to spawn (#6442) (#6454)

* change bert fork to spawn



* num_workers=0 fix



---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* fix replace_bos_with_pad not found (#6443) (#6450)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* reduce workers on NMT CI (#6472) (#6474)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* 1. Added KERPLE positional embeddings to encoder-decoder.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Added a missing file.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Fixing commits.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Signed-off-by: Jim O’Regan <jaoregan@tcd.ie>
Signed-off-by: smajumdar <titu1994@gmail.com>
Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Signed-off-by: Micha Livne <mlivne@nvidia.com>
Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Dima Rekesh <bmwshop@gmail.com>
Co-authored-by: Jim O’Regan <jaoregan@tcd.ie>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* 1. Added external index sample. (#6462)

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* Fix cache aware hybrid bugs (#6466)

* Update README to add core installation (#6488)

* update README for megatron-core

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* fix

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Fix typos (#6494)

Signed-off-by: smajumdar <titu1994@gmail.com>

* fix broken links r1.18.0 (#6501)

* fix broken links

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* fix broken links

Signed-off-by: Evelina <ebakhturina@nvidia.com>

---------

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* 1. Fixed gaussian hidden transform.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Finished updating hidden loss for MIM.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix custom forward_torch_softmax (#6512)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* [BugFix] Force _get_batch_preds() to keep logits in decoder timestamp… (#6500)

* [BugFix] Force _get_batch_preds() to keep logits in decoder timestamps generator r1.18.0

Signed-off-by: Taejin Park <tango4j@gmail.com>

* ignore keep_logits in FrameBatchASRLogits

Signed-off-by: Taejin Park <tango4j@gmail.com>

---------

Signed-off-by: Taejin Park <tango4j@gmail.com>

* [TTS] fixed broken path. (#6514)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* 1. Added a hiddens module.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix typos (#6523) (#6539)

* Fix typos

Signed-off-by: smajumdar <titu1994@gmail.com>

* Fix typos

Signed-off-by: smajumdar <titu1994@gmail.com>

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
(cherry picked from commit 5468077f5127be1a4c88065de2544f4268b9a6e4)

* added back the fast emit section to the configs. (#6540)

* added back the fast emit section to the configs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added back the fast emit section to the configs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

---------

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* Fix fp16 (#6543)

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* fix (#6529)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* pass .scale instead of scaler object to core (#6545)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Change Megatron Enc Dec model to use persistent_workers (#6548)

* persistent workers

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* fix

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Add FastConformer Hybrid ASR models for EN, ES, IT, DE, PL, HR, UA, BY (#6549)

* Added fastconfomer hybrid asr models for en, es, it, de, pl, hr, ua, by

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>

* updated ASR docs with the fastconformer hybrid checkpoints

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>

* added the fastconformer RNNT and CTC models

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>

---------

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>

* Add scores for FastConformer models (#6557)

Signed-off-by: smajumdar <titu1994@gmail.com>

* Patch transcribe and support offline transcribe for hybrid model (#6550)

Signed-off-by: fayejf <fayejf07@gmail.com>

* Not doing CastToFloat by default (#6524)

* Not doing CastToFloat by default

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

* Added docustring

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

* Dummy commit

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

---------

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

* temp rtd fix (#6568)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update manifest.py for speedup (#6565)

* Update manifest.py

Re-order the checks for faster processing audio filepaths that are already absolute paths

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* Update manifest.py

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

---------

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com>

* Turn autocast off when precision is fp32 (#6554)

* Turn autocast off when precision is fp32

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* address review

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* merge

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* More streaming conformer export fixes (#6567)

Signed-off-by: Greg Clark <grclark@nvidia.com>
Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com>

* Fix batch size reconf for T5 FT for multi-validation (#6582)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Updated Megatron LM encoder/decoder to use cfg for hiddens.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Added support to register externalhidden loss / transforms.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Make tensor split contiguous (#6580)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Patches from main to r1.18.0 for Virtual Parallel (#6592)

* Add interleaved pp support (#6498)

* Add support for Virtual Pipeline Parallel conversion

Signed-off-by: smajumdar <titu1994@gmail.com>

* Add support for Virtual Pipeline Parallel conversion

Signed-off-by: smajumdar <titu1994@gmail.com>

* Switch to megatron core

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
(cherry picked from commit 892987169ef277f328e15b71a5a0c9bd961c8ee7)

* Add patches for Virtual Parallel conversion (#6589)

* Add patches for Virtual Parllel conversion

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
(cherry picked from commit 1d813a372ab51688e3af6395d905a4c0366ffd23)

* Documentation for ASR-TTS models (#6594)

* Add docs about hybrid ASR-TTS models

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Add docs about text-only datasets

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Add docs about ASR-TTS checkpoints

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Add docs about ASR-TTS configs and training

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Clean up

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* ASR-TTS docs: add to api, fix imports

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Clean up

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Wrap optional import

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Revert general ASR import

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

---------

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Update SDP docs (#6485)

* add info about SDP e.g. processor classes in docs

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* add link to SDP docs in README

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* address code review comments and add SDP overview diagram

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* Fix spelling typo

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

---------

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* Create dummy iters to satisy len checks (#6600)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* Restore GPT support for interleaved pipeline parallelism (#6528)

* Restore logic for data-parallel communication with pipeline parallelism in GPT

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Support dynamic attention masks in GPT

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Debug typos

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Debug data iterator caching with interleaved pipeline parallelism

Each model chunk accesses the data iterator multiple times, so we need to cache multiple samples.

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update Megatron-LM commit

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Distinguish between list of data iterators and data iterator that is a list

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Create dummy iters to satisy len checks

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Kludge while waiting for Megatron-LM update

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* set transformers offline to avoid rate limiting

Signed-off-by: ericharper <complex451@gmail.com>

---------

Signed-off-by: Tim Moon <tmoon@nvidia.com>
Signed-off-by: Eric Harper <complex451@gmail.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: ericharper <complex451@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* Patch transcribe_util for steaming mode and add wer calculation back to inference scripts (#6601)

* fix write

Signed-off-by: fayejf <fayejf07@gmail.com>

* decoding ctc

Signed-off-by: fayejf <fayejf07@gmail.com>

* temp set rnnt decoding return_best_hypothesis to true

Signed-off-by: fayejf <fayejf07@gmail.com>

* add wer cal back to transcribe_speech as requested

Signed-off-by: fayejf <fayejf07@gmail.com>

* add wer cal back to speech_to_text_buffered_infer_rnnt  as requested

Signed-off-by: fayejf <fayejf07@gmail.com>

* add wer cal back to speech_to_text_buffered_infer_ctc as requested

Signed-off-by: fayejf <fayejf07@gmail.com>

* style fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* reflect change in asr_evaluator

Signed-off-by: fayejf <fayejf07@gmail.com>

* reflect som and vahid comment

Signed-off-by: fayejf <fayejf07@gmail.com>

* remove return_best_hy=true in transcribe_speech

Signed-off-by: fayejf <fayejf07@gmail.com>

* no text skip

Signed-off-by: fayejf <fayejf07@gmail.com>

---------

Signed-off-by: fayejf <fayejf07@gmail.com>

* 1. Added example conf YAML.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Added support in tensor_parallel.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add hat image to docs (#6619)

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* update core commit hash in readme (#6622)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Patch decoding for PC models (#6630)

* Patch decoding logic for PC models

Signed-off-by: smajumdar <titu1994@gmail.com>

* Patch decoding logic for PC models

Signed-off-by: smajumdar <titu1994@gmail.com>

---------

Signed-off-by: smajumdar <titu1994@gmail.com>

* Fix wer.py where 'errors' variable was not set (#6633)

Fix wer.py where 'errors' variable was not set when both reference and hypothesis are empty strings

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* fix att_context_size bug for older models. (#6635)

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* Add megatron_core to requirements (#6639)

* add megatron_core to requirements

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: ericharper <complex451@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Remove from jenkins (#6641)

* add megatron_core to requirements

Signed-off-by: ericharper <complex451@gmail.com>

* remove from jenkins

Signed-off-by: ericharper <complex451@gmail.com>

---------

Signed-off-by: ericharper <complex451@gmail.com>

* remove dup (#6643)

Signed-off-by: ericharper <complex451@gmail.com>

* 1. Fixed config to use names, and added better error messages.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Added support to pass extra data to hiddens for loss computation.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Working on passing extra data to hiddnes.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Fixed support in loading .nemo without hiddnes module.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Improved and fixed logging of validation and testing.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Fixed training logging.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Fixed logging of hidden loss.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Fixed logging names.
2. Added logging to hiddens and tokens loss.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Fixed conflicts.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

---------

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Oleksii Kuchaiev <okuchaiev@nvidia.com>
Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>
Signed-off-by: smajumdar <smajumdar@nvidia.com>
Signed-off-by: fayejf <fayejf07@gmail.com>
Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com>
Signed-off-by: Virginia Adams <vadams@nvidia.com>
Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
Signed-off-by: ericharper <complex451@gmail.com>
Signed-off-by: Vahid <vnoroozi@nvidia.com>
Signed-off-by: Ante Jukić <ajukic@nvidia.com>
Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>
Signed-off-by: sam1373 <samuelkriman@gmail.com>
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Signed-off-by: ekmb <ebakhturina@nvidia.com>
Signed-off-by: Yang Zhang <yangzhang@nvidia.com>
Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>
Signed-off-by: Tim Moon <tmoon@nvidia.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: smajumdar <titu1994@gmail.com>
Signed-off-by: Micha Livne <mlivne@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Signed-off-by: Jim O’Regan <jaoregan@tcd.ie>
Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Signed-off-by: Evelina <ebakhturina@nvidia.com>
Signed-off-by: Taejin Park <tango4j@gmail.com>
Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Signed-off-by: Greg Clark <grclark@nvidia.com>
Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>
Signed-off-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Oleksii Kuchaiev <okuchaiev@nvidia.com>
Co-authored-by: Jocelyn <jocelynh@nvidia.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Co-authored-by: fayejf <36722593+fayejf@users.noreply.github.com>
Co-authored-by: bene-ges <61418381+bene-ges@users.noreply.github.com>
Co-authored-by: Alexandra Antonova <aleksandraa@nvidia.com>
Co-authored-by: Virginia Adams <78445382+vadam5@users.noreply.github.com>
Co-authored-by: Zhilin Wang <wangzhilin12061996@hotmail.com>
Co-authored-by: Vladimir Bataev <vbataev@nvidia.com>
Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com>
Co-authored-by: anteju <108555623+anteju@users.noreply.github.com>
Co-authored-by: Ante Jukić <ajukic@nvidia.com>
Co-authored-by: David <amosalla@asu.edu>
Co-authored-by: David Mosallanezhad <dmosallanezh@nvidia.com>
Co-authored-by: Samuel Kriman <samuelkriman@gmail.com>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>
Co-authored-by: Evelina <10428420+ekmb@users.noreply.github.com>
Co-authored-by: Sean Naren <sean.narenthiran@gmail.com>
Co-authored-by: Yang Zhang <yzhang123@users.noreply.github.com>
Co-authored-by: Sean Naren <snarenthiran@nvidia.com>
Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>
Co-authored-by: Neha Tadimeti <ntadimeti@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>
Co-authored-by: Dima Rekesh <bmwshop@gmail.com>
Co-authored-by: Jim O’Regan <jaoregan@tcd.ie>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Co-authored-by: Taejin Park <tango4j@gmail.com>
Co-authored-by: Boris Fomitchev <borisfom@users.noreply.github.com>
Co-authored-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Co-authored-by: Greg Clark <grclark@nvidia.com>
Co-authored-by: Elena Rastorgueva <80532067+erastorgueva-nv@users.noreply.github.com>
guyueh1 pushed a commit to guyueh1/NeMo that referenced this pull request Aug 14, 2023
* [TTS] bugfix for missing configs. (#4725)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* docs typo fix

Signed-off-by: Oleksii Kuchaiev <okuchaiev@nvidia.com>

* Fix pynini install in TTS tutorials (#4729)

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

* Fix ASR notebooks (#4738)

Signed-off-by: smajumdar <smajumdar@nvidia.com>

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Multilingual VAD model (#4734)

* add ngc link

Signed-off-by: fayejf <fayejf07@gmail.com>

* add tuned VAD config on ASR data

Signed-off-by: fayejf <fayejf07@gmail.com>

* yaml note

Signed-off-by: fayejf <fayejf07@gmail.com>

* update vad asr notebook with mVAD

Signed-off-by: fayejf <fayejf07@gmail.com>

* update vad infer config comment

Signed-off-by: fayejf <fayejf07@gmail.com>

* fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* mvad sd config for ch109

Signed-off-by: fayejf <fayejf07@gmail.com>

* update sd readme

Signed-off-by: fayejf <fayejf07@gmail.com>

* add new mVAD model to doc

Signed-off-by: fayejf <fayejf07@gmail.com>

* style fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* update sd tutorial with mVAD

Signed-off-by: fayejf <fayejf07@gmail.com>

* typo fix

Signed-off-by: fayejf <fayejf07@gmail.com>

Signed-off-by: fayejf <fayejf07@gmail.com>

* publish pretrained itn t5 model for English (#4748)

Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com>

Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com>
Co-authored-by: Alexandra Antonova <aleksandraa@nvidia.com>

* Updated docs and doc paths (#4754)

* Updated docs and doc paths

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Update Multitask_Prompt_and_PTuning.ipynb

* Update README.rst

* Changed branch name to use single quotes

Signed-off-by: Virginia Adams <vadams@nvidia.com>

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* fix bug relating to ddp strategy in joint intent slot classification tutorial (#4762)

* [TTS] updated config with a German IPA phoneme tokenizer (#4756)

* [TTS] added a German IPA phoneme tokenizer
* [TTS][ASR] enabled customized arguments for trimming the leading and trailing silence.
* [TTS] disabled spline interpolation for beta-binomial distribution. Let it generate align prior and save to disks. Use a new phoneme tokenizer.
* [TTS] use consistent spline interpolation with fastpitch checkpoint when generating mel-spectrograms for hifigan finetune.

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* Update r1.11 to new heteronyms list (#4745)

* Update configs to new heteronyms list
* Remove old heteronyms list, add alt 'merchandise' pron to CMUdict
* Update remaining references to old heteronyms list

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* [TTS] Add multi-speaker German FastPitch and HiFiGAN NGC checkpoints (#4763)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* [TTS] Add single male speaker German FastPitch and HiFiGAN NGC checkpoints (#4770)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* Update CMUdict with more recent 0.7b entries (#4768)

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* Install pynini in docker container (#4733)

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Fix tutorial formatting (#4778)

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

* [TTS] deprecated old scripts for ljspeech. (#4780)

* deprecated old scripts for ljspeech.
* removed relevent function calls in TTS docs.

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* update branch and typos (#4788)

Signed-off-by: ericharper <complex451@gmail.com>

Signed-off-by: ericharper <complex451@gmail.com>

* Adding support for models trained with full context for cache-aware streaming. (#4687)

* added support for models trained with full context.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* dropped seq_range

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed indexing in caching methods.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated docs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* addressed comments.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* change frame-wise to cache-aware.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* change frame-wise to cache-aware.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* change frame-wise to cache-aware.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* Update megatron encoder decoder model to support py37 for colab (#4791)

* [ASR] Add pretrained ASR models for Croatian (#4682)

* [ASR] Add pretrained ASR models for Croatian

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

* Fix style for import

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

Signed-off-by: Ante Jukić <ajukic@nvidia.com>
Co-authored-by: Ante Jukić <ajukic@nvidia.com>
Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>

* added/fixed export for Megatron models (#4712)

* added/fixed export for Megatron models

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

* fixed style

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

* fixed FusedScaleMaskSoftmax in BioMegatron

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

* included comments

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>
Co-authored-by: David Mosallanezhad <dmosallanezh@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* update branch for qa notebook

Signed-off-by: ericharper <complex451@gmail.com>

* Fix initializing weights from ptl ckpt with exclude (#4807)

Signed-off-by: sam1373 <samuelkriman@gmail.com>

Signed-off-by: sam1373 <samuelkriman@gmail.com>

* Fix index error from addition of voiced_mask and p_voiced (#4811)

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

* T5 prompt learning fixes (#4771)

* RPE, hidden size and config fixes

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update to reflect new config names

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Sentencepiece fixes

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix finetuning

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Add encoder seq len to gpt

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Add finetune eval script

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix name

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update Jenkinsfile

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update config

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix CI test

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update check

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Backward compat

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update CI test

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Split rank for Enc-Dec models

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Address comments

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Co-authored-by: Virginia Adams <78445382+vadam5@users.noreply.github.com>

* G2P docs (#4841)

* g2p docs added

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* fix references

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* address review feedback

Signed-off-by: ekmb <ebakhturina@nvidia.com>

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* Fix providing glue in seq2seq eval (#4843)

* Fix providing glue in seq2seq eval

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Updated inference code and squad scripts (#4835)

* Updated inference code and squad scripts

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Reverted GPT & T5 inference files back to use NLPDDPlugin

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Overwrite frozen LM to use fused adam

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Added padded vocab size

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Fixed val check interval value

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Python format fix

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Make t5 prompt learning preds write to file

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Added back dp=1 check

Signed-off-by: Virginia Adams <vadams@nvidia.com>

Signed-off-by: Virginia Adams <vadams@nvidia.com>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>

* Update README.rst

* Fix uppercasing mismatch for IPA heteronyms (#4860)

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

* Set the number of workers to 0 for validation and test sets in all enc-dec models (#4790)

* Set workers to 0 for validation and test

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Revert pin memory

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Co-authored-by: Sean Naren <sean.narenthiran@gmail.com>

* Fix mha (#4866)

* fix bug in mha forward function related to cache update return type

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* fix lgtm

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>
Co-authored-by: Sean Naren <snarenthiran@nvidia.com>

* ipa bug fix (#4871)

Signed-off-by: ekmb <ebakhturina@nvidia.com>

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* Fix Megatron NMT consumed samples and ckpt_to_nemo split rank (#4884)

* Fix nmt and ckpt_to_nemo

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* added utf8 encoding (#4892)

Signed-off-by: Virginia Adams <vadams@nvidia.com>

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* 1. Applying the same patch to r1.11.0 (#4894)

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

* Update tutorials.rst (#4897)

* update readme with apex commit

Signed-off-by: ericharper <complex451@gmail.com>

* Add support for Apex distributed Adam optimizer with GPT-3 (#4487)

* Add support for Apex distributed Adam optimizer with GPT-3

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Fix bug in grad clipping with dist Adam

Grad norm was computed over all params, not respecting model parallelism.

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Fix bug with DDP initialization

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Make distopt dependent on megatron_amp_o2

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Fix code formatting

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Handle dist Adam in optimizer unit tests

Signed-off-by: Tim Moon <tmoon@nvidia.com>

Signed-off-by: Tim Moon <tmoon@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* update readme

Signed-off-by: ericharper <complex451@gmail.com>

* update readme

Signed-off-by: ericharper <complex451@gmail.com>

* latent model support

* 1. Debugging.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging.

* update branch

Signed-off-by: ericharper <complex451@gmail.com>

* fix replace_bos_with_pad not found (#6443)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Support Swiglu in TP PP Conversion (#6437)

* Support Swiglu in TP PP Conversion

Signed-off-by: smajumdar <titu1994@gmail.com>

* Guard activation

Signed-off-by: smajumdar <titu1994@gmail.com>

* Guard activation

Signed-off-by: smajumdar <titu1994@gmail.com>

---------

Signed-off-by: smajumdar <titu1994@gmail.com>

* BERT pre-training mp fork to spawn (#6442)

* change bert fork to spawn

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* num_workers=0 fix

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Meagtron encoder decoder fix for empty validation outputs (#6459)

* 1. Meagtron encoder decoder fix for empty validation outputs.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging.

---------

Signed-off-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: Micha Livne <mlivne@nvidia.com>

* Added/updated new Conformer configs (#6426)

* updated conf files.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added confs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* moved longconformer confs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated readme.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated readme.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated batch sizes and added fastconformer ctc streaming configs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated batch sizes.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added hybrid support.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added hybrid support.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

---------

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* reduce workers on NMT CI (#6472)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* move to nvidia megatron repo (#6465)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Megatron KERPLE positional embeddings (#6478)

* [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

[TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [TTS] whitelist broken path fix. (#6412)

* [TTS] whitelist broken path fix.

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [TTS] FastPitch speaker encoder (#6417)

* Add initial codes

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove wemb

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix import

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Restore aligner loss

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add ConditionalInput

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix error and support pre-trained config

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Follow comments

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Rename config

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Change copyright and random weight test

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add initial codes

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix import error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add initial codes

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix dataset error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove reference speaker embedding

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove SV encoder

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Follow comments

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix length type

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix append

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Move error msg

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add look-up into speaker encoder

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add valueerror msg

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Move lookup

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove unused

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Rebase and Fix error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix spk encoder

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Rename n_speakers

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Follow comments

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix n_speakers None error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Sharded manifests for tarred datasets (#6395)

* testing sharded manifests

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* compatibility

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* proper fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* adding flag tot convert_to_tarred_audio_dataset

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* shard_manifests conf param

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* propagating the shard_manifests param

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* propagating the shard_manifests param

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* distributed checks

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* typo

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* typo

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes based on PR comments and tests

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes to convert_to_tarred_audio_dataset.py

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* reversing manifest shards flag

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* excluding manifests from webdataset url expansion

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* expand manifest paths before attempting to cache from datastore

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* explicit use of UTF-8 for manifest i/o

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

---------

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update wfst_text_normalization.rst (#6374)

Add Hungarian (incoming in NeMo-text-processing)

Signed-off-by: Jim O’Regan <jaoregan@tcd.ie>

* Support Swiglu in TP PP Conversion (#6437) (#6451)

* Support Swiglu in TP PP Conversion



* Guard activation



* Guard activation



---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>

* Update NeMo_TTS_Primer.ipynb (#6436)

* Update NeMo_TTS_Primer.ipynb

Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch.

Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>

* Update NeMo_TTS_Primer.ipynb

Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point.

Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>

---------

Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>

* add rampup batch size support for Megatron GPT (#6424)

* added rampup batch size support

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* added tests for rampup batch size

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* fixed the typos

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* added assertions

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* changed assertion rules

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* deleted unused imports

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* changed tests for rampup batch size

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* updated rampup batch size tests

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed styling

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* rampup batch size tests changes

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

---------

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461)

* 1. Meagtron encoder decoder fix for empty validation outputs.



* 1. Debugging.

---------

Signed-off-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: Micha Livne <michalivne@users.noreply.github.com>
Co-authored-by: Micha Livne <mlivne@nvidia.com>

* Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448)

* added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

* updated README with the new agg_tokenizer_manifest flag

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

* fixed typo in scripts/speech_recognition/code_switching/README.md

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

* changed agg_tokenizer_manifest to is_lid_manifest

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

---------

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Co-authored-by: Dima Rekesh <bmwshop@gmail.com>

* Added/updated new Conformer configs (#6426) (#6467)

* Update script for ngram rnnt and hat beam search decoding (#6370)

* add rnnt ngram beamsearch script

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add return encoding embedding option

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* update script

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add rnnt and hat ngram decoding script

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add some parameters

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add return_encoder_embeddings parameter to RNNTDecodingConfig

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* replace return_encoder_embeddings parameter

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* generalization of scipt behavior

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove return_encoder_embeddings parameter

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* remove return_encoder_embeddings parameter

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add manual encoder_embeddings calculation

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix beam_width value to 8

Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>

* fix rescoring description

Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>

---------

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>

* BERT pre-training mp fork to spawn (#6442) (#6454)

* change bert fork to spawn



* num_workers=0 fix



---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* fix replace_bos_with_pad not found (#6443) (#6450)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* reduce workers on NMT CI (#6472) (#6474)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* 1. Added KERPLE positional embeddings to encoder-decoder.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Added a missing file.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Fixing commits.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Signed-off-by: Jim O’Regan <jaoregan@tcd.ie>
Signed-off-by: smajumdar <titu1994@gmail.com>
Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Signed-off-by: Micha Livne <mlivne@nvidia.com>
Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Dima Rekesh <bmwshop@gmail.com>
Co-authored-by: Jim O’Regan <jaoregan@tcd.ie>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* 1. Added external index sample. (#6462)

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* Fix cache aware hybrid bugs (#6466)

* Update README to add core installation (#6488)

* update README for megatron-core

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* fix

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Fix typos (#6494)

Signed-off-by: smajumdar <titu1994@gmail.com>

* fix broken links r1.18.0 (#6501)

* fix broken links

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* fix broken links

Signed-off-by: Evelina <ebakhturina@nvidia.com>

---------

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* 1. Fixed gaussian hidden transform.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Finished updating hidden loss for MIM.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix custom forward_torch_softmax (#6512)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* [BugFix] Force _get_batch_preds() to keep logits in decoder timestamp… (#6500)

* [BugFix] Force _get_batch_preds() to keep logits in decoder timestamps generator r1.18.0

Signed-off-by: Taejin Park <tango4j@gmail.com>

* ignore keep_logits in FrameBatchASRLogits

Signed-off-by: Taejin Park <tango4j@gmail.com>

---------

Signed-off-by: Taejin Park <tango4j@gmail.com>

* [TTS] fixed broken path. (#6514)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* 1. Added a hiddens module.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix typos (#6523) (#6539)

* Fix typos

Signed-off-by: smajumdar <titu1994@gmail.com>

* Fix typos

Signed-off-by: smajumdar <titu1994@gmail.com>

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
(cherry picked from commit 5468077f5127be1a4c88065de2544f4268b9a6e4)

* added back the fast emit section to the configs. (#6540)

* added back the fast emit section to the configs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added back the fast emit section to the configs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

---------

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* Fix fp16 (#6543)

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* fix (#6529)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* pass .scale instead of scaler object to core (#6545)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Change Megatron Enc Dec model to use persistent_workers (#6548)

* persistent workers

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* fix

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Add FastConformer Hybrid ASR models for EN, ES, IT, DE, PL, HR, UA, BY (#6549)

* Added fastconfomer hybrid asr models for en, es, it, de, pl, hr, ua, by

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>

* updated ASR docs with the fastconformer hybrid checkpoints

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>

* added the fastconformer RNNT and CTC models

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>

---------

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>

* Add scores for FastConformer models (#6557)

Signed-off-by: smajumdar <titu1994@gmail.com>

* Patch transcribe and support offline transcribe for hybrid model (#6550)

Signed-off-by: fayejf <fayejf07@gmail.com>

* Not doing CastToFloat by default (#6524)

* Not doing CastToFloat by default

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

* Added docustring

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

* Dummy commit

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

---------

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

* temp rtd fix (#6568)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update manifest.py for speedup (#6565)

* Update manifest.py

Re-order the checks for faster processing audio filepaths that are already absolute paths

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* Update manifest.py

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

---------

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com>

* Turn autocast off when precision is fp32 (#6554)

* Turn autocast off when precision is fp32

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* address review

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* merge

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* More streaming conformer export fixes (#6567)

Signed-off-by: Greg Clark <grclark@nvidia.com>
Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com>

* Fix batch size reconf for T5 FT for multi-validation (#6582)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Updated Megatron LM encoder/decoder to use cfg for hiddens.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Added support to register externalhidden loss / transforms.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Make tensor split contiguous (#6580)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Patches from main to r1.18.0 for Virtual Parallel (#6592)

* Add interleaved pp support (#6498)

* Add support for Virtual Pipeline Parallel conversion

Signed-off-by: smajumdar <titu1994@gmail.com>

* Add support for Virtual Pipeline Parallel conversion

Signed-off-by: smajumdar <titu1994@gmail.com>

* Switch to megatron core

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
(cherry picked from commit 892987169ef277f328e15b71a5a0c9bd961c8ee7)

* Add patches for Virtual Parallel conversion (#6589)

* Add patches for Virtual Parllel conversion

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
(cherry picked from commit 1d813a372ab51688e3af6395d905a4c0366ffd23)

* Documentation for ASR-TTS models (#6594)

* Add docs about hybrid ASR-TTS models

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Add docs about text-only datasets

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Add docs about ASR-TTS checkpoints

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Add docs about ASR-TTS configs and training

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Clean up

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* ASR-TTS docs: add to api, fix imports

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Clean up

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Wrap optional import

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Revert general ASR import

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

---------

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Update SDP docs (#6485)

* add info about SDP e.g. processor classes in docs

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* add link to SDP docs in README

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* address code review comments and add SDP overview diagram

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* Fix spelling typo

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

---------

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* Create dummy iters to satisy len checks (#6600)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* Restore GPT support for interleaved pipeline parallelism (#6528)

* Restore logic for data-parallel communication with pipeline parallelism in GPT

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Support dynamic attention masks in GPT

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Debug typos

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Debug data iterator caching with interleaved pipeline parallelism

Each model chunk accesses the data iterator multiple times, so we need to cache multiple samples.

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update Megatron-LM commit

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Distinguish between list of data iterators and data iterator that is a list

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Create dummy iters to satisy len checks

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Kludge while waiting for Megatron-LM update

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* set transformers offline to avoid rate limiting

Signed-off-by: ericharper <complex451@gmail.com>

---------

Signed-off-by: Tim Moon <tmoon@nvidia.com>
Signed-off-by: Eric Harper <complex451@gmail.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: ericharper <complex451@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* Patch transcribe_util for steaming mode and add wer calculation back to inference scripts (#6601)

* fix write

Signed-off-by: fayejf <fayejf07@gmail.com>

* decoding ctc

Signed-off-by: fayejf <fayejf07@gmail.com>

* temp set rnnt decoding return_best_hypothesis to true

Signed-off-by: fayejf <fayejf07@gmail.com>

* add wer cal back to transcribe_speech as requested

Signed-off-by: fayejf <fayejf07@gmail.com>

* add wer cal back to speech_to_text_buffered_infer_rnnt  as requested

Signed-off-by: fayejf <fayejf07@gmail.com>

* add wer cal back to speech_to_text_buffered_infer_ctc as requested

Signed-off-by: fayejf <fayejf07@gmail.com>

* style fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* reflect change in asr_evaluator

Signed-off-by: fayejf <fayejf07@gmail.com>

* reflect som and vahid comment

Signed-off-by: fayejf <fayejf07@gmail.com>

* remove return_best_hy=true in transcribe_speech

Signed-off-by: fayejf <fayejf07@gmail.com>

* no text skip

Signed-off-by: fayejf <fayejf07@gmail.com>

---------

Signed-off-by: fayejf <fayejf07@gmail.com>

* 1. Added example conf YAML.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Added support in tensor_parallel.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add hat image to docs (#6619)

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* update core commit hash in readme (#6622)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Patch decoding for PC models (#6630)

* Patch decoding logic for PC models

Signed-off-by: smajumdar <titu1994@gmail.com>

* Patch decoding logic for PC models

Signed-off-by: smajumdar <titu1994@gmail.com>

---------

Signed-off-by: smajumdar <titu1994@gmail.com>

* Fix wer.py where 'errors' variable was not set (#6633)

Fix wer.py where 'errors' variable was not set when both reference and hypothesis are empty strings

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* fix att_context_size bug for older models. (#6635)

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* Add megatron_core to requirements (#6639)

* add megatron_core to requirements

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: ericharper <complex451@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Remove from jenkins (#6641)

* add megatron_core to requirements

Signed-off-by: ericharper <complex451@gmail.com>

* remove from jenkins

Signed-off-by: ericharper <complex451@gmail.com>

---------

Signed-off-by: ericharper <complex451@gmail.com>

* remove dup (#6643)

Signed-off-by: ericharper <complex451@gmail.com>

* 1. Fixed config to use names, and added better error messages.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Added support to pass extra data to hiddens for loss computation.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Working on passing extra data to hiddnes.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Fixed support in loading .nemo without hiddnes module.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Improved and fixed logging of validation and testing.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Fixed training logging.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Fixed logging of hidden loss.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Fixed logging names.
2. Added logging to hiddens and tokens loss.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Fixed conflicts.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

---------

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Oleksii Kuchaiev <okuchaiev@nvidia.com>
Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>
Signed-off-by: smajumdar <smajumdar@nvidia.com>
Signed-off-by: fayejf <fayejf07@gmail.com>
Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com>
Signed-off-by: Virginia Adams <vadams@nvidia.com>
Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
Signed-off-by: ericharper <complex451@gmail.com>
Signed-off-by: Vahid <vnoroozi@nvidia.com>
Signed-off-by: Ante Jukić <ajukic@nvidia.com>
Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>
Signed-off-by: sam1373 <samuelkriman@gmail.com>
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Signed-off-by: ekmb <ebakhturina@nvidia.com>
Signed-off-by: Yang Zhang <yangzhang@nvidia.com>
Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>
Signed-off-by: Tim Moon <tmoon@nvidia.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: smajumdar <titu1994@gmail.com>
Signed-off-by: Micha Livne <mlivne@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Signed-off-by: Jim O’Regan <jaoregan@tcd.ie>
Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Signed-off-by: Evelina <ebakhturina@nvidia.com>
Signed-off-by: Taejin Park <tango4j@gmail.com>
Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Signed-off-by: Greg Clark <grclark@nvidia.com>
Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>
Signed-off-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Oleksii Kuchaiev <okuchaiev@nvidia.com>
Co-authored-by: Jocelyn <jocelynh@nvidia.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Co-authored-by: fayejf <36722593+fayejf@users.noreply.github.com>
Co-authored-by: bene-ges <61418381+bene-ges@users.noreply.github.com>
Co-authored-by: Alexandra Antonova <aleksandraa@nvidia.com>
Co-authored-by: Virginia Adams <78445382+vadam5@users.noreply.github.com>
Co-authored-by: Zhilin Wang <wangzhilin12061996@hotmail.com>
Co-authored-by: Vladimir Bataev <vbataev@nvidia.com>
Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com>
Co-authored-by: anteju <108555623+anteju@users.noreply.github.com>
Co-authored-by: Ante Jukić <ajukic@nvidia.com>
Co-authored-by: David <amosalla@asu.edu>
Co-authored-by: David Mosallanezhad <dmosallanezh@nvidia.com>
Co-authored-by: Samuel Kriman <samuelkriman@gmail.com>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>
Co-authored-by: Evelina <10428420+ekmb@users.noreply.github.com>
Co-authored-by: Sean Naren <sean.narenthiran@gmail.com>
Co-authored-by: Yang Zhang <yzhang123@users.noreply.github.com>
Co-authored-by: Sean Naren <snarenthiran@nvidia.com>
Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>
Co-authored-by: Neha Tadimeti <ntadimeti@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>
Co-authored-by: Dima Rekesh <bmwshop@gmail.com>
Co-authored-by: Jim O’Regan <jaoregan@tcd.ie>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Co-authored-by: Taejin Park <tango4j@gmail.com>
Co-authored-by: Boris Fomitchev <borisfom@users.noreply.github.com>
Co-authored-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Co-authored-by: Greg Clark <grclark@nvidia.com>
Co-authored-by: Elena Rastorgueva <80532067+erastorgueva-nv@users.noreply.github.com>
dorotat-nv pushed a commit to dorotat-nv/NeMo that referenced this pull request Aug 24, 2023
* [TTS] bugfix for missing configs. (#4725)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* docs typo fix

Signed-off-by: Oleksii Kuchaiev <okuchaiev@nvidia.com>

* Fix pynini install in TTS tutorials (#4729)

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

* Fix ASR notebooks (#4738)

Signed-off-by: smajumdar <smajumdar@nvidia.com>

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Multilingual VAD model (#4734)

* add ngc link

Signed-off-by: fayejf <fayejf07@gmail.com>

* add tuned VAD config on ASR data

Signed-off-by: fayejf <fayejf07@gmail.com>

* yaml note

Signed-off-by: fayejf <fayejf07@gmail.com>

* update vad asr notebook with mVAD

Signed-off-by: fayejf <fayejf07@gmail.com>

* update vad infer config comment

Signed-off-by: fayejf <fayejf07@gmail.com>

* fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* mvad sd config for ch109

Signed-off-by: fayejf <fayejf07@gmail.com>

* update sd readme

Signed-off-by: fayejf <fayejf07@gmail.com>

* add new mVAD model to doc

Signed-off-by: fayejf <fayejf07@gmail.com>

* style fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* update sd tutorial with mVAD

Signed-off-by: fayejf <fayejf07@gmail.com>

* typo fix

Signed-off-by: fayejf <fayejf07@gmail.com>

Signed-off-by: fayejf <fayejf07@gmail.com>

* publish pretrained itn t5 model for English (#4748)

Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com>

Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com>
Co-authored-by: Alexandra Antonova <aleksandraa@nvidia.com>

* Updated docs and doc paths (#4754)

* Updated docs and doc paths

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Update Multitask_Prompt_and_PTuning.ipynb

* Update README.rst

* Changed branch name to use single quotes

Signed-off-by: Virginia Adams <vadams@nvidia.com>

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* fix bug relating to ddp strategy in joint intent slot classification tutorial (#4762)

* [TTS] updated config with a German IPA phoneme tokenizer (#4756)

* [TTS] added a German IPA phoneme tokenizer
* [TTS][ASR] enabled customized arguments for trimming the leading and trailing silence.
* [TTS] disabled spline interpolation for beta-binomial distribution. Let it generate align prior and save to disks. Use a new phoneme tokenizer.
* [TTS] use consistent spline interpolation with fastpitch checkpoint when generating mel-spectrograms for hifigan finetune.

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* Update r1.11 to new heteronyms list (#4745)

* Update configs to new heteronyms list
* Remove old heteronyms list, add alt 'merchandise' pron to CMUdict
* Update remaining references to old heteronyms list

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* [TTS] Add multi-speaker German FastPitch and HiFiGAN NGC checkpoints (#4763)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* [TTS] Add single male speaker German FastPitch and HiFiGAN NGC checkpoints (#4770)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* Update CMUdict with more recent 0.7b entries (#4768)

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* Install pynini in docker container (#4733)

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Fix tutorial formatting (#4778)

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

* [TTS] deprecated old scripts for ljspeech. (#4780)

* deprecated old scripts for ljspeech.
* removed relevent function calls in TTS docs.

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* update branch and typos (#4788)

Signed-off-by: ericharper <complex451@gmail.com>

Signed-off-by: ericharper <complex451@gmail.com>

* Adding support for models trained with full context for cache-aware streaming. (#4687)

* added support for models trained with full context.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* dropped seq_range

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed indexing in caching methods.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated docs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* addressed comments.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* change frame-wise to cache-aware.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* change frame-wise to cache-aware.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* change frame-wise to cache-aware.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* Update megatron encoder decoder model to support py37 for colab (#4791)

* [ASR] Add pretrained ASR models for Croatian (#4682)

* [ASR] Add pretrained ASR models for Croatian

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

* Fix style for import

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

Signed-off-by: Ante Jukić <ajukic@nvidia.com>
Co-authored-by: Ante Jukić <ajukic@nvidia.com>
Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>

* added/fixed export for Megatron models (#4712)

* added/fixed export for Megatron models

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

* fixed style

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

* fixed FusedScaleMaskSoftmax in BioMegatron

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

* included comments

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>
Co-authored-by: David Mosallanezhad <dmosallanezh@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* update branch for qa notebook

Signed-off-by: ericharper <complex451@gmail.com>

* Fix initializing weights from ptl ckpt with exclude (#4807)

Signed-off-by: sam1373 <samuelkriman@gmail.com>

Signed-off-by: sam1373 <samuelkriman@gmail.com>

* Fix index error from addition of voiced_mask and p_voiced (#4811)

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

* T5 prompt learning fixes (#4771)

* RPE, hidden size and config fixes

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update to reflect new config names

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Sentencepiece fixes

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix finetuning

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Add encoder seq len to gpt

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Add finetune eval script

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix name

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update Jenkinsfile

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update config

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix CI test

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update check

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Backward compat

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update CI test

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Split rank for Enc-Dec models

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Address comments

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Co-authored-by: Virginia Adams <78445382+vadam5@users.noreply.github.com>

* G2P docs (#4841)

* g2p docs added

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* fix references

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* address review feedback

Signed-off-by: ekmb <ebakhturina@nvidia.com>

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* Fix providing glue in seq2seq eval (#4843)

* Fix providing glue in seq2seq eval

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Updated inference code and squad scripts (#4835)

* Updated inference code and squad scripts

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Reverted GPT & T5 inference files back to use NLPDDPlugin

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Overwrite frozen LM to use fused adam

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Added padded vocab size

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Fixed val check interval value

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Python format fix

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Make t5 prompt learning preds write to file

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Added back dp=1 check

Signed-off-by: Virginia Adams <vadams@nvidia.com>

Signed-off-by: Virginia Adams <vadams@nvidia.com>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>

* Update README.rst

* Fix uppercasing mismatch for IPA heteronyms (#4860)

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

* Set the number of workers to 0 for validation and test sets in all enc-dec models (#4790)

* Set workers to 0 for validation and test

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Revert pin memory

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Co-authored-by: Sean Naren <sean.narenthiran@gmail.com>

* Fix mha (#4866)

* fix bug in mha forward function related to cache update return type

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* fix lgtm

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>
Co-authored-by: Sean Naren <snarenthiran@nvidia.com>

* ipa bug fix (#4871)

Signed-off-by: ekmb <ebakhturina@nvidia.com>

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* Fix Megatron NMT consumed samples and ckpt_to_nemo split rank (#4884)

* Fix nmt and ckpt_to_nemo

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* added utf8 encoding (#4892)

Signed-off-by: Virginia Adams <vadams@nvidia.com>

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* 1. Applying the same patch to r1.11.0 (#4894)

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

* Update tutorials.rst (#4897)

* update readme with apex commit

Signed-off-by: ericharper <complex451@gmail.com>

* Add support for Apex distributed Adam optimizer with GPT-3 (#4487)

* Add support for Apex distributed Adam optimizer with GPT-3

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Fix bug in grad clipping with dist Adam

Grad norm was computed over all params, not respecting model parallelism.

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Fix bug with DDP initialization

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Make distopt dependent on megatron_amp_o2

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Fix code formatting

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Handle dist Adam in optimizer unit tests

Signed-off-by: Tim Moon <tmoon@nvidia.com>

Signed-off-by: Tim Moon <tmoon@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* update readme

Signed-off-by: ericharper <complex451@gmail.com>

* update readme

Signed-off-by: ericharper <complex451@gmail.com>

* latent model support

* 1. Debugging.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging.

* update branch

Signed-off-by: ericharper <complex451@gmail.com>

* fix replace_bos_with_pad not found (#6443)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Support Swiglu in TP PP Conversion (#6437)

* Support Swiglu in TP PP Conversion

Signed-off-by: smajumdar <titu1994@gmail.com>

* Guard activation

Signed-off-by: smajumdar <titu1994@gmail.com>

* Guard activation

Signed-off-by: smajumdar <titu1994@gmail.com>

---------

Signed-off-by: smajumdar <titu1994@gmail.com>

* BERT pre-training mp fork to spawn (#6442)

* change bert fork to spawn

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* num_workers=0 fix

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Meagtron encoder decoder fix for empty validation outputs (#6459)

* 1. Meagtron encoder decoder fix for empty validation outputs.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging.

---------

Signed-off-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: Micha Livne <mlivne@nvidia.com>

* Added/updated new Conformer configs (#6426)

* updated conf files.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added confs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* moved longconformer confs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated readme.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated readme.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated batch sizes and added fastconformer ctc streaming configs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated batch sizes.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added hybrid support.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added hybrid support.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

---------

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* reduce workers on NMT CI (#6472)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* move to nvidia megatron repo (#6465)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Megatron KERPLE positional embeddings (#6478)

* [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

[TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [TTS] whitelist broken path fix. (#6412)

* [TTS] whitelist broken path fix.

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [TTS] FastPitch speaker encoder (#6417)

* Add initial codes

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove wemb

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix import

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Restore aligner loss

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add ConditionalInput

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix error and support pre-trained config

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Follow comments

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Rename config

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Change copyright and random weight test

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add initial codes

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix import error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add initial codes

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix dataset error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove reference speaker embedding

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove SV encoder

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Follow comments

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix length type

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix append

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Move error msg

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add look-up into speaker encoder

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add valueerror msg

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Move lookup

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove unused

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Rebase and Fix error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix spk encoder

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Rename n_speakers

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Follow comments

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix n_speakers None error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Sharded manifests for tarred datasets (#6395)

* testing sharded manifests

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* compatibility

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* proper fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* adding flag tot convert_to_tarred_audio_dataset

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* shard_manifests conf param

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* propagating the shard_manifests param

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* propagating the shard_manifests param

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* distributed checks

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* typo

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* typo

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes based on PR comments and tests

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes to convert_to_tarred_audio_dataset.py

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* reversing manifest shards flag

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* excluding manifests from webdataset url expansion

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* expand manifest paths before attempting to cache from datastore

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* explicit use of UTF-8 for manifest i/o

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

---------

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update wfst_text_normalization.rst (#6374)

Add Hungarian (incoming in NeMo-text-processing)

Signed-off-by: Jim O’Regan <jaoregan@tcd.ie>

* Support Swiglu in TP PP Conversion (#6437) (#6451)

* Support Swiglu in TP PP Conversion

* Guard activation

* Guard activation

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>

* Update NeMo_TTS_Primer.ipynb (#6436)

* Update NeMo_TTS_Primer.ipynb

Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch.

Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>

* Update NeMo_TTS_Primer.ipynb

Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point.

Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>

---------

Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>

* add rampup batch size support for Megatron GPT (#6424)

* added rampup batch size support

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* added tests for rampup batch size

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* fixed the typos

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* added assertions

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* changed assertion rules

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* deleted unused imports

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* changed tests for rampup batch size

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* updated rampup batch size tests

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed styling

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* rampup batch size tests changes

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

---------

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461)

* 1. Meagtron encoder decoder fix for empty validation outputs.

* 1. Debugging.

---------

Signed-off-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: Micha Livne <michalivne@users.noreply.github.com>
Co-authored-by: Micha Livne <mlivne@nvidia.com>

* Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448)

* added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

* updated README with the new agg_tokenizer_manifest flag

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

* fixed typo in scripts/speech_recognition/code_switching/README.md

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

* changed agg_tokenizer_manifest to is_lid_manifest

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

---------

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Co-authored-by: Dima Rekesh <bmwshop@gmail.com>

* Added/updated new Conformer configs (#6426) (#6467)

* Update script for ngram rnnt and hat beam search decoding (#6370)

* add rnnt ngram beamsearch script

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add return encoding embedding option

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* update script

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add rnnt and hat ngram decoding script

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add some parameters

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add return_encoder_embeddings parameter to RNNTDecodingConfig

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* replace return_encoder_embeddings parameter

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* generalization of scipt behavior

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove return_encoder_embeddings parameter

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* remove return_encoder_embeddings parameter

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add manual encoder_embeddings calculation

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix beam_width value to 8

Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>

* fix rescoring description

Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>

---------

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>

* BERT pre-training mp fork to spawn (#6442) (#6454)

* change bert fork to spawn

* num_workers=0 fix

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* fix replace_bos_with_pad not found (#6443) (#6450)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* reduce workers on NMT CI (#6472) (#6474)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* 1. Added KERPLE positional embeddings to encoder-decoder.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Added a missing file.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Fixing commits.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Signed-off-by: Jim O’Regan <jaoregan@tcd.ie>
Signed-off-by: smajumdar <titu1994@gmail.com>
Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Signed-off-by: Micha Livne <mlivne@nvidia.com>
Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Dima Rekesh <bmwshop@gmail.com>
Co-authored-by: Jim O’Regan <jaoregan@tcd.ie>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* 1. Added external index sample. (#6462)

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* Fix cache aware hybrid bugs (#6466)

* Update README to add core installation (#6488)

* update README for megatron-core

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* fix

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Fix typos (#6494)

Signed-off-by: smajumdar <titu1994@gmail.com>

* fix broken links r1.18.0 (#6501)

* fix broken links

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* fix broken links

Signed-off-by: Evelina <ebakhturina@nvidia.com>

---------

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* 1. Fixed gaussian hidden transform.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Finished updating hidden loss for MIM.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix custom forward_torch_softmax (#6512)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* [BugFix] Force _get_batch_preds() to keep logits in decoder timestamp… (#6500)

* [BugFix] Force _get_batch_preds() to keep logits in decoder timestamps generator r1.18.0

Signed-off-by: Taejin Park <tango4j@gmail.com>

* ignore keep_logits in FrameBatchASRLogits

Signed-off-by: Taejin Park <tango4j@gmail.com>

---------

Signed-off-by: Taejin Park <tango4j@gmail.com>

* [TTS] fixed broken path. (#6514)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* 1. Added a hiddens module.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix typos (#6523) (#6539)

* Fix typos

Signed-off-by: smajumdar <titu1994@gmail.com>

* Fix typos

Signed-off-by: smajumdar <titu1994@gmail.com>

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
(cherry picked from commit 5468077f5127be1a4c88065de2544f4268b9a6e4)

* added back the fast emit section to the configs. (#6540)

* added back the fast emit section to the configs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added back the fast emit section to the configs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

---------

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* Fix fp16 (#6543)

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* fix (#6529)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* pass .scale instead of scaler object to core (#6545)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Change Megatron Enc Dec model to use persistent_workers (#6548)

* persistent workers

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* fix

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Add FastConformer Hybrid ASR models for EN, ES, IT, DE, PL, HR, UA, BY (#6549)

* Added fastconfomer hybrid asr models for en, es, it, de, pl, hr, ua, by

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>

* updated ASR docs with the fastconformer hybrid checkpoints

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>

* added the fastconformer RNNT and CTC models

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>

---------

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>

* Add scores for FastConformer models (#6557)

Signed-off-by: smajumdar <titu1994@gmail.com>

* Patch transcribe and support offline transcribe for hybrid model (#6550)

Signed-off-by: fayejf <fayejf07@gmail.com>

* Not doing CastToFloat by default (#6524)

* Not doing CastToFloat by default

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

* Added docustring

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

* Dummy commit

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

---------

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

* temp rtd fix (#6568)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update manifest.py for speedup (#6565)

* Update manifest.py

Re-order the checks for faster processing audio filepaths that are already absolute paths

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* Update manifest.py

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

---------

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com>

* Turn autocast off when precision is fp32 (#6554)

* Turn autocast off when precision is fp32

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* address review

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* merge

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* More streaming conformer export fixes (#6567)

Signed-off-by: Greg Clark <grclark@nvidia.com>
Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com>

* Fix batch size reconf for T5 FT for multi-validation (#6582)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Updated Megatron LM encoder/decoder to use cfg for hiddens.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Added support to register externalhidden loss / transforms.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Make tensor split contiguous (#6580)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Patches from main to r1.18.0 for Virtual Parallel (#6592)

* Add interleaved pp support (#6498)

* Add support for Virtual Pipeline Parallel conversion

Signed-off-by: smajumdar <titu1994@gmail.com>

* Add support for Virtual Pipeline Parallel conversion

Signed-off-by: smajumdar <titu1994@gmail.com>

* Switch to megatron core

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
(cherry picked from commit 892987169ef277f328e15b71a5a0c9bd961c8ee7)

* Add patches for Virtual Parallel conversion (#6589)

* Add patches for Virtual Parllel conversion

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
(cherry picked from commit 1d813a372ab51688e3af6395d905a4c0366ffd23)

* Documentation for ASR-TTS models (#6594)

* Add docs about hybrid ASR-TTS models

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Add docs about text-only datasets

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Add docs about ASR-TTS checkpoints

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Add docs about ASR-TTS configs and training

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Clean up

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* ASR-TTS docs: add to api, fix imports

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Clean up

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Wrap optional import

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Revert general ASR import

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

---------

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Update SDP docs (#6485)

* add info about SDP e.g. processor classes in docs

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* add link to SDP docs in README

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* address code review comments and add SDP overview diagram

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* Fix spelling typo

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

---------

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* Create dummy iters to satisy len checks (#6600)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* Restore GPT support for interleaved pipeline parallelism (#6528)

* Restore logic for data-parallel communication with pipeline parallelism in GPT

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Support dynamic attention masks in GPT

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Debug typos

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Debug data iterator caching with interleaved pipeline parallelism

Each model chunk accesses the data iterator multiple times, so we need to cache multiple samples.

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update Megatron-LM commit

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Distinguish between list of data iterators and data iterator that is a list

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Create dummy iters to satisy len checks

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Kludge while waiting for Megatron-LM update

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* set transformers offline to avoid rate limiting

Signed-off-by: ericharper <complex451@gmail.com>

---------

Signed-off-by: Tim Moon <tmoon@nvidia.com>
Signed-off-by: Eric Harper <complex451@gmail.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: ericharper <complex451@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* Patch transcribe_util for steaming mode and add wer calculation back to inference scripts (#6601)

* fix write

Signed-off-by: fayejf <fayejf07@gmail.com>

* decoding ctc

Signed-off-by: fayejf <fayejf07@gmail.com>

* temp set rnnt decoding return_best_hypothesis to true

Signed-off-by: fayejf <fayejf07@gmail.com>

* add wer cal back to transcribe_speech as requested

Signed-off-by: fayejf <fayejf07@gmail.com>

* add wer cal back to speech_to_text_buffered_infer_rnnt  as requested

Signed-off-by: fayejf <fayejf07@gmail.com>

* add wer cal back to speech_to_text_buffered_infer_ctc as requested

Signed-off-by: fayejf <fayejf07@gmail.com>

* style fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* reflect change in asr_evaluator

Signed-off-by: fayejf <fayejf07@gmail.com>

* reflect som and vahid comment

Signed-off-by: fayejf <fayejf07@gmail.com>

* remove return_best_hy=true in transcribe_speech

Signed-off-by: fayejf <fayejf07@gmail.com>

* no text skip

Signed-off-by: fayejf <fayejf07@gmail.com>

---------

Signed-off-by: fayejf <fayejf07@gmail.com>

* 1. Added example conf YAML.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Added support in tensor_parallel.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add hat image to docs (#6619)

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* update core commit hash in readme (#6622)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Patch decoding for PC models (#6630)

* Patch decoding logic for PC models

Signed-off-by: smajumdar <titu1994@gmail.com>

* Patch decoding logic for PC models

Signed-off-by: smajumdar <titu1994@gmail.com>

---------

Signed-off-by: smajumdar <titu1994@gmail.com>

* Fix wer.py where 'errors' variable was not set (#6633)

Fix wer.py where 'errors' variable was not set when both reference and hypothesis are empty strings

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* fix att_context_size bug for older models. (#6635)

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* Add megatron_core to requirements (#6639)

* add megatron_core to requirements

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: ericharper <complex451@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Remove from jenkins (#6641)

* add megatron_core to requirements

Signed-off-by: ericharper <complex451@gmail.com>

* remove from jenkins

Signed-off-by: ericharper <complex451@gmail.com>

---------

Signed-off-by: ericharper <complex451@gmail.com>

* remove dup (#6643)

Signed-off-by: ericharper <complex451@gmail.com>

* 1. Fixed config to use names, and added better error messages.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Added support to pass extra data to hiddens for loss computation.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Working on passing extra data to hiddnes.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Fixed support in loading .nemo without hiddnes module.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Improved and fixed logging of validation and testing.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Fixed training logging.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Fixed logging of hidden loss.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Fixed logging names.
2. Added logging to hiddens and tokens loss.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Fixed conflicts.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

---------

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Oleksii Kuchaiev <okuchaiev@nvidia.com>
Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>
Signed-off-by: smajumdar <smajumdar@nvidia.com>
Signed-off-by: fayejf <fayejf07@gmail.com>
Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com>
Signed-off-by: Virginia Adams <vadams@nvidia.com>
Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
Signed-off-by: ericharper <complex451@gmail.com>
Signed-off-by: Vahid <vnoroozi@nvidia.com>
Signed-off-by: Ante Jukić <ajukic@nvidia.com>
Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>
Signed-off-by: sam1373 <samuelkriman@gmail.com>
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Signed-off-by: ekmb <ebakhturina@nvidia.com>
Signed-off-by: Yang Zhang <yangzhang@nvidia.com>
Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>
Signed-off-by: Tim Moon <tmoon@nvidia.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: smajumdar <titu1994@gmail.com>
Signed-off-by: Micha Livne <mlivne@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Signed-off-by: Jim O’Regan <jaoregan@tcd.ie>
Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Signed-off-by: Evelina <ebakhturina@nvidia.com>
Signed-off-by: Taejin Park <tango4j@gmail.com>
Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Signed-off-by: Greg Clark <grclark@nvidia.com>
Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>
Signed-off-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Oleksii Kuchaiev <okuchaiev@nvidia.com>
Co-authored-by: Jocelyn <jocelynh@nvidia.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Co-authored-by: fayejf <36722593+fayejf@users.noreply.github.com>
Co-authored-by: bene-ges <61418381+bene-ges@users.noreply.github.com>
Co-authored-by: Alexandra Antonova <aleksandraa@nvidia.com>
Co-authored-by: Virginia Adams <78445382+vadam5@users.noreply.github.com>
Co-authored-by: Zhilin Wang <wangzhilin12061996@hotmail.com>
Co-authored-by: Vladimir Bataev <vbataev@nvidia.com>
Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com>
Co-authored-by: anteju <108555623+anteju@users.noreply.github.com>
Co-authored-by: Ante Jukić <ajukic@nvidia.com>
Co-authored-by: David <amosalla@asu.edu>
Co-authored-by: David Mosallanezhad <dmosallanezh@nvidia.com>
Co-authored-by: Samuel Kriman <samuelkriman@gmail.com>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>
Co-authored-by: Evelina <10428420+ekmb@users.noreply.github.com>
Co-authored-by: Sean Naren <sean.narenthiran@gmail.com>
Co-authored-by: Yang Zhang <yzhang123@users.noreply.github.com>
Co-authored-by: Sean Naren <snarenthiran@nvidia.com>
Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>
Co-authored-by: Neha Tadimeti <ntadimeti@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>
Co-authored-by: Dima Rekesh <bmwshop@gmail.com>
Co-authored-by: Jim O’Regan <jaoregan@tcd.ie>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Co-authored-by: Taejin Park <tango4j@gmail.com>
Co-authored-by: Boris Fomitchev <borisfom@users.noreply.github.com>
Co-authored-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Co-authored-by: Greg Clark <grclark@nvidia.com>
Co-authored-by: Elena Rastorgueva <80532067+erastorgueva-nv@users.noreply.github.com>
Signed-off-by: dorotat <dorotat@nvidia.com>
yaoyu-33 pushed a commit that referenced this pull request Oct 16, 2023
* [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

[TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

---------




* [TTS] whitelist broken path fix. (#6412)

* [TTS] whitelist broken path fix.



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------




* [TTS] FastPitch speaker encoder (#6417)

* Add initial codes



* Remove wemb



* Fix import



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Restore aligner loss



* Add ConditionalInput



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix error and support pre-trained config



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Follow comments



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Rename config



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Change copyright and random weight test



* Add initial codes



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix import error



* Add initial codes



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix dataset error



* Remove reference speaker embedding



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Remove SV encoder



* Follow comments



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix length type



* Fix append



* Move error msg



* Add look-up into speaker encoder



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Add valueerror msg



* Move lookup



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Remove unused



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci



* Fix error



* Rebase and Fix error



* Fix spk encoder



* Rename n_speakers



* Follow comments



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix n_speakers None error



---------




* Sharded manifests for tarred datasets (#6395)

* testing sharded manifests



* compatibility



* proper fixes



* adding flag tot convert_to_tarred_audio_dataset



* shard_manifests conf param



* propagating the shard_manifests param



* propagating the shard_manifests param



* distributed checks



* typo



* typo



* fixes



* fixes



* fixes



* fixes



* fixes



* fixes



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes based on PR comments and tests



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes to convert_to_tarred_audio_dataset.py



* reversing manifest shards flag



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests



* excluding manifests from webdataset url expansion



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* expand manifest paths before attempting to cache from datastore



* explicit use of UTF-8 for manifest i/o



---------




* Update wfst_text_normalization.rst (#6374)

Add Hungarian (incoming in NeMo-text-processing)



* Support Swiglu in TP PP Conversion (#6437) (#6451)

* Support Swiglu in TP PP Conversion



* Guard activation



* Guard activation



---------




* Update NeMo_TTS_Primer.ipynb (#6436)

* Update NeMo_TTS_Primer.ipynb

Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch.



* Update NeMo_TTS_Primer.ipynb

Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point.



---------



* add rampup batch size support for Megatron GPT (#6424)

* added rampup batch size support



* added tests for rampup batch size



* fixed the typos



* added assertions



* changed assertion rules



* deleted unused imports



* changed tests for rampup batch size



* updated rampup batch size tests



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed styling



* rampup batch size tests changes



---------







* Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461)

* 1. Meagtron encoder decoder fix for empty validation outputs.



* 1. Debugging.

---------





* Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448)

* added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default



* updated README with the new agg_tokenizer_manifest flag



* fixed typo in scripts/speech_recognition/code_switching/README.md



* changed agg_tokenizer_manifest to is_lid_manifest



---------




* Added/updated new Conformer configs (#6426) (#6467)

* Update script for ngram rnnt and hat beam search decoding (#6370)

* add rnnt ngram beamsearch script



* add return encoding embedding option



* update script



* add rnnt and hat ngram decoding script



* add some parameters



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add return_encoder_embeddings parameter to RNNTDecodingConfig



* replace return_encoder_embeddings parameter



* generalization of scipt behavior



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove return_encoder_embeddings parameter



* remove return_encoder_embeddings parameter



* add manual encoder_embeddings calculation



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix beam_width value to 8



* fix rescoring description



---------






* BERT pre-training mp fork to spawn (#6442) (#6454)

* change bert fork to spawn



* num_workers=0 fix



---------




* fix replace_bos_with_pad not found (#6443) (#6450)




* reduce workers on NMT CI (#6472) (#6474)




* 1. Added KERPLE positional embeddings to encoder-decoder.



* 1. Added a missing file.



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Fixing commits.



* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Signed-off-by: Jim O’Regan <jaoregan@tcd.ie>
Signed-off-by: smajumdar <titu1994@gmail.com>
Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Signed-off-by: Micha Livne <mlivne@nvidia.com>
Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Micha Livne <michalivne@users.noreply.github.com>
Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Dima Rekesh <bmwshop@gmail.com>
Co-authored-by: Jim O’Regan <jaoregan@tcd.ie>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
rohitrango pushed a commit to rohitrango/NeMo that referenced this pull request Jun 25, 2024
* [TTS] bugfix for missing configs. (#4725)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* docs typo fix

Signed-off-by: Oleksii Kuchaiev <okuchaiev@nvidia.com>

* Fix pynini install in TTS tutorials (#4729)

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

* Fix ASR notebooks (#4738)

Signed-off-by: smajumdar <smajumdar@nvidia.com>

Signed-off-by: smajumdar <smajumdar@nvidia.com>

* Multilingual VAD model (#4734)

* add ngc link

Signed-off-by: fayejf <fayejf07@gmail.com>

* add tuned VAD config on ASR data

Signed-off-by: fayejf <fayejf07@gmail.com>

* yaml note

Signed-off-by: fayejf <fayejf07@gmail.com>

* update vad asr notebook with mVAD

Signed-off-by: fayejf <fayejf07@gmail.com>

* update vad infer config comment

Signed-off-by: fayejf <fayejf07@gmail.com>

* fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* mvad sd config for ch109

Signed-off-by: fayejf <fayejf07@gmail.com>

* update sd readme

Signed-off-by: fayejf <fayejf07@gmail.com>

* add new mVAD model to doc

Signed-off-by: fayejf <fayejf07@gmail.com>

* style fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* update sd tutorial with mVAD

Signed-off-by: fayejf <fayejf07@gmail.com>

* typo fix

Signed-off-by: fayejf <fayejf07@gmail.com>

Signed-off-by: fayejf <fayejf07@gmail.com>

* publish pretrained itn t5 model for English (#4748)

Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com>

Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com>
Co-authored-by: Alexandra Antonova <aleksandraa@nvidia.com>

* Updated docs and doc paths (#4754)

* Updated docs and doc paths

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Update Multitask_Prompt_and_PTuning.ipynb

* Update README.rst

* Changed branch name to use single quotes

Signed-off-by: Virginia Adams <vadams@nvidia.com>

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* fix bug relating to ddp strategy in joint intent slot classification tutorial (#4762)

* [TTS] updated config with a German IPA phoneme tokenizer (#4756)

* [TTS] added a German IPA phoneme tokenizer
* [TTS][ASR] enabled customized arguments for trimming the leading and trailing silence.
* [TTS] disabled spline interpolation for beta-binomial distribution. Let it generate align prior and save to disks. Use a new phoneme tokenizer.
* [TTS] use consistent spline interpolation with fastpitch checkpoint when generating mel-spectrograms for hifigan finetune.

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* Update r1.11 to new heteronyms list (#4745)

* Update configs to new heteronyms list
* Remove old heteronyms list, add alt 'merchandise' pron to CMUdict
* Update remaining references to old heteronyms list

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* [TTS] Add multi-speaker German FastPitch and HiFiGAN NGC checkpoints (#4763)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* [TTS] Add single male speaker German FastPitch and HiFiGAN NGC checkpoints (#4770)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* Update CMUdict with more recent 0.7b entries (#4768)

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* Install pynini in docker container (#4733)

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Fix tutorial formatting (#4778)

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

* [TTS] deprecated old scripts for ljspeech. (#4780)

* deprecated old scripts for ljspeech.
* removed relevent function calls in TTS docs.

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* update branch and typos (#4788)

Signed-off-by: ericharper <complex451@gmail.com>

Signed-off-by: ericharper <complex451@gmail.com>

* Adding support for models trained with full context for cache-aware streaming. (#4687)

* added support for models trained with full context.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* dropped seq_range

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed indexing in caching methods.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated docs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* addressed comments.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* change frame-wise to cache-aware.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* change frame-wise to cache-aware.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* change frame-wise to cache-aware.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* fixed code style.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* Update megatron encoder decoder model to support py37 for colab (#4791)

* [ASR] Add pretrained ASR models for Croatian (#4682)

* [ASR] Add pretrained ASR models for Croatian

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

* Fix style for import

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

Signed-off-by: Ante Jukić <ajukic@nvidia.com>
Co-authored-by: Ante Jukić <ajukic@nvidia.com>
Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>

* added/fixed export for Megatron models (#4712)

* added/fixed export for Megatron models

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

* fixed style

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

* fixed FusedScaleMaskSoftmax in BioMegatron

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

* included comments

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>
Co-authored-by: David Mosallanezhad <dmosallanezh@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* update branch for qa notebook

Signed-off-by: ericharper <complex451@gmail.com>

* Fix initializing weights from ptl ckpt with exclude (#4807)

Signed-off-by: sam1373 <samuelkriman@gmail.com>

Signed-off-by: sam1373 <samuelkriman@gmail.com>

* Fix index error from addition of voiced_mask and p_voiced (#4811)

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

* T5 prompt learning fixes (#4771)

* RPE, hidden size and config fixes

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update to reflect new config names

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Sentencepiece fixes

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix finetuning

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Add encoder seq len to gpt

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Add finetune eval script

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix name

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update Jenkinsfile

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update config

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix CI test

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update check

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Backward compat

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Update CI test

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Split rank for Enc-Dec models

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Address comments

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Co-authored-by: Virginia Adams <78445382+vadam5@users.noreply.github.com>

* G2P docs (#4841)

* g2p docs added

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* fix references

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* address review feedback

Signed-off-by: ekmb <ebakhturina@nvidia.com>

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* Fix providing glue in seq2seq eval (#4843)

* Fix providing glue in seq2seq eval

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Fix

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Updated inference code and squad scripts (#4835)

* Updated inference code and squad scripts

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Reverted GPT & T5 inference files back to use NLPDDPlugin

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Overwrite frozen LM to use fused adam

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Added padded vocab size

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Fixed val check interval value

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Python format fix

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Make t5 prompt learning preds write to file

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Added back dp=1 check

Signed-off-by: Virginia Adams <vadams@nvidia.com>

Signed-off-by: Virginia Adams <vadams@nvidia.com>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>

* Update README.rst

* Fix uppercasing mismatch for IPA heteronyms (#4860)

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

* Set the number of workers to 0 for validation and test sets in all enc-dec models (#4790)

* Set workers to 0 for validation and test

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Revert pin memory

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Co-authored-by: Sean Naren <sean.narenthiran@gmail.com>

* Fix mha (#4866)

* fix bug in mha forward function related to cache update return type

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* fix lgtm

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>
Co-authored-by: Sean Naren <snarenthiran@nvidia.com>

* ipa bug fix (#4871)

Signed-off-by: ekmb <ebakhturina@nvidia.com>

Signed-off-by: ekmb <ebakhturina@nvidia.com>

* Fix Megatron NMT consumed samples and ckpt_to_nemo split rank (#4884)

* Fix nmt and ckpt_to_nemo

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* Style

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* added utf8 encoding (#4892)

Signed-off-by: Virginia Adams <vadams@nvidia.com>

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* 1. Applying the same patch to r1.11.0 (#4894)

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

* Update tutorials.rst (#4897)

* update readme with apex commit

Signed-off-by: ericharper <complex451@gmail.com>

* Add support for Apex distributed Adam optimizer with GPT-3 (#4487)

* Add support for Apex distributed Adam optimizer with GPT-3

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Fix bug in grad clipping with dist Adam

Grad norm was computed over all params, not respecting model parallelism.

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Fix bug with DDP initialization

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Make distopt dependent on megatron_amp_o2

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Fix code formatting

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Handle dist Adam in optimizer unit tests

Signed-off-by: Tim Moon <tmoon@nvidia.com>

Signed-off-by: Tim Moon <tmoon@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* update readme

Signed-off-by: ericharper <complex451@gmail.com>

* update readme

Signed-off-by: ericharper <complex451@gmail.com>

* latent model support

* 1. Debugging.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging.

* update branch

Signed-off-by: ericharper <complex451@gmail.com>

* fix replace_bos_with_pad not found (#6443)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Support Swiglu in TP PP Conversion (#6437)

* Support Swiglu in TP PP Conversion

Signed-off-by: smajumdar <titu1994@gmail.com>

* Guard activation

Signed-off-by: smajumdar <titu1994@gmail.com>

* Guard activation

Signed-off-by: smajumdar <titu1994@gmail.com>

---------

Signed-off-by: smajumdar <titu1994@gmail.com>

* BERT pre-training mp fork to spawn (#6442)

* change bert fork to spawn

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* num_workers=0 fix

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Meagtron encoder decoder fix for empty validation outputs (#6459)

* 1. Meagtron encoder decoder fix for empty validation outputs.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging.

---------

Signed-off-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: Micha Livne <mlivne@nvidia.com>

* Added/updated new Conformer configs (#6426)

* updated conf files.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added confs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* moved longconformer confs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated readme.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated readme.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated batch sizes and added fastconformer ctc streaming configs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* updated batch sizes.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added hybrid support.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added hybrid support.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

---------

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* reduce workers on NMT CI (#6472)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* move to nvidia megatron repo (#6465)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Megatron KERPLE positional embeddings (#6478)

* [TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

[TTS] FastPitch adapter fine-tune and conditional layer normalization (#6416)

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [TTS] whitelist broken path fix. (#6412)

* [TTS] whitelist broken path fix.

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [TTS] FastPitch speaker encoder (#6417)

* Add initial codes

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove wemb

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix import

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Restore aligner loss

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add ConditionalInput

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix error and support pre-trained config

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Follow comments

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Rename config

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Change copyright and random weight test

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add initial codes

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix import error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add initial codes

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix dataset error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove reference speaker embedding

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove SV encoder

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Follow comments

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix length type

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix append

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Move error msg

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add look-up into speaker encoder

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Add valueerror msg

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Move lookup

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Remove unused

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Rebase and Fix error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Fix spk encoder

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Rename n_speakers

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* Follow comments

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix n_speakers None error

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Sharded manifests for tarred datasets (#6395)

* testing sharded manifests

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* compatibility

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* proper fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* adding flag tot convert_to_tarred_audio_dataset

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* shard_manifests conf param

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* propagating the shard_manifests param

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* propagating the shard_manifests param

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* distributed checks

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* typo

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* typo

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* fixes

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes based on PR comments and tests

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes to convert_to_tarred_audio_dataset.py

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* reversing manifest shards flag

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* excluding manifests from webdataset url expansion

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* expand manifest paths before attempting to cache from datastore

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

* explicit use of UTF-8 for manifest i/o

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>

---------

Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update wfst_text_normalization.rst (#6374)

Add Hungarian (incoming in NeMo-text-processing)

Signed-off-by: Jim O’Regan <jaoregan@tcd.ie>

* Support Swiglu in TP PP Conversion (#6437) (#6451)

* Support Swiglu in TP PP Conversion



* Guard activation



* Guard activation



---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>

* Update NeMo_TTS_Primer.ipynb (#6436)

* Update NeMo_TTS_Primer.ipynb

Changed a mistake in line 782. Instead of frequency band (ie. pitch) we should write frequency bin. Note that frequency bins in FFT are not related to pitch.

Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>

* Update NeMo_TTS_Primer.ipynb

Corrected the description of spectrogram and mel spectrogram calculations in lines 782 & 783 and added a fourth point to the description and added a reference for more mathematical details at the end of this point.

Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>

---------

Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>

* add rampup batch size support for Megatron GPT (#6424)

* added rampup batch size support

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* added tests for rampup batch size

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* fixed the typos

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* added assertions

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* changed assertion rules

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* deleted unused imports

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* changed tests for rampup batch size

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* updated rampup batch size tests

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed styling

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

* rampup batch size tests changes

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

---------

Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Meagtron encoder decoder fix for empty validation outputs (#6459) (#6461)

* 1. Meagtron encoder decoder fix for empty validation outputs.



* 1. Debugging.

---------

Signed-off-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: Micha Livne <michalivne@users.noreply.github.com>
Co-authored-by: Micha Livne <mlivne@nvidia.com>

* Code-Switching dataset creation - upgrading to aggregate tokenizer manifest format (#6448)

* added functionality to create agg tokenizer compatible manifest for CS, flag to use this mode by default

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

* updated README with the new agg_tokenizer_manifest flag

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

* fixed typo in scripts/speech_recognition/code_switching/README.md

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

* changed agg_tokenizer_manifest to is_lid_manifest

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>

---------

Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Co-authored-by: Dima Rekesh <bmwshop@gmail.com>

* Added/updated new Conformer configs (#6426) (#6467)

* Update script for ngram rnnt and hat beam search decoding (#6370)

* add rnnt ngram beamsearch script

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add return encoding embedding option

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* update script

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add rnnt and hat ngram decoding script

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add some parameters

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add return_encoder_embeddings parameter to RNNTDecodingConfig

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* replace return_encoder_embeddings parameter

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* generalization of scipt behavior

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove return_encoder_embeddings parameter

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* remove return_encoder_embeddings parameter

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* add manual encoder_embeddings calculation

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix beam_width value to 8

Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>

* fix rescoring description

Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>

---------

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>

* BERT pre-training mp fork to spawn (#6442) (#6454)

* change bert fork to spawn



* num_workers=0 fix



---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* fix replace_bos_with_pad not found (#6443) (#6450)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* reduce workers on NMT CI (#6472) (#6474)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* 1. Added KERPLE positional embeddings to encoder-decoder.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Added a missing file.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Fixing commits.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

* 1. Debugging.

---------

Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Signed-off-by: Jim O’Regan <jaoregan@tcd.ie>
Signed-off-by: smajumdar <titu1994@gmail.com>
Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Signed-off-by: Micha Livne <mlivne@nvidia.com>
Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Dima Rekesh <bmwshop@gmail.com>
Co-authored-by: Jim O’Regan <jaoregan@tcd.ie>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Micha Livne <mlivne@nvidia.com>
Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* 1. Added external index sample. (#6462)

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* Fix cache aware hybrid bugs (#6466)

* Update README to add core installation (#6488)

* update README for megatron-core

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* fix

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Fix typos (#6494)

Signed-off-by: smajumdar <titu1994@gmail.com>

* fix broken links r1.18.0 (#6501)

* fix broken links

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* fix broken links

Signed-off-by: Evelina <ebakhturina@nvidia.com>

---------

Signed-off-by: Evelina <ebakhturina@nvidia.com>

* 1. Fixed gaussian hidden transform.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Finished updating hidden loss for MIM.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix custom forward_torch_softmax (#6512)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* [BugFix] Force _get_batch_preds() to keep logits in decoder timestamp… (#6500)

* [BugFix] Force _get_batch_preds() to keep logits in decoder timestamps generator r1.18.0

Signed-off-by: Taejin Park <tango4j@gmail.com>

* ignore keep_logits in FrameBatchASRLogits

Signed-off-by: Taejin Park <tango4j@gmail.com>

---------

Signed-off-by: Taejin Park <tango4j@gmail.com>

* [TTS] fixed broken path. (#6514)

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

* 1. Added a hiddens module.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix typos (#6523) (#6539)

* Fix typos

Signed-off-by: smajumdar <titu1994@gmail.com>

* Fix typos

Signed-off-by: smajumdar <titu1994@gmail.com>

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
(cherry picked from commit 7fa47bfcd359f1ea80ad056d8bb2d78f97c6206a)

* added back the fast emit section to the configs. (#6540)

* added back the fast emit section to the configs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* added back the fast emit section to the configs.

Signed-off-by: Vahid <vnoroozi@nvidia.com>

---------

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* Fix fp16 (#6543)

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

* fix (#6529)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* pass .scale instead of scaler object to core (#6545)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Change Megatron Enc Dec model to use persistent_workers (#6548)

* persistent workers

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* fix

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* Add FastConformer Hybrid ASR models for EN, ES, IT, DE, PL, HR, UA, BY (#6549)

* Added fastconfomer hybrid asr models for en, es, it, de, pl, hr, ua, by

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>

* updated ASR docs with the fastconformer hybrid checkpoints

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>

* added the fastconformer RNNT and CTC models

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>

---------

Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>

* Add scores for FastConformer models (#6557)

Signed-off-by: smajumdar <titu1994@gmail.com>

* Patch transcribe and support offline transcribe for hybrid model (#6550)

Signed-off-by: fayejf <fayejf07@gmail.com>

* Not doing CastToFloat by default (#6524)

* Not doing CastToFloat by default

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

* Added docustring

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

* Dummy commit

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

---------

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

* temp rtd fix (#6568)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update manifest.py for speedup (#6565)

* Update manifest.py

Re-order the checks for faster processing audio filepaths that are already absolute paths

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* Update manifest.py

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

---------

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com>

* Turn autocast off when precision is fp32 (#6554)

* Turn autocast off when precision is fp32

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* address review

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* merge

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>

* More streaming conformer export fixes (#6567)

Signed-off-by: Greg Clark <grclark@nvidia.com>
Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com>

* Fix batch size reconf for T5 FT for multi-validation (#6582)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Updated Megatron LM encoder/decoder to use cfg for hiddens.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Added support to register externalhidden loss / transforms.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Make tensor split contiguous (#6580)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Patches from main to r1.18.0 for Virtual Parallel (#6592)

* Add interleaved pp support (#6498)

* Add support for Virtual Pipeline Parallel conversion

Signed-off-by: smajumdar <titu1994@gmail.com>

* Add support for Virtual Pipeline Parallel conversion

Signed-off-by: smajumdar <titu1994@gmail.com>

* Switch to megatron core

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
(cherry picked from commit fa9ab5b41301c9d92c97bf37894f9c94de6b91b5)

* Add patches for Virtual Parallel conversion (#6589)

* Add patches for Virtual Parllel conversion

Signed-off-by: smajumdar <titu1994@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: smajumdar <titu1994@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
(cherry picked from commit 4d639ef36b10d8536db02bc47ea30f4acbc061bb)

* Documentation for ASR-TTS models (#6594)

* Add docs about hybrid ASR-TTS models

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Add docs about text-only datasets

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Add docs about ASR-TTS checkpoints

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Add docs about ASR-TTS configs and training

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Clean up

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* ASR-TTS docs: add to api, fix imports

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Clean up

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Wrap optional import

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Revert general ASR import

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

---------

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

* Update SDP docs (#6485)

* add info about SDP e.g. processor classes in docs

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* add link to SDP docs in README

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* address code review comments and add SDP overview diagram

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* Fix spelling typo

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

---------

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

* Create dummy iters to satisy len checks (#6600)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* Restore GPT support for interleaved pipeline parallelism (#6528)

* Restore logic for data-parallel communication with pipeline parallelism in GPT

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Support dynamic attention masks in GPT

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Debug typos

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Debug data iterator caching with interleaved pipeline parallelism

Each model chunk accesses the data iterator multiple times, so we need to cache multiple samples.

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update Megatron-LM commit

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Distinguish between list of data iterators and data iterator that is a list

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Create dummy iters to satisy len checks

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Kludge while waiting for Megatron-LM update

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* set transformers offline to avoid rate limiting

Signed-off-by: ericharper <complex451@gmail.com>

---------

Signed-off-by: Tim Moon <tmoon@nvidia.com>
Signed-off-by: Eric Harper <complex451@gmail.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: ericharper <complex451@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>

* Patch transcribe_util for steaming mode and add wer calculation back to inference scripts (#6601)

* fix write

Signed-off-by: fayejf <fayejf07@gmail.com>

* decoding ctc

Signed-off-by: fayejf <fayejf07@gmail.com>

* temp set rnnt decoding return_best_hypothesis to true

Signed-off-by: fayejf <fayejf07@gmail.com>

* add wer cal back to transcribe_speech as requested

Signed-off-by: fayejf <fayejf07@gmail.com>

* add wer cal back to speech_to_text_buffered_infer_rnnt  as requested

Signed-off-by: fayejf <fayejf07@gmail.com>

* add wer cal back to speech_to_text_buffered_infer_ctc as requested

Signed-off-by: fayejf <fayejf07@gmail.com>

* style fix

Signed-off-by: fayejf <fayejf07@gmail.com>

* reflect change in asr_evaluator

Signed-off-by: fayejf <fayejf07@gmail.com>

* reflect som and vahid comment

Signed-off-by: fayejf <fayejf07@gmail.com>

* remove return_best_hy=true in transcribe_speech

Signed-off-by: fayejf <fayejf07@gmail.com>

* no text skip

Signed-off-by: fayejf <fayejf07@gmail.com>

---------

Signed-off-by: fayejf <fayejf07@gmail.com>

* 1. Added example conf YAML.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Added support in tensor_parallel.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add hat image to docs (#6619)

Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>

* update core commit hash in readme (#6622)

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

* Patch decoding for PC models (#6630)

* Patch decoding logic for PC models

Signed-off-by: smajumdar <titu1994@gmail.com>

* Patch decoding logic for PC models

Signed-off-by: smajumdar <titu1994@gmail.com>

---------

Signed-off-by: smajumdar <titu1994@gmail.com>

* Fix wer.py where 'errors' variable was not set (#6633)

Fix wer.py where 'errors' variable was not set when both reference and hypothesis are empty strings

Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>

* fix att_context_size bug for older models. (#6635)

Signed-off-by: Vahid <vnoroozi@nvidia.com>

* Add megatron_core to requirements (#6639)

* add megatron_core to requirements

Signed-off-by: ericharper <complex451@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: ericharper <complex451@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Remove from jenkins (#6641)

* add megatron_core to requirements

Signed-off-by: ericharper <complex451@gmail.com>

* remove from jenkins

Signed-off-by: ericharper <complex451@gmail.com>

---------

Signed-off-by: ericharper <complex451@gmail.com>

* remove dup (#6643)

Signed-off-by: ericharper <complex451@gmail.com>

* 1. Fixed config to use names, and added better error messages.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Added support to pass extra data to hiddens for loss computation.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Working on passing extra data to hiddnes.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Fixed support in loading .nemo without hiddnes module.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Improved and fixed logging of validation and testing.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Fixed training logging.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Fixed logging of hidden loss.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Fixed logging names.
2. Added logging to hiddens and tokens loss.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Fixed conflicts.

Signed-off-by: Micha Livne <mlivne@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

* 1. Debugging. Signed-off-by: Micha Livne <mlivne@nvidia.com>

---------

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Oleksii Kuchaiev <okuchaiev@nvidia.com>
Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>
Signed-off-by: smajumdar <smajumdar@nvidia.com>
Signed-off-by: fayejf <fayejf07@gmail.com>
Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com>
Signed-off-by: Virginia Adams <vadams@nvidia.com>
Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
Signed-off-by: ericharper <complex451@gmail.com>
Signed-off-by: Vahid <vnoroozi@nvidia.com>
Signed-off-by: Ante Jukić <ajukic@nvidia.com>
Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>
Signed-off-by: sam1373 <samuelkriman@gmail.com>
Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>
Signed-off-by: ekmb <ebakhturina@nvidia.com>
Signed-off-by: Yang Zhang <yangzhang@nvidia.com>
Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>
Signed-off-by: Tim Moon <tmoon@nvidia.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Signed-off-by: smajumdar <titu1994@gmail.com>
Signed-off-by: Micha Livne <mlivne@nvidia.com>
Signed-off-by: hsiehjackson <c2hsieh@ucsd.edu>
Signed-off-by: Dima Rekesh <bmwshop@gmail.com>
Signed-off-by: Jim O’Regan <jaoregan@tcd.ie>
Signed-off-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Signed-off-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Signed-off-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
Signed-off-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Signed-off-by: Evelina <ebakhturina@nvidia.com>
Signed-off-by: Taejin Park <tango4j@gmail.com>
Signed-off-by: KunalDhawan <kunaldhawan97@gmail.com>
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
Signed-off-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Signed-off-by: Greg Clark <grclark@nvidia.com>
Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>
Signed-off-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Co-authored-by: Oleksii Kuchaiev <okuchaiev@nvidia.com>
Co-authored-by: Jocelyn <jocelynh@nvidia.com>
Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
Co-authored-by: fayejf <36722593+fayejf@users.noreply.github.com>
Co-authored-by: bene-ges <61418381+bene-ges@users.noreply.github.com>
Co-authored-by: Alexandra Antonova <aleksandraa@nvidia.com>
Co-authored-by: Virginia Adams <78445382+vadam5@users.noreply.github.com>
Co-authored-by: Zhilin Wang <wangzhilin12061996@hotmail.com>
Co-authored-by: Vladimir Bataev <vbataev@nvidia.com>
Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com>
Co-authored-by: Eric Harper <complex451@gmail.com>
Co-authored-by: Vahid Noroozi <VahidooX@users.noreply.github.com>
Co-authored-by: anteju <108555623+anteju@users.noreply.github.com>
Co-authored-by: Ante Jukić <ajukic@nvidia.com>
Co-authored-by: David <amosalla@asu.edu>
Co-authored-by: David Mosallanezhad <dmosallanezh@nvidia.com>
Co-authored-by: Samuel Kriman <samuelkriman@gmail.com>
Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca>
Co-authored-by: Evelina <10428420+ekmb@users.noreply.github.com>
Co-authored-by: Sean Naren <sean.narenthiran@gmail.com>
Co-authored-by: Yang Zhang <yzhang123@users.noreply.github.com>
Co-authored-by: Sean Naren <snarenthiran@nvidia.com>
Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>
Co-authored-by: Neha Tadimeti <ntadimeti@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Cheng-Ping Hsieh <37269846+hsiehjackson@users.noreply.github.com>
Co-authored-by: Dima Rekesh <bmwshop@gmail.com>
Co-authored-by: Jim O’Regan <jaoregan@tcd.ie>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Mostafa Ghorbandoost <mos.ghorbandoost@gmail.com>
Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Co-authored-by: Kunal Dhawan <kunaldhawan97@gmail.com>
Co-authored-by: Andrei Andrusenko <52885736+andrusenkoau@users.noreply.github.com>
Co-authored-by: Taejin Park <tango4j@gmail.com>
Co-authored-by: Boris Fomitchev <borisfom@users.noreply.github.com>
Co-authored-by: He Huang (Steve) <105218074+stevehuang52@users.noreply.github.com>
Co-authored-by: Greg Clark <grclark@nvidia.com>
Co-authored-by: Elena Rastorgueva <80532067+erastorgueva-nv@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant