-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding Hybrid RNNT-CTC model #5364
Conversation
Signed-off-by: Vahid <vnoroozi@nvidia.com>
Signed-off-by: Vahid <vnoroozi@nvidia.com>
Signed-off-by: Vahid <vnoroozi@nvidia.com>
for more information, see https://pre-commit.ci
Signed-off-by: Vahid <vnoroozi@nvidia.com>
for more information, see https://pre-commit.ci
Signed-off-by: Vahid <vnoroozi@nvidia.com>
Signed-off-by: Vahid <vnoroozi@nvidia.com>
This pull request introduces 16 alerts when merging fa92433 into 265056e - view on LGTM.com new alerts:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Really awesome. Needs plenty of cleanup especially for the subword model but it should be doable.
examples/asr/conf/conformer/hybrid_transducer_ctc/conformer_hybrid_transducer_ctc_bpe.yaml
Show resolved
Hide resolved
Signed-off-by: Vahid <vnoroozi@nvidia.com>
for more information, see https://pre-commit.ci
Signed-off-by: Vahid <vnoroozi@nvidia.com>
This pull request introduces 6 alerts when merging 2b3902a into 5665f14 - view on LGTM.com new alerts:
Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog. |
Signed-off-by: Vahid <vnoroozi@nvidia.com>
for more information, see https://pre-commit.ci
Signed-off-by: Vahid <vnoroozi@nvidia.com>
This pull request introduces 9 alerts when merging bab1fc2 into 5665f14 - view on LGTM.com new alerts:
Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog. |
Signed-off-by: Vahid <vnoroozi@nvidia.com>
This pull request introduces 9 alerts when merging a0d584e into 5665f14 - view on LGTM.com new alerts:
Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog. |
Signed-off-by: Vahid <vnoroozi@nvidia.com>
Signed-off-by: Vahid <vnoroozi@nvidia.com>
This pull request introduces 8 alerts when merging 31d9f07 into 25f9ab9 - view on LGTM.com new alerts:
Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog. |
Signed-off-by: Vahid <vnoroozi@nvidia.com>
Signed-off-by: Vahid <vnoroozi@nvidia.com>
Signed-off-by: Vahid <vnoroozi@nvidia.com>
Signed-off-by: Vahid <vnoroozi@nvidia.com>
Signed-off-by: Vahid <vnoroozi@nvidia.com>
Signed-off-by: Vahid <vnoroozi@nvidia.com>
Signed-off-by: Vahid <vnoroozi@nvidia.com>
This pull request introduces 1 alert when merging b5a56b4 into 5c1d59e - view on LGTM.com new alerts:
Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. It looks like GitHub code scanning with CodeQL is already set up for this repo, so no further action is needed 🚀. For more information, please check out our post on the GitHub blog. |
Signed-off-by: Vahid <vnoroozi@nvidia.com>
for more information, see https://pre-commit.ci
Signed-off-by: Vahid <vnoroozi@nvidia.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks awesome, @bmwshop for final review
examples/asr/conf/conformer/hybrid_transducer_ctc/conformer_hybrid_transducer_ctc_bpe.yaml
Show resolved
Hide resolved
examples/asr/conf/conformer/hybrid_transducer_ctc/conformer_hybrid_transducer_ctc_char.yaml
Show resolved
Hide resolved
Returns: None | ||
|
||
""" | ||
if isinstance(new_tokenizer_dir, DictConfig): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@bmwshop can you look into whether you can pull up this preconfig stuff into utility method of ASRBPEMixin in the future.
Signed-off-by: Vahid <vnoroozi@nvidia.com>
Signed-off-by: Vahid <vnoroozi@nvidia.com>
Signed-off-by: Vahid <vnoroozi@nvidia.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PR looks great, thanks ! We should add docstring to some functions, plus note it in README.md cause its a very cool functionality.
Minor comments
|
||
// stage('L2: Hybrid ASR RNNT-CTC dev run') { | ||
// when { | ||
// anyOf { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Dont we want to uncomment the test now?
|
||
|
||
class TestEncDecHybridRNNTCTCModel: | ||
@pytest.mark.skipif( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Where is subword level test file?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks awesome !
* added initial code. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added the confs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added the confs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * changed name from joint to hybrid. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed format. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed format. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * addressed comments. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * addressed comments. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added docs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added docs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added docs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added docs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * addec CI test. Signed-off-by: Vahid <vnoroozi@nvidia.com> * addec CI test. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bugs in change_vocabs. Signed-off-by: vahidoox <vnoroozi@nvidia.com> * fixed bugs in change_vocabs. Signed-off-by: vahidoox <vnoroozi@nvidia.com> * fixed style. Signed-off-by: vahidoox <vnoroozi@nvidia.com> * fixed style. Signed-off-by: vahidoox <vnoroozi@nvidia.com> * fixed style. Signed-off-by: vahidoox <vnoroozi@nvidia.com> * raise error for aux_ctc. Signed-off-by: Vahid <vnoroozi@nvidia.com> * raise error for aux_ctc. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * raise error for aux_ctc. Signed-off-by: Vahid <vnoroozi@nvidia.com> * raise error for aux_ctc. Signed-off-by: Vahid <vnoroozi@nvidia.com> * updated the streaming names. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added unittests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added unittests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added unittests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed tests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed tests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed tests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed tests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed tests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed tests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added methods. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added decoding. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fxied the tests. Signed-off-by: Vahid <vnoroozi@nvidia.com> Signed-off-by: Vahid <vnoroozi@nvidia.com> Signed-off-by: vahidoox <vnoroozi@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
* added initial code. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added the confs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added the confs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * changed name from joint to hybrid. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed format. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed format. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * addressed comments. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * addressed comments. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added docs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added docs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added docs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added docs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * addec CI test. Signed-off-by: Vahid <vnoroozi@nvidia.com> * addec CI test. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bugs in change_vocabs. Signed-off-by: vahidoox <vnoroozi@nvidia.com> * fixed bugs in change_vocabs. Signed-off-by: vahidoox <vnoroozi@nvidia.com> * fixed style. Signed-off-by: vahidoox <vnoroozi@nvidia.com> * fixed style. Signed-off-by: vahidoox <vnoroozi@nvidia.com> * fixed style. Signed-off-by: vahidoox <vnoroozi@nvidia.com> * raise error for aux_ctc. Signed-off-by: Vahid <vnoroozi@nvidia.com> * raise error for aux_ctc. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * raise error for aux_ctc. Signed-off-by: Vahid <vnoroozi@nvidia.com> * raise error for aux_ctc. Signed-off-by: Vahid <vnoroozi@nvidia.com> * updated the streaming names. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added unittests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added unittests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added unittests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed tests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed tests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed tests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed tests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed tests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed tests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added methods. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added decoding. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fxied the tests. Signed-off-by: Vahid <vnoroozi@nvidia.com> Signed-off-by: Vahid <vnoroozi@nvidia.com> Signed-off-by: vahidoox <vnoroozi@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: andrusenkoau <andrusenkoau@gmail.com>
* added initial code. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added the confs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added the confs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * changed name from joint to hybrid. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed format. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed format. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * addressed comments. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * addressed comments. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added docs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added docs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added docs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added docs. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bug. Signed-off-by: Vahid <vnoroozi@nvidia.com> * addec CI test. Signed-off-by: Vahid <vnoroozi@nvidia.com> * addec CI test. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed bugs in change_vocabs. Signed-off-by: vahidoox <vnoroozi@nvidia.com> * fixed bugs in change_vocabs. Signed-off-by: vahidoox <vnoroozi@nvidia.com> * fixed style. Signed-off-by: vahidoox <vnoroozi@nvidia.com> * fixed style. Signed-off-by: vahidoox <vnoroozi@nvidia.com> * fixed style. Signed-off-by: vahidoox <vnoroozi@nvidia.com> * raise error for aux_ctc. Signed-off-by: Vahid <vnoroozi@nvidia.com> * raise error for aux_ctc. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * raise error for aux_ctc. Signed-off-by: Vahid <vnoroozi@nvidia.com> * raise error for aux_ctc. Signed-off-by: Vahid <vnoroozi@nvidia.com> * updated the streaming names. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added unittests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added unittests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added unittests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed tests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed tests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed tests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed tests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fixed tests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed tests. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added methods. Signed-off-by: Vahid <vnoroozi@nvidia.com> * added decoding. Signed-off-by: Vahid <vnoroozi@nvidia.com> * fxied the tests. Signed-off-by: Vahid <vnoroozi@nvidia.com> Signed-off-by: Vahid <vnoroozi@nvidia.com> Signed-off-by: vahidoox <vnoroozi@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
What does this PR do ?
This PR is a refactored version of the following PR created by https://github.com/iankur:
#4854
It adds the Hybrid RNNT/CTC model which has two decoders of CTC and RNNT(Transducer) over the encoder.
It enables to train a single model instead of two which works with both CTC and RNNT decoding. It also speeds up the convergence for CTC models.
Added the following examples:
./examples/asr/asr_hybrid_transducer_ctc/speech_to_text_hybrid_rnnt_ctc_char.py
./examples/asr/asr_hybrid_transducer_ctc/speech_to_text_hybrid_rnnt_ctc_bpe.py
Along with these sample configs for hybrid Conformer model:
./examples/asr/conf/conformer/hybrid_transducer_ctc/conformer_hybrid_transducer_ctc_char.yaml
./examples/asr/conf/conformer/hybrid_transducer_ctc/conformer_hybrid_transducer_ctc_bpe.yaml
Collection:
ASR
Changelog
PR Type: