Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding tutorial for confidence ensembles #6932

Merged
merged 4 commits into from
Jul 12, 2023
Merged

Conversation

Kipok
Copy link
Collaborator

@Kipok Kipok commented Jun 28, 2023

What does this PR do?

There are a couple of bugfixes to the confidence computation and also a tutorial.

It currently misses a few links:

  • to the paper (already submitted to arxiv, waiting to get the link)
  • to the confidence tutorial from Aleksandr, which is not merged yet
  • to the documentation of the confidence ensembles, which I hope to update in the next few days either in this PR or in the follow-up

Collection: ASR

Changelog

  • Add specific line by line info of high level changes in this PR.

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this 

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

  • New Feature
  • Bugfix
  • Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

  • Related to # (issue)

@github-actions github-actions bot added the ASR label Jun 28, 2023
@Kipok Kipok requested review from titu1994, vsl9 and GNroy June 28, 2023 01:08
@Kipok Kipok force-pushed the ensemble-tutorial branch from fa35614 to af85b63 Compare June 29, 2023 16:41
@Kipok
Copy link
Collaborator Author

Kipok commented Jun 29, 2023

Update - fixed all links, except for the confidence tutorial, added docs and also a few simple standalone tests. Bug fixes are removed, since they are merged in another PR

@Kipok Kipok force-pushed the ensemble-tutorial branch from 1f7da2d to c8b07ef Compare June 30, 2023 16:09
@Kipok Kipok force-pushed the ensemble-tutorial branch from c8b07ef to 3bf9abd Compare June 30, 2023 20:55
Copy link
Collaborator

@GNroy GNroy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a preliminary reviewas I was unable to finish the Colab run.
Overall, great PR!
Please confider the following comments.
It would also be great to somehow highlight the difference between the last and penultimate cells, as they long but mostly similar.

tutorials/asr/Confidence_Ensembles.ipynb Show resolved Hide resolved
tutorials/asr/Confidence_Ensembles.ipynb Show resolved Hide resolved
tutorials/asr/Confidence_Ensembles.ipynb Show resolved Hide resolved
tutorials/asr/Confidence_Ensembles.ipynb Outdated Show resolved Hide resolved
tutorials/asr/Confidence_Ensembles.ipynb Outdated Show resolved Hide resolved
tutorials/asr/Confidence_Ensembles.ipynb Outdated Show resolved Hide resolved
tutorials/asr/Confidence_Ensembles.ipynb Show resolved Hide resolved
@Kipok Kipok force-pushed the ensemble-tutorial branch from 7a6841a to 4570c09 Compare July 6, 2023 23:41
@Kipok
Copy link
Collaborator Author

Kipok commented Jul 6, 2023

Thanks for the review, @GNroy! I addressed all of the feedback - also changed the history to remove the image I originally pushed and moved it over to the release files, so that it's accessible in colab and not in the git history.

Sorry about the colab issue - only tested locally because was developing that way. I now verified that it works well in both colab and local mode. Just make sure to restart the kernel after running the first cell to pick up the installed nemo package (there is a comment at the bottom of that cell explaining this)

@Kipok Kipok requested a review from GNroy July 6, 2023 23:44
GNroy
GNroy previously approved these changes Jul 11, 2023
Copy link
Collaborator

@GNroy GNroy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, all my comments were addressed.
Thanks!

titu1994
titu1994 previously approved these changes Jul 11, 2023
Copy link
Collaborator

@titu1994 titu1994 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs and code looks great, minor comments about the tutorial but overall I only skimmed it so please do get a thorough review of it

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems a few images in the tutorial are missing. You can add them to the latest release

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you put the SDP download and install at the very top of the notebook

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which images are missing?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh in the preview the image in the first section of the tutorial (explanation of confidence) was missing. Maybe it will be fixed upob merge c

Kipok added 4 commits July 11, 2023 15:08
Signed-off-by: Igor Gitman <igitman@nvidia.com>
Signed-off-by: Igor Gitman <igitman@nvidia.com>
Signed-off-by: Igor Gitman <igitman@nvidia.com>
Signed-off-by: Igor Gitman <igitman@nvidia.com>
@Kipok Kipok dismissed stale reviews from titu1994 and GNroy via 83fe75e July 11, 2023 23:59
@Kipok Kipok force-pushed the ensemble-tutorial branch from 4570c09 to 83fe75e Compare July 11, 2023 23:59
@Kipok Kipok requested a review from titu1994 July 12, 2023 00:34
Copy link
Collaborator

@titu1994 titu1994 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tutorial is excellently written and documented. Great work !

Maybe it can be improved with more pictures or some other form of explanation in the text heavy sections but it's also ok as is.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh in the preview the image in the first section of the tutorial (explanation of confidence) was missing. Maybe it will be fixed upob merge c

"\n",
"A short answer — you can use any ASR models. E.g., you can combine a number of CTC models, or Transducer models, or even mix-and-match. \n",
"\n",
"A more detailed answer is that hte performance of the confidence ensemble is upper-bounded by the performance of the best model on each of the input examples. Thus you will benefit if some of your models work really well on part of the input compared to other models. This way you will get more gains compared to each separate model, and it will also make correct model identification easier.\n",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imo this section is quite heavy to read in one sitting. Breaking apart the subsection into digestible chunks may be better c

This is fine as well, just a bit heavy.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was not sure how much details to put, so decided to put enough for people to understand a bit more than just examples, but also encourage to skip around the text and go straight to code and return if they want to learn more. Maybe I should make it more explicit - right now I just say

Each cell is mostly self-contained, so feel free to skip around or jump directly to the code part if you want to see usage examples right away.

at the end of the first text cell.

"Let's now talk about the \"model selection block\". First of all — you don't need to know the details to use confidence ensembles, calibration is always automatically performed when you build the model. But if you want to learn more, read on!\n",
"\n",
"First, let's discuss why we need a separate \"model selection block\" to pick the most confident model. If we had an access to the perfect confidence, which would exactly equal to the probability of the model's output being correct, we wouldn't need this block. In this idealized case we can simply take the model with the maximum confidence score. But in practice, models tend to be over- or under-confident, which means that their confidence scores need to be calibrated together to be comparable. E.g., one model might mostly produce scores from 0 to 0.8, while another model tend to produce scores from 0 to 0.5, even though they have the same average accuracy. So we want to multiply the first model's score by 1.25 and the second model's score by 2.0 to put the on the same \"scale\".\n",
"\n",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wall of text, hard to maintain concentration for so much text. Maybe split apart or use some form of visual aid to give a high level understanding then dig into details.

It's fine for now but very heavy section.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it's a bit too much, but that's why I have

First of all — you don't need to know the details to use confidence ensembles, calibration is always automatically performed when you build the model. But if you want to learn more, read on!

in the first paragraph :)

@Kipok Kipok merged commit 77c666f into NVIDIA:main Jul 12, 2023
@Kipok Kipok deleted the ensemble-tutorial branch July 12, 2023 16:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants