Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SDE unt lvl comparison #6669

Merged
merged 37 commits into from
Jun 3, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
35f071c
Beta version of unt lvl comparison
Mar 1, 2023
159e367
Beta SDE utt lvl
Apr 12, 2023
aa3daf4
Alpha
Apr 17, 2023
b870540
current v
Jorjeous May 10, 2023
7bb037f
Alpha version, need to fix text diff alignment
Jorjeous May 10, 2023
f70ce94
sde unt final
Jorjeous May 17, 2023
828811f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 17, 2023
1514909
Merge branch 'main' into SDE_unt_lvl
Jorjeous May 17, 2023
b7631c9
update docks
Jorjeous May 17, 2023
54c4612
del unused
Jorjeous May 17, 2023
dcc436f
Merge branch 'main' into SDE_unt_lvl
Jorjeous May 18, 2023
441a9b9
Merge branch 'main' into SDE_unt_lvl
Jorjeous May 19, 2023
9a08997
Merge branch 'main' into SDE_unt_lvl
Jorjeous May 21, 2023
d426e96
Merge branch 'main' into SDE_unt_lvl
Jorjeous May 22, 2023
a533793
Merge branch 'main' into SDE_unt_lvl
Jorjeous May 23, 2023
244710d
del wrong pictire
Jorjeous May 24, 2023
2ba9c17
checkpout from main
Jorjeous May 24, 2023
75baebf
fix scale
Jorjeous May 24, 2023
43249dc
wiped out torchmetrics
Jorjeous May 24, 2023
16872d4
Merge branch 'main' into SDE_unt_lvl
Jorjeous May 24, 2023
bf74f2b
switch to editdist
Jorjeous Jun 2, 2023
05a9c0b
Merge branch 'SDE_unt_lvl' of github.com:NVIDIA/NeMo into SDE_unt_lvl
Jorjeous Jun 2, 2023
d8dd174
Merge branch 'main' into SDE_unt_lvl
Jorjeous Jun 2, 2023
217fcaa
Merge branch 'SDE_unt_lvl' of github.com:NVIDIA/NeMo into SDE_unt_lvl
Jorjeous Jun 2, 2023
7f135a5
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 2, 2023
4416408
rm import
Jorjeous Jun 2, 2023
8c1a1a2
fix
Jorjeous Jun 2, 2023
a89adcd
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 2, 2023
42b96e9
rm imports
Jorjeous Jun 2, 2023
691e625
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 2, 2023
8cf0fc7
rm comments
Jorjeous Jun 2, 2023
085b19b
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 2, 2023
9b75822
Merge branch 'SDE_unt_lvl' of github.com:NVIDIA/NeMo into SDE_unt_lvl
Jorjeous Jun 2, 2023
0d61fc8
Merge branch 'main' into SDE_unt_lvl
vsl9 Jun 2, 2023
8f3acc4
Merge branch 'main' into SDE_unt_lvl
vsl9 Jun 2, 2023
d2d5c6a
Merge branch 'main' into SDE_unt_lvl
vsl9 Jun 2, 2023
9c72957
Merge branch 'main' into SDE_unt_lvl
vsl9 Jun 3, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 30 additions & 1 deletion docs/source/tools/comparison_tool.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Comparison tool for ASR Models
==============================

The Comparison Tool (CT) allows to compare predictions of different ASR models at word accuracy level.
The Comparison Tool (CT) allows to compare predictions of different ASR models at word accuracy and utterance level.

+--------------------------------------------------------------------------------------------------------------------------+
| **Comparison tool features:** |
Expand All @@ -12,6 +12,10 @@ The Comparison Tool (CT) allows to compare predictions of different ASR models a
+--------------------------------------------------------------------------------------------------------------------------+
| visual comparison of predictions of different models |
+--------------------------------------------------------------------------------------------------------------------------+
| visual comparison of utterances by their WER/CER |
+--------------------------------------------------------------------------------------------------------------------------+
| listening selected utterance |
+--------------------------------------------------------------------------------------------------------------------------+

Getting Started
---------------
Expand Down Expand Up @@ -151,3 +155,28 @@ In this case, all points lying above the diagonal have higher accuracy with the
Points marked with circles should be explored first.

Words in the first quarter were well recognized by both models, and conversely, words in the third quarter were poorly recognized by both models.

To compare models at utterance level, pick it at top dropdown field.

At the next field you could choose metric: WER or CER

.. image:: images/scr_10.png
:align: center
:width: 800px
:alt: Switch mode

When an utterance level is selected, it is possible to click on a point on the graph, and the corresponding utterance will be automatically selected.

If audio files are available, there will be an option to listen to the audio recording and view its waveform.

.. image:: images/scr_11.png
:align: center
:width: 800px
:alt: Audio player

In this mode, filtering is still available as well.

**Limitations**

To ensure efficient processing and avoid issues with memory limitations and slow performance, it is recommended to keep the manifests within the limits of 320 hours or around 170,000 utterances.
Exceeding these limits may result in both memory constraints and slower processing.
Binary file added docs/source/tools/images/scr_10.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/source/tools/images/scr_11.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading