Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tutorial on ITN with Thutmose tagger and small fixes #4117

Merged
merged 3 commits into from
May 5, 2022
Merged

Tutorial on ITN with Thutmose tagger and small fixes #4117

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f89e63e
1. Add tutorial. 2. Move a function to fix import in tutorial. 3. Mer…
May 5, 2022
d6a297d
fixes for code review
May 5, 2022
4dcae04
Add tutorial to tutorials.rst
May 5, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions .../nlp/text_normalization_as_tagging/dataset_preparation/prepare_corpora_after_alignment.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,7 @@
from collections import Counter
from typing import Dict, Optional, TextIO, Tuple

from examples.nlp.text_normalization_as_tagging.dataset_preparation.utils import get_src_and_dst_for_alignment

from nemo.collections.nlp.data.text_normalization_as_tagging.utils import get_src_and_dst_for_alignment
from nemo.utils import logging

parser = ArgumentParser(description="Produce data for the ThutmoseTaggerModel")
Expand Down
2 changes: 1 addition & 1 deletion ...es/nlp/text_normalization_as_tagging/dataset_preparation/prepare_corpora_for_alignment.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@
from os.path import isdir, join
from shutil import rmtree

from examples.nlp.text_normalization_as_tagging.dataset_preparation.utils import get_src_and_dst_for_alignment
from nemo.collections.nlp.data.text_normalization_as_tagging.utils import get_src_and_dst_for_alignment
ekmb marked this conversation as resolved.
Show resolved Hide resolved

parser = ArgumentParser(description='Split corpus to subcorpora for giza alignment')
parser.add_argument('--data_dir', type=str, required=True, help='Path to folder with data')
Expand Down
207 changes: 0 additions & 207 deletions examples/nlp/text_normalization_as_tagging/dataset_preparation/utils.py
View file
Open in desktop

This file was deleted.

5 changes: 3 additions & 2 deletions nemo/collections/nlp/data/text_normalization_as_tagging/tagging.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,9 +207,10 @@ def realize_output(self, tags: List[Tag], semiotic_labels: List[str]) -> Tuple[s
output_tokens.append(frag.replace(" ", "").replace("_", ""))
else:
output_tokens.append(frag.strip().replace("_", ""))

output_str = " ".join(output_tokens)
output_str = re.sub(r" +", " ", output_str)
return (
" ".join(output_tokens),
output_str,
" ".join(self.source_tokens),
" ".join(out_tags_without_swap),
output_tags_with_swap_str,
Expand Down
Loading