Merge r1.9.0 main (NVIDIA#4331)

* update branch Signed-off-by: ericharper <complex451@gmail.com> * update package info Signed-off-by: ericharper <complex451@gmail.com> * cleaned up TN/ ITN doc (NVIDIA#4119) * cleaned up TN/ ITN doc Signed-off-by: Yang Zhang <yangzhang@nvidia.com> * fix typo Signed-off-by: Yang Zhang <yangzhang@nvidia.com> * fix image Signed-off-by: Yang Zhang <yangzhang@nvidia.com> * fix image Signed-off-by: Yang Zhang <yangzhang@nvidia.com> * Draft: Fix restoring from checkpoint for case when `model.common_dataset_parameters.label_vocab_dir` is provided (NVIDIA#4136) * Fix restoring from checkpoint with label vocab dir Signed-off-by: PeganovAnton <peganoff2@mail.ru> * Add tests for various ways to pass label ids to model Signed-off-by: PeganovAnton <peganoff2@mail.ru> * Fix typo Signed-off-by: PeganovAnton <peganoff2@mail.ru> * Fix typo Signed-off-by: PeganovAnton <peganoff2@mail.ru> * Do not create tmp directory Signed-off-by: PeganovAnton <peganoff2@mail.ru> * Fix parameter name Signed-off-by: PeganovAnton <peganoff2@mail.ru> * finish cherry-pick op Signed-off-by: PeganovAnton <peganoff2@mail.ru> * Fix labels errors Signed-off-by: PeganovAnton <peganoff2@mail.ru> * Remove duplicate stage Signed-off-by: PeganovAnton <peganoff2@mail.ru> * Change target branch Signed-off-by: PeganovAnton <peganoff2@mail.ru> * fix doc (NVIDIA#4146) Signed-off-by: Yang Zhang <yangzhang@nvidia.com> * Tacotron2 retrain (NVIDIA#4103) * fix yaml Signed-off-by: treacker <emshabalin@yandex.ru> * Fix for new TTSDataset class Signed-off-by: treacker <emshabalin@yandex.ru> * added wandb logging Signed-off-by: treacker <emshabalin@yandex.ru> * added wandb logging Signed-off-by: treacker <emshabalin@yandex.ru> * fix numpy version Signed-off-by: treacker <emshabalin@yandex.ru> * fix numpy version Signed-off-by: treacker <emshabalin@yandex.ru> * inference fix Signed-off-by: treacker <emshabalin@yandex.ru> * removed old code Signed-off-by: treacker <emshabalin@yandex.ru> * updated parser logic Signed-off-by: treacker <emshabalin@yandex.ru> * reverted version update Signed-off-by: treacker <emshabalin@yandex.ru> * refactored parser logic Signed-off-by: treacker <emshabalin@yandex.ru> * Updated Jenkinsfile Signed-off-by: treacker <emshabalin@yandex.ru> * Refactored tutorial for Tacotron2 Signed-off-by: treacker <emshabalin@yandex.ru> * Made backward compatibility Signed-off-by: treacker <emshabalin@yandex.ru> * Made backward compatibility Signed-off-by: treacker <emshabalin@yandex.ru> * Update Jenkinsfile Signed-off-by: treacker <emshabalin@yandex.ru> * Update tacotron.yaml Signed-off-by: treacker <emshabalin@yandex.ru> * Refactoring Signed-off-by: treacker <emshabalin@yandex.ru> * cleaned up TN/ ITN doc (NVIDIA#4119) * cleaned up TN/ ITN doc Signed-off-by: Yang Zhang <yangzhang@nvidia.com> * fix typo Signed-off-by: Yang Zhang <yangzhang@nvidia.com> * fix image Signed-off-by: Yang Zhang <yangzhang@nvidia.com> * fix image Signed-off-by: Yang Zhang <yangzhang@nvidia.com> Signed-off-by: treacker <emshabalin@yandex.ru> * Check implicit grad acc in GLUE dataset building (NVIDIA#4123) * Check implicit grad acc in GLUE dataset building Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix jenkins test for GLUE/XNLI Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Signed-off-by: treacker <emshabalin@yandex.ru> * Refactoring Signed-off-by: treacker <emshabalin@yandex.ru> * Refactoring Signed-off-by: treacker <emshabalin@yandex.ru> * Fixed jenkins Signed-off-by: treacker <emshabalin@yandex.ru> * Refactoring Signed-off-by: treacker <emshabalin@yandex.ru> * Refactoring Signed-off-by: treacker <emshabalin@yandex.ru> * Refactoring Signed-off-by: treacker <emshabalin@yandex.ru> Co-authored-by: Yang Zhang <yzhang123@users.noreply.github.com> Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca> * Multiprocess improvements (NVIDIA#4127) * initial commit Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com> * start fix Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com> * improve multiprocessing speed while creating speaker dataset Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com> * updated scp to filelist Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com> * notebooks' link, typo and import fix (NVIDIA#4158) * redo missing pr 4007 Signed-off-by: fayejf <fayejf07@gmail.com> * remove extremely unreliable links Signed-off-by: fayejf <fayejf07@gmail.com> * update speaker docs (NVIDIA#4164) * update speaker docs Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com> * chunks -> segments Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com> * Khz -> kHz Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com> * small fix (NVIDIA#4180) Signed-off-by: fayejf <fayejf07@gmail.com> * fix the server key value problem (NVIDIA#4196) Signed-off-by: Yi Dong <yidong@nvidia.com> * Fix/punctuation/trainer required for setting test data (NVIDIA#4199) * Draft of fix Signed-off-by: PeganovAnton <peganoff2@mail.ru> * Add warnings and replace globa_step with current_epoch Signed-off-by: PeganovAnton <peganoff2@mail.ru> * Small improvements to warnings Signed-off-by: PeganovAnton <peganoff2@mail.ru> * Error and warning messages improvements Signed-off-by: PeganovAnton <peganoff2@mail.ru> * Replace self.trainer with self._trainer Signed-off-by: PeganovAnton <peganoff2@mail.ru> * Update ContextNet version (NVIDIA#4207) Signed-off-by: smajumdar <smajumdar@nvidia.com> * fix bugs for dialogue tutorial (NVIDIA#4211) Signed-off-by: Zhilin Wang <wangzhilin12061996@hotmail.com> * Dialogue tutorial fix (NVIDIA#4214) * fix bugs for dialogue tutorial Signed-off-by: Zhilin Wang <wangzhilin12061996@hotmail.com> * update path for convert_datasets.py due to conflict PR Signed-off-by: Zhilin Wang <wangzhilin12061996@hotmail.com> * Add docs for Thutmose Tagger (NVIDIA#4173) * Add docs for Thutmose Tagger Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com> * add level in docs Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com> * delete folder to avoid error with running when folder exists from previous run Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com> Co-authored-by: Alexandra Antonova <aleksandraa@nvidia.com> Co-authored-by: ekmb <ebakhturina@nvidia.com> * Dialogue tutorial fix (NVIDIA#4218) * fix bugs for dialogue tutorial Signed-off-by: Zhilin Wang <wangzhilin12061996@hotmail.com> * update path for convert_datasets.py due to conflict PR Signed-off-by: Zhilin Wang <wangzhilin12061996@hotmail.com> * restore previously deleted files Signed-off-by: Zhilin Wang <wangzhilin12061996@hotmail.com> * style fix Signed-off-by: Zhilin Wang <wangzhilin12061996@hotmail.com> * Dialogue tutorial fix (NVIDIA#4221) * fix bugs for dialogue tutorial Signed-off-by: Zhilin Wang <wangzhilin12061996@hotmail.com> * update path for convert_datasets.py due to conflict PR Signed-off-by: Zhilin Wang <wangzhilin12061996@hotmail.com> * restore previously deleted files Signed-off-by: Zhilin Wang <wangzhilin12061996@hotmail.com> * style fix Signed-off-by: Zhilin Wang <wangzhilin12061996@hotmail.com> * update tutorial Signed-off-by: Zhilin Wang <wangzhilin12061996@hotmail.com> * fix syntax error in ipynb-file (NVIDIA#4228) Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com> Co-authored-by: Alexandra Antonova <aleksandraa@nvidia.com> * fix json serialize (NVIDIA#4235) Signed-off-by: Yi Dong <yidong@nvidia.com> * Prompt Learning Typo Fixes (NVIDIA#4238) * Prompt tuning notebook typo fixes Signed-off-by: Virginia Adams <vadams@nvidia.com> * Update tutorials.rst * Update prompt_learning.rst * Update prompt_learning.rst * fixing bug 3642622 (NVIDIA#4250) * fixing bug 3642622 Signed-off-by: Ghasem Pasandi <gpasandi@nvidia.com> * fixing bug 3642622 Signed-off-by: Ghasem Pasandi <gpasandi@nvidia.com> Co-authored-by: Ghasem Pasandi <gpasandi@nvidia.com> * fix broken link in the tutorial (NVIDIA#4257) Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com> Co-authored-by: Alexandra Antonova <aleksandraa@nvidia.com> * Typo fix, branch change, better download messagae (NVIDIA#4262) Signed-off-by: Virginia Adams <vadams@nvidia.com> * Raise error if bicleaner is not installed in NMT Data preprocesing notebook (NVIDIA#4264) * Raise error if bicleaner is not installed Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Clear cells Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Fix missing validation dataset, whitelist certain keywords for datasets (NVIDIA#4269) * Fix missing validation dataset, whitelist certain keywords for datasets Signed-off-by: smajumdar <smajumdar@nvidia.com> * Fix missing validation dataset, whitelist certain keywords for datasets Signed-off-by: smajumdar <smajumdar@nvidia.com> * Update asr configs with num_workers and pin_memory (NVIDIA#4270) Signed-off-by: smajumdar <smajumdar@nvidia.com> * Fix epoch end (NVIDIA#4265) Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Eric Harper <complex451@gmail.com> * Set Save on train end to false (NVIDIA#4274) * Set Save on train end to false Signed-off-by: Virginia Adams <vadams@nvidia.com> * Update prompt_learning.rst * Update prompt_learning.rst * Update YAML (NVIDIA#4261) Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca> * Updated config to fix CI test OOM error (NVIDIA#4279) * Updated config to fix CI test issue Signed-off-by: Virginia Adams <vadams@nvidia.com> * Increased num workers Signed-off-by: Virginia Adams <vadams@nvidia.com> * verbose k2 install, skip if failed (NVIDIA#4289) Signed-off-by: Aleksandr Laptev <alaptev@nvidia.com> Co-authored-by: Aleksandr Laptev <alaptev@nvidia.com> * Changed total virtual prompt tokens (NVIDIA#4295) * Changed total virtual prompt tokens Signed-off-by: Virginia Adams <vadams@nvidia.com> * put number of workers back Signed-off-by: Virginia Adams <vadams@nvidia.com> * upper bound lightning Signed-off-by: ericharper <complex451@gmail.com> * update branch Signed-off-by: ericharper <complex451@gmail.com> * update config Signed-off-by: ericharper <complex451@gmail.com> * remove duplicate test Signed-off-by: ericharper <complex451@gmail.com> * fix tn test cases Signed-off-by: ericharper <complex451@gmail.com> * add another safe.directory Signed-off-by: ericharper <complex451@gmail.com> * typo Signed-off-by: ericharper <complex451@gmail.com> Co-authored-by: Yang Zhang <yzhang123@users.noreply.github.com> Co-authored-by: PeganovAnton <peganoff2@mail.ru> Co-authored-by: treacker <36159472+treacker@users.noreply.github.com> Co-authored-by: Sandeep Subramanian <sandeep.subramanian.1@umontreal.ca> Co-authored-by: Nithin Rao <nithinrao.koluguri@gmail.com> Co-authored-by: fayejf <36722593+fayejf@users.noreply.github.com> Co-authored-by: Yi Dong <43824965+yidong72@users.noreply.github.com> Co-authored-by: Somshubra Majumdar <titu1994@gmail.com> Co-authored-by: Zhilin Wang <wangzhilin12061996@hotmail.com> Co-authored-by: bene-ges <61418381+bene-ges@users.noreply.github.com> Co-authored-by: Alexandra Antonova <aleksandraa@nvidia.com> Co-authored-by: ekmb <ebakhturina@nvidia.com> Co-authored-by: Virginia Adams <78445382+vadam5@users.noreply.github.com> Co-authored-by: Ghasem <35242805+pasandi20@users.noreply.github.com> Co-authored-by: Ghasem Pasandi <gpasandi@nvidia.com> Co-authored-by: Aleksandr Laptev <laptevsasha12@gmail.com> Co-authored-by: Aleksandr Laptev <alaptev@nvidia.com> Signed-off-by: Hainan Xu <hainanx@nvidia.com>
hainan-xv · Nov 29, 2022 · 1c5c2ae · 1c5c2ae
1 parent 7079c8a
commit 1c5c2ae
Show file tree

Hide file tree

Showing 67 changed files with 2,223 additions and 1,777 deletions.
diff --git a/Dockerfile b/Dockerfile
@@ -55,7 +55,7 @@ RUN for f in $(ls requirements*.txt); do pip install --disable-pip-version-check
 
 # install k2, skip if installation fails
 COPY scripts /tmp/nemo/scripts/
-RUN /bin/bash /tmp/nemo/scripts/speech_recognition/k2/setup.sh; exit 0
+RUN /bin/bash /tmp/nemo/scripts/speech_recognition/k2/setup.sh || exit 0
 
 # copy nemo source into a scratch image
 FROM scratch as nemo-src

diff --git a/Jenkinsfile b/Jenkinsfile
@@ -15,6 +15,7 @@ pipeline {
     stage('Add git safe directory'){
       steps{
         sh 'git config --global --add safe.directory /var/lib/jenkins/workspace/NeMo_$GIT_BRANCH'
+        sh 'git config --global --add safe.directory /raid/JenkinsWorkDir/workspace/NeMo_$GIT_BRANCH'
       }
     }
 
@@ -1590,22 +1591,20 @@ pipeline {
       }
       failFast true
       stages {
-        stage('Punctuation & Capitalization, Using model.common_dataset_parameters.label_vocab_dir') {
+        stage('Punctuation & Capitalization, Using model.common_datasest_parameters.label_vocab_dir') {
           steps {
             sh 'cd examples/nlp/token_classification && \
-            output_dir="$(mktemp -d -p "$(pwd)")" && \
-            data_dir="$(mktemp -d -p "$(pwd)")" && \
-            cp /home/TestData/nlp/token_classification_punctuation/*.txt "${data_dir}"/ && \
-            label_vocab_dir="$(mktemp -d -p "$(pwd)")" && \
+            label_vocab_dir=label_vocab_dir && \
+            mkdir -p ${label_vocab_dir} && \
             punct_label_vocab="${label_vocab_dir}/punct_label_vocab.csv" && \
             capit_label_vocab="${label_vocab_dir}/capit_label_vocab.csv" && \
             printf "O\n,\n.\n?\n" > "${punct_label_vocab}" && \
             printf "O\nU\n" > "${capit_label_vocab}" && \
-            python punctuation_capitalization_train_evaluate.py \
+            CUDA_LAUNCH_BLOCKING=1 python punctuation_capitalization_train_evaluate.py \
               model.train_ds.use_tarred_dataset=false \
-              model.train_ds.ds_item="${data_dir}" \
-              model.validation_ds.ds_item="${data_dir}" \
-              model.test_ds.ds_item="${data_dir}" \
+              model.train_ds.ds_item=/home/TestData/nlp/token_classification_punctuation \
+              model.validation_ds.ds_item=/home/TestData/nlp/token_classification_punctuation \
+              model.test_ds.ds_item=/home/TestData/nlp/token_classification_punctuation \
               model.language_model.pretrained_model_name=distilbert-base-uncased \
               model.common_dataset_parameters.label_vocab_dir="${label_vocab_dir}" \
               model.class_labels.punct_labels_file="$(basename "${punct_label_vocab}")" \
@@ -1616,69 +1615,68 @@ pipeline {
               trainer.devices=[0,1] \
               trainer.strategy=ddp \
               trainer.max_epochs=1 \
-              +exp_manager.explicit_log_dir="${output_dir}" \
+              +exp_manager.explicit_log_dir=/home/TestData/nlp/token_classification_punctuation/output \
               +do_testing=false && \
-            python punctuation_capitalization_train_evaluate.py \
+            CUDA_LAUNCH_BLOCKING=1 python punctuation_capitalization_train_evaluate.py \
               +do_training=false \
               +do_testing=true \
               ~model.train_ds \
               ~model.validation_ds \
-              model.test_ds.ds_item="${data_dir}" \
-              pretrained_model="${output_dir}/checkpoints/Punctuation_and_Capitalization.nemo" \
+              model.test_ds.ds_item=/home/TestData/nlp/token_classification_punctuation \
+              pretrained_model=/home/TestData/nlp/token_classification_punctuation/output/checkpoints/Punctuation_and_Capitalization.nemo \
               +model.train_ds.use_cache=false \
               +model.validation_ds.use_cache=false \
               +model.test_ds.use_cache=false \
               trainer.devices=[0,1] \
               trainer.strategy=ddp \
               trainer.max_epochs=1 \
               exp_manager=null && \
-            rm -rf "${label_vocab_dir}" "${data_dir}" "${output_dir}"'
+            rm -r "${label_vocab_dir}" && \
+            rm -rf /home/TestData/nlp/token_classification_punctuation/output/*'
           }
         }
-        stage('Punctuation & Capitalization, Using model.common_dataset_parameters.{punct,capit}_label_ids') {
+        stage('Punctuation & Capitalization, Using model.common_datasest_parameters.{punct,capit}_label_ids') {
           steps {
             sh 'cd examples/nlp/token_classification && \
-            output_dir="$(mktemp -d -p "$(pwd)")" && \
-            data_dir="$(mktemp -d -p "$(pwd)")" && \
-            cp /home/TestData/nlp/token_classification_punctuation/*.txt "${data_dir}"/ && \
-            conf_path="$(mktemp -d -p "$(pwd)")" && \
+            conf_path=/home/TestData/nlp/token_classification_punctuation && \
             conf_name=punctuation_capitalization_config_with_ids && \
             cp conf/punctuation_capitalization_config.yaml "${conf_path}/${conf_name}.yaml" && \
             sed -i $\'s/punct_label_ids: null/punct_label_ids: {O: 0, \\\',\\\': 1, .: 2, \\\'?\\\': 3}/\' \
               "${conf_path}/${conf_name}.yaml" && \
             sed -i $\'s/capit_label_ids: null/capit_label_ids: {O: 0, U: 1}/\' \
               "${conf_path}/${conf_name}.yaml" && \
-            python punctuation_capitalization_train_evaluate.py \
+            CUDA_LAUNCH_BLOCKING=1 python punctuation_capitalization_train_evaluate.py \
               --config-path "${conf_path}" \
               --config-name "${conf_name}" \
               model.train_ds.use_tarred_dataset=false \
-              model.train_ds.ds_item="${data_dir}" \
-              model.validation_ds.ds_item="${data_dir}" \
-              model.test_ds.ds_item="${data_dir}" \
+              model.train_ds.ds_item=/home/TestData/nlp/token_classification_punctuation \
+              model.validation_ds.ds_item=/home/TestData/nlp/token_classification_punctuation \
+              model.test_ds.ds_item=/home/TestData/nlp/token_classification_punctuation \
               model.language_model.pretrained_model_name=distilbert-base-uncased \
               +model.train_ds.use_cache=false \
               +model.validation_ds.use_cache=false \
               +model.test_ds.use_cache=false \
               trainer.devices=[0,1] \
               trainer.strategy=ddp \
               trainer.max_epochs=1 \
-              +exp_manager.explicit_log_dir="${output_dir}" \
+              +exp_manager.explicit_log_dir=/home/TestData/nlp/token_classification_punctuation/output \
               +do_testing=false && \
-            python punctuation_capitalization_train_evaluate.py \
+            CUDA_LAUNCH_BLOCKING=1 python punctuation_capitalization_train_evaluate.py \
               +do_training=false \
               +do_testing=true \
               ~model.train_ds \
               ~model.validation_ds \
-              model.test_ds.ds_item="${data_dir}" \
-              pretrained_model="${output_dir}/checkpoints/Punctuation_and_Capitalization.nemo" \
+              model.test_ds.ds_item=/home/TestData/nlp/token_classification_punctuation \
+              pretrained_model=/home/TestData/nlp/token_classification_punctuation/output/checkpoints/Punctuation_and_Capitalization.nemo \
               +model.train_ds.use_cache=false \
               +model.validation_ds.use_cache=false \
               +model.test_ds.use_cache=false \
               trainer.devices=[0,1] \
               trainer.strategy=ddp \
               trainer.max_epochs=1 \
               exp_manager=null && \
-            rm -rf "${output_dir}" "${data_dir}" "${conf_path}"'
+            rm -rf /home/TestData/nlp/token_classification_punctuation/output/* && \
+            rm "${conf_path}/${conf_name}.yaml"'
           }
         }
       }

diff --git a/docs/source/nlp/prompt_learning.rst b/docs/source/nlp/prompt_learning.rst
@@ -10,6 +10,8 @@ Instead of selecting discrete text prompts in a manual or automated fashion, pro
 
 Our continuous learning capability for combined p-tuning and prompt tuning with GPT style models is a NeMo specific extension of the author's original work.
 
+Please also checkout our `prompt learning tutorial notebook. <https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Multitask_Prompt_and_PTuning.ipynb>`_
+
 
 Terminology
 ^^^^^^^^^^
@@ -89,14 +91,17 @@ the input will be translated into ``VVV Hypothesis: And he said, Mama, I'm home.
         "prompt_template": "<|VIRTUAL_PROMPT_0|> {sentence} sentiment: {label}",
         "total_virtual_tokens": 10,
         "virtual_token_splits": [10],
-        "truncate_field": "sentence"
+        "truncate_field": "sentence",
+        "answer_only_loss": False,
     },
     {
         "taskname": "intent_and_slot",
         "prompt_template": "<|VIRTUAL_PROMPT_0|> Predict intent and slot <|VIRTUAL_PROMPT_1|> :\n{utterance}{label}",
         "total_virtual_tokens": 10,
         "virtual_token_splits": [7, 3],
-        "truncate_field": None
+        "truncate_field": None,
+        "answer_only_loss": True,
+        "answer_field": "label"
     }
   ]
 
@@ -198,9 +203,9 @@ Setting New Tasks
 
 After you p-tune or prompt-tune your model, you can always go back and p-tune or prompt-tune your model on more tasks without over writing the virtual prompts who've trained already. You can also use a different number of ``total_virtual_tokens`` between each training session as long as tasks ptuned or prompt tuned at the same time have the same number of ``total_virtual_tokens``. For this reason, when you ptune on a new task, you need to tell your model which of your tasks are new and which ones already exist (and thus you don't want to tune them). You do this by setting the ``new_tasks`` and ``existing_tasks`` values in the config file.
 
-Example Multi-Task Prompt Tuning Command
+Example Multi-Task Prompt Tuning Config and Command
 ^^^^^^^^^^
-First define a config called ``multitask-prompt-learning.yaml`` that looks like:
+First define a config called ``multitask-prompt-learning.yaml`` demonstrated below. **In the** ``exp_manager`` **portion of the config,** ``save_on_train_end`` **should be set to** ``False`` **to avoid unnecessarily saving the incorrect model weights.** This is already done in the example `megatron_gpt_prompt_learning_config.yaml config <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/conf/megatron_gpt_prompt_learning_config.yaml>`_ that you should use as your starting point. The correct prompt learning model will be saved at the ``model.nemo_path`` you set. 
 
 .. code::
   
@@ -229,12 +234,15 @@ First define a config called ``multitask-prompt-learning.yaml`` that looks like:
       total_virtual_tokens: 100 
       virtual_token_splits: [100] 
       truncate_field: null
+      answer_only_loss: False
 
     - taskname: "intent_and_slot"
       prompt_template: "<|VIRTUAL_PROMPT_0|> Predict intent and slot <|VIRTUAL_PROMPT_1|> :\n{utterance}{label}" 
       total_virtual_tokens: 100 
       virtual_token_splits: [80, 20]
       truncate_field: null
+      answer_only_loss: True
+      answer_field: "label"
 
     prompt_tuning: 
       new_prompt_init_methods: ["text", "text"] 
@@ -259,7 +267,7 @@ Then run the command
   python megatron_gpt_prompt_learning.py --config-name=multitask-prompt-learning.yaml
          
 
-Example Multi-Task P-Tuning Command After Prompt-Tuning
+Example Multi-Task P-Tuning Config and Command After Prompt-Tuning
 ^^^^^^^^^^
 Update ``multitask-prompt-learning.yaml`` from the example above with p-tuning parameters for the new task. Be sure to update ``model.existing_tasks`` with the tasknames from previous prompt learning runs and to use the ``.nemo`` file saved at the end of your last prompt learning session. Values different from the config above have stars commented next to them. 
 
@@ -284,28 +292,31 @@ In this example, the SQuAD task includes the question context as part of the pro
   restore_path: multitask_prompt_tuning.nemo # ***
   language_model_path: models/megatron_125M_gpt.nemo
   existing_tasks: ["sentiment", "intent_and_slot"] # ***
-  new_tasks: ["sentiment", "intent_and_slot"] 
+  new_tasks: ["squad"] 
 
   task_templates: 
   - taskname: "sentiment" 
     prompt_template: "<|VIRTUAL_PROMPT_0|> {sentence} sentiment: {label}" 
     total_virtual_tokens: 100 
     virtual_token_splits: [100] 
     truncate_field: null
+    answer_only_loss: False
 
   - taskname: "intent_and_slot"
     prompt_template: "<|VIRTUAL_PROMPT_0|> Predict intent and slot <|VIRTUAL_PROMPT_1|> :\n{utterance}{label}" 
     total_virtual_tokens: 100 
     virtual_token_splits: [80, 20]
     truncate_field: null
+    answer_only_loss: True
+    answer_field: "label"
 
   - taskname: "squad" # ***
-    prompt_template: "<|VIRTUAL_PROMPT_0|> Answer the question from the context <|VIRTUAL_PROMPT_1|> {question} <|VIRTUAL_PROMPT_2|> {context} <|VIRTUAL_PROMPT_3|>  Answer: {answer}" # *** 
-    total_virtual_tokens: 16 # ***
-    virtual_token_splits: [4, 4, 4, 4] # ***
+    prompt_template: "<|VIRTUAL_PROMPT_0|> Answer the question from the context {question} {context} Answer: {answer}" # *** 
+    total_virtual_tokens: 9 # ***
+    virtual_token_splits: [9] # ***
     truncate_field: context # ***
     answer_only_loss: True # ***
-    answer_field: 'answer # ***
+    answer_field: "answer" # ***
 
   p_tuning: # ***
       dropout: 0.0 # ***

diff --git a/docs/source/nlp/text_normalization/intro.rst b/docs/source/nlp/text_normalization/intro.rst
@@ -1,6 +1,8 @@
 (Inverse) Text Normalization
 ============================
 
+NeMo supports Text Normalization (TN) and Inverse Text Normalization (ITN) tasks via rule-based `nemo_text_processing` python package and Neural-based TN/ITN models.
+
 Rule-based (WFST) TN/ITN:
 
 .. toctree::
@@ -9,11 +11,10 @@ Rule-based (WFST) TN/ITN:
    wfst/intro
 
 
-Neural TN/ITN:
+Neural-based TN/ITN:
 
 .. toctree::
    :maxdepth: 1
 
-   nn_text_normalization
-
+   neural_models
 
diff --git a/docs/source/nlp/text_normalization/neural_models.rst b/docs/source/nlp/text_normalization/neural_models.rst
@@ -0,0 +1,23 @@
+.. _neural_models:
+
+Neural Models for (Inverse) Text Normalization
+==============================================
+
+NeMo provides two types of neural models:
+
+
+Duplex T5-based TN/ITN:
+
+.. toctree::
+   :maxdepth: 1
+
+   nn_text_normalization
+
+
+Single-pass Tagger-based ITN:
+
+.. toctree::
+   :maxdepth: 1
+
+   text_normalization_as_tagging
+