Skip to content

Commit

Permalink
Multiprocessing fix (NVIDIA#5106) (NVIDIA#5107)
Browse files Browse the repository at this point in the history
Signed-off-by: Matvei Novikov <mattyson.so@gmail.com>

Signed-off-by: Matvei Novikov <mattyson.so@gmail.com>

Signed-off-by: Matvei Novikov <mattyson.so@gmail.com>
Co-authored-by: Matvei Novikov <mattyson.so@gmail.com>
Signed-off-by: Hainan Xu <hainanx@nvidia.com>
  • Loading branch information
2 people authored and Hainan Xu committed Nov 29, 2022
1 parent c3b64e0 commit 75bcc1e
Showing 1 changed file with 3 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ model:
# Number of jobs for tokenization and labels encoding. If 0, then multiprocessing is not used. If null,
# number of jobs is equal to the number of CPU cores.
# WARNING: can cause deadlocks with tokenizers, which use multiprocessing (e.g. SentencePiece)
n_jobs: null
n_jobs: 0

# Path to tarred dataset metadata file. Required if tarred dataset is used. Metadata file is a JSON file which
# contains total number of batches in the dataset, a list of paths to tar files and paths to label vocabularies.
Expand Down Expand Up @@ -143,7 +143,7 @@ model:
# Number of jobs for tokenization and labels encoding. If 0, then multiprocessing is not used. If null,
# number of jobs is equal to the number of CPU cores.
# WARNING: can cause deadlocks with tokenizers, which use multiprocessing (e.g. SentencePiece)
n_jobs: null
n_jobs: 0

# For more details see `train_ds` section.
tar_metadata_file: null
Expand Down Expand Up @@ -174,7 +174,7 @@ model:
# Number of jobs for tokenization and labels encoding. If 0, then multiprocessing is not used. If null,
# number of jobs is equal to the number of CPU cores.
# WARNING: can cause deadlocks with tokenizers, which use multiprocessing (e.g. SentencePiece)
n_jobs: null
n_jobs: 0

# For more details see `train_ds` section.
tar_metadata_file: null
Expand Down

0 comments on commit 75bcc1e

Please sign in to comment.