Fix data preprocessing

[nsakkine]

Potential bug in tools/prerpocess_data.py.

Bookcorpus preprocessing fails under certain conditions when using rocm/pytorch-training:v25.2 image and installing Megatron-LM with pip install . from the latest ROCm/Megatron-LM rocm_dev branch. With

/workspace/Megatron-LM# ls bookcorpus
bookcorpus_megatron.json  tokenizer.model

and running

/workspace/Megatron-LM# python3 tools/preprocess_data.py --input bookcorpus/bookcorpus_megatron.json  --tokenizer-type GPTSentencePieceTokenizer --tokenizer-model bookcorpus/tokenizer.model --output-prefix bookcorpus/bookcorpus --workers `nproc` --split-sentences --partitions 1

only executes sentence splitting while doing the same with --partitions 2 runs both sentence splitting and tokenization. Furthermore, running the above with --partitions 1 twice should also be a work-around since then split-sentence data is available in the second go.

Proposed solution is to remove 2 lines in tools/preprocess_data.py.

[nsakkine]

@aivanni @gurpreet-dhami @wenchenvincent request to confirm if this is a bug