[ds launcher] un-hijack PYTHONPATH

[stas00]

I'm not sure why PYTHONPATH is set to python executable here as it expects dirs and not files:
https://github.com/huggingface/accelerate/pull/514/files#r987305929

but regardless accelerate shouldn't hijack PYTHONPATH

This PR extends the PYTHONPATH instead if it was preset.

Thank you

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[sgugger]

Thanks for your PR! LGTM but let's wait to see what @pacman100 thinks.

[pacman100]

Thank you Stas for the fix 🤗! As you mentioned, it shouldn't be a path to a file. Instead, it should be the pwd of the launched-from-dir.

[pacman100]

Oops, Thanks a lot for pointing this out! Post the above PR, a sample run is shown below:

>>> export PYTHONPATH="sample_dir/fancy_library"
...

>>> accelerate launch --config_file ds_mn_hf.yaml accelerate/examples/complete_nlp_example.py --mixed_precision "fp16"
[2022-10-06 12:00:16,739] [INFO] [runner.py:504:main] cmd = pdsh -S -f 1024 -w localhost,sourab_huggingface_co@sourab-vm-image-1 
export PYTHONPATH=sample_dir/fancy_library:/home/sourab_huggingface_co;

...

export MIXED_PRECISION=fp16; export USE_DEEPSPEED=true; export DEEPSPEED_ZERO_STAGE=2; export GRADIENT_ACCUMULATION_STEPS=1; export GRADIENT_CLIPPING=1.0; export DEEPSPEED_OFFLOAD_OPTIMIZER_DEVICE=none; export DEEPSPEED_OFFLOAD_PARAM_DEVICE=none; export DEEPSPEED_ZERO3_INIT=false; export DEEPSPEED_ZERO3_SAVE_16BIT_MODEL=none;  cd /home/sourab_huggingface_co; /opt/conda/envs/ml/bin/python3.9 -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMF0sICJzb3VyYWJfaHVnZ2luZ2ZhY2VfY29Ac291cmFiLXZtLWltYWdlLTEiOiBbMF19 --node_rank=%n --master_addr=xx.xxx.x.x --master_port=29500 --no_local_rank accelerate/examples/complete_nlp_example.py --mixed_precision 'fp16'
localhost: [2022-10-06 12:00:22,720] [INFO] [launch.py:136:main] WORLD INFO DICT: {'localhost': [0], 'sourab_huggingface_co@sourab-vm-image-1': [0]}
localhost: [2022-10-06 12:00:22,720] [INFO] [launch.py:142:main] nnodes=2, num_local_procs=1, node_rank=0
localhost: [2022-10-06 12:00:22,720] [INFO] [launch.py:155:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0], 'sourab_huggingface_co@sourab-vm-image-1': [1]})
localhost: [2022-10-06 12:00:22,720] [INFO] [launch.py:156:main] dist_world_size=2
localhost: [2022-10-06 12:00:22,720] [INFO] [launch.py:158:main] Setting CUDA_VISIBLE_DEVICES=0
sourab-vm-image-1: [2022-10-06 12:00:23,404] [INFO] [launch.py:136:main] WORLD INFO DICT: {'localhost': [0], 'sourab_huggingface_co@sourab-vm-image-1': [0]}
sourab-vm-image-1: [2022-10-06 12:00:23,404] [INFO] [launch.py:142:main] nnodes=2, num_local_procs=1, node_rank=1
sourab-vm-image-1: [2022-10-06 12:00:23,404] [INFO] [launch.py:155:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0], 'sourab_huggingface_co@sourab-vm-image-1': [1]})
sourab-vm-image-1: [2022-10-06 12:00:23,404] [INFO] [launch.py:156:main] dist_world_size=2
sourab-vm-image-1: [2022-10-06 12:00:23,404] [INFO] [launch.py:158:main] Setting CUDA_VISIBLE_DEVICES=0
localhost: [2022-10-06 12:00:27,932] [INFO] [comm.py:633:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl

...

localhost: [2022-10-06 12:01:10,571] [INFO] [config.py:976:print_user_config]   json = {
localhost:     "train_batch_size": 32, 
localhost:     "train_micro_batch_size_per_gpu": 16, 
localhost:     "gradient_accumulation_steps": 1, 
localhost:     "zero_optimization": {
localhost:         "stage": 2, 
localhost:         "offload_optimizer": {
localhost:             "device": "none"
localhost:         }, 
localhost:         "offload_param": {
localhost:             "device": "none"
localhost:         }, 
localhost:         "stage3_gather_16bit_weights_on_model_save": false
localhost:     }, 
localhost:     "gradient_clipping": 1.0, 
localhost:     "steps_per_print": inf, 
localhost:     "fp16": {
localhost:         "enabled": true, 
localhost:         "auto_cast": true
localhost:     }, 
localhost:     "zero_allow_untested_optimizer": true
localhost: }
localhost: Using /home/sourab_huggingface_co/.cache/torch_extensions/py39_cu113 as PyTorch extensions root...

...

localhost: epoch 0: {'accuracy': 0.7303921568627451, 'f1': 0.8318042813455658}
sourab-vm-image-1: epoch 0: {'accuracy': 0.7303921568627451, 'f1': 0.8318042813455658}
localhost: epoch 1: {'accuracy': 0.8308823529411765, 'f1': 0.880415944540728}
sourab-vm-image-1: epoch 1: {'accuracy': 0.8308823529411765, 'f1': 0.880415944540728}
localhost: epoch 2: {'accuracy': 0.8333333333333334, 'f1': 0.8811188811188811}
sourab-vm-image-1: epoch 2: {'accuracy': 0.8333333333333334, 'f1': 0.8811188811188811}

[stas00]

@pacman100, please feel free to merge when you feel it's ready. Thanks.