Can't get Deepspeed to work on Ubuntu 22.04

[AlpsAficionado]

Hello esteemed Deepspeed community.

I've spent several hours bashing my head against getting deepspeed to function properly on my system. I run oobabooga/text-generation-webui inside an Ubuntu 22.04 VM on my server (Nvidia Quadro RTX 8000 with 48 VRAM; 128GB system RAM; the VM is given total control of the 8000 via PCI passthrough/iommu, and has 96GB of system RAM allocated to it).

After hours of frustration, I was unable to get deepspeed to function with oobabooga on Ubuntu 22.04.

I was able to get it to run on Ubuntu 23.04; however, 22.04 is the current long-term supported version (and it still took me significant manual intervention to get it going in 23.04).

A rundown of my issues with 22.04:

1. Error: ModuleNotFoundError: No module named 'deepspeed'
   Solution: pip install deepspeed

2. Error: ModuleNotFoundError: No module named 'mpi4py'
   Solution: sudo apt install libopenmpi-dev ; pip install mpi4py

3. Error: 'pip install mpi4py' won't work; it crashes like so:

/home/user/oobabooga_linux/installer_files/env/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to \opal_class_finalize'
/home/user/oobabooga_linux/installer_files/env/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to \mca_base_var_group_get_count'
/home/user/oobabooga_linux/installer_files/env/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to \opal_datatype_resize'
/home/user/oobabooga_linux/installer_files/env/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to \opal_hash_table_set_value_uint64'
/home/user/oobabooga_linux/installer_files/env/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to \MPIR_being_debugged'
/home/user/oobabooga_linux/installer_files/env/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to \opal_list_t_class'
collect2: error: ld returned 1 exit status
failure.
removing: _configtest.c _configtest.o
error: Cannot link MPI programs. Check your configuration!!!

Solution: env LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/ pip install --no-cache-dir mpi4py

4. Error:

module = importlib.util.module_from_spec(spec)
File "<frozen importlib._bootstrap>", line 571, in module_from_spec
File "<frozen importlib._bootstrap_external>", line 1176, in create_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
ImportError: /home/user/oobabooga_linux/installer_files/env/bin/../lib/libstdc++.so.6: version \GLIBCXX_3.4.30' not found (required by /home/user/.cache/torch_extensions/py310_cu117/async_io/async_io.so)

Solution: conda install -c conda-forge gcc=12.1.0 (rebuilds/reinstalls a whole bunch of crap, see below:

Downloading and Extracting Packages
_libgcc_mutex-0.1 | 3 KB | ###################################################################################### | 100%
ca-certificates-2023 | 145 KB | ###################################################################################### | 100%
gcc_impl_linux-64-12 | 56.2 MB | ####################################################################################2 | 98%
libgomp-12.2.0 | 455 KB | ###################################################################################### | 100%
kernel-headers_linux | 707 KB | ###################################################################################### | 100%
libgcc-ng-12.2.0 | 931 KB | ###################################################################################### | 100%
libgcc-devel_linux-6 | 3.3 MB | ###################################################################################### | 100%
openssl-1.1.1t | 1.9 MB | ###################################################################################### | 100%
requests-2.29.0 | 55 KB | ###################################################################################### | 100%
libsanitizer-12.1.0 | 6.2 MB | ###################################################################################### | 100%
libstdcxx-ng-12.2.0 | 4.3 MB | ###################################################################################### | 100%
sysroot_linux-64-2.1 | 31.4 MB | ####################################################################################6 | 98%
gcc-12.1.0 | 24 KB | ###################################################################################### | 100%
certifi-2023.5.7 | 149 KB | ###################################################################################### | 100%
_openmp_mutex-4.5 | 23 KB | ###################################################################################### | 100%
binutils_impl_linux- | 5.2 MB | ###################################################################################### | 100%

5. Error: RuntimeError: Unable to JIT load the async_io op due to it not being compatible due to hardware/software issue.

Full error spew:

INFO:Loading decapoda-research_llama-65b-hf...
Traceback (most recent call last):
File "/home/user/oobabooga_linux/text-generation-webui/server.py", line 919, in <module>
shared.model, shared.tokenizer = load_model(shared.model_name)
File "/home/user/oobabooga_linux/text-generation-webui/modules/models.py", line 117, in load_model
model = LoaderClass.from_pretrained(Path(f"{shared.args.model_dir}/{model_name}"), torch_dtype=torch.bfloat16 if shared.args.bf16 else torch.float16)
File "/home/user/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 471, in from_pretrained
return model_class.from_pretrained(
File "/home/user/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2624, in from_pretrained
init_contexts = [deepspeed.zero.Init(config_dict_or_path=deepspeed_config())] + init_contexts
File "/home/user/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 762, in __init__
self.param_swapper = AsyncPartitionedParameterSwapper(_ds_config, self.dtype)
File "/home/user/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/deepspeed/runtime/swap_tensor/partitioned_param_swapper.py", line 40, in __init__
aio_op = AsyncIOBuilder().load(verbose=False)
File "/home/user/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/deepspeed/ops/op_builder/builder.py", line 445, in load
return self.jit_load(verbose)
File "/home/user/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/deepspeed/ops/op_builder/builder.py", line 449, in jit_load
raise RuntimeError(
RuntimeError: Unable to JIT load the async_io op due to it not being compatible due to hardware/software issue. None

Solution: I don't know; this is where I am stuck. #1037 suggests that I just need to 'apt install libaio-dev', but I've done that and it doesn't help.

I'm still stuck and cannot for the life of me get the --deepspeed option working on 22.04. I'd truly appreciate any help. Thanks to all in advance!

[AlpsAficionado]

Here is the output of 'ds_report' in my working conda environment in the VM.

DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
async_io ............... [YES] ...... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.0
 [WARNING]  using untested triton version (2.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/home/user/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/torch']
torch version .................... 2.0.0
deepspeed install path ........... ['/home/user/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/deepspeed']
deepspeed info ................... 0.9.2, unknown, unknown
torch cuda version ............... 11.7
torch hip version ................ None
nvcc version ..................... 11.7
deepspeed wheel compiled w. ...... torch 2.0, cuda 11.7

[lucasjinreal]

Just same error...

[loadams]

Hi @AlpsAficionado - this looks like something incorrect is setup in your environment gcc wise. If you're still having an issue, could you open an issue where we can dig into the specifics more?