Using distconv and model-parallel for GAN #2447
-
|
I am trying to run a multi-gpu GAN similar to Below I have put the log file for each run, batch script used to run (maybe I am missing some mpi flag?), and a short python code snippet showing how I try to invoke model parallelism. (Happy to also provide the prototext if it is useful). one GPU: Two GPUs: batch file used to run: #!/bin/bash
export DISTCONV_WS_CAPACITY_FACTOR=0.8
export LBANN_DISTCONV_HALO_EXCHANGE=AL
export LBANN_DISTCONV_TENSOR_SHUFFLER=AL
export LBANN_DISTCONV_CONVOLUTION_FWD_ALGORITHM=AUTOTUNE
export LBANN_DISTCONV_CONVOLUTION_BWD_DATA_ALGORITHM=AUTOTUNE
export LBANN_DISTCONV_CONVOLUTION_BWD_FILTER_ALGORITHM=AUTOTUNE
export LBANN_DISTCONV_RANK_STRIDE=1
export LBANN_DISTCONV_NUM_IO_PARTITIONS=2
export LBANN_KEEP_ERROR_SIGNALS=1
echo "Started at $(date)"
mpiexec -n 2 --map-by ppr:2:node -wdir /home/jwilliams/project/lbann_gan/stylegan2_withdiscriminator_weightDemodTal_modelparallel_filterLastLayer/20240507_114638_gan_HW256_nonsaturatingloss_n1_ppn2 /home/jwilliams/lbann-builds/lbann-latest/build_appspack/install/bin/lbann --use_data_store --preload_data_store --prototext=/home/jwilliams/project/lbann_gan/stylegan2_withdiscriminator_weightDemodTal_modelparallel_filterLastLayer/20240507_114638_gan_HW256_nonsaturatingloss_n1_ppn2/experiment.prototext
status=$?
echo "Finished at $(date)"
exit ${status}I am setting parallel strategy in the following way: environment = {}
parallel_strategy = None
procs_per_node = 1
if args.modelparallel:
height_groups = 2
procs_per_node = 2
environment = get_distconv_environment(num_io_partitions=procs_per_node)
parallel_strategy = lbann.core.util.get_parallel_strategy_args(
height_groups=procs_per_node,
)
# setup model
# (code omitted for brevity)...
# assign parallel strategy to each layer
layers = list(lbann.traverse_layer_graph(loss))
for l in layers:
l.parallel_strategy = parallel_strategy
kwargs = {
"nodes": 1,
"procs_per_node": procs_per_node,
"scheduler": "openmpi",
}
# run
lbann.run(
trainer,
model,
data_reader,
opt,
job_name=job_name,
environment=environment,
lbann_args=lbann_args,
**kwargs,
) |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 7 replies
-
|
FYI, on this system we built the dependencies with spack: and then compiled lbann itself with cmake (there was an issue in building lbann itself with spack). |
Beta Was this translation helpful? Give feedback.
-
|
Just checked the cmake log. Distconv was not enabled... my bad! thanks. Recompiling now |
Beta Was this translation helpful? Give feedback.
-
|
You may need to modify the spack command to include |
Beta Was this translation helpful? Give feedback.
-
|
Is there a repo for distconv that I could use to build from source (not spack)? I only found this old one https://github.com/bvanessen/lbann_distconv |
Beta Was this translation helpful? Give feedback.
-
|
DistConv is part of DiHydrogen. In spack, building |
Beta Was this translation helpful? Give feedback.
-
|
Ok thanks, I built with rebuilt lbann with spack using Get similar behaviour though. This time the maximum memory increases from 6GB with 1 GPU to "20.5 GiB mean, 15.9 GiB median, 25.1 GiB max, 15.9 GiB min" with 2 GPUs. Here is my batch.sh file (I am using openmpi scheduler), and prototext below if that helps. #!/bin/bash
export DISTCONV_WS_CAPACITY_FACTOR=0.8
export LBANN_DISTCONV_HALO_EXCHANGE=AL
export LBANN_DISTCONV_TENSOR_SHUFFLER=AL
export LBANN_DISTCONV_CONVOLUTION_FWD_ALGORITHM=AUTOTUNE
export LBANN_DISTCONV_CONVOLUTION_BWD_DATA_ALGORITHM=AUTOTUNE
export LBANN_DISTCONV_CONVOLUTION_BWD_FILTER_ALGORITHM=AUTOTUNE
export LBANN_DISTCONV_RANK_STRIDE=1
export LBANN_DISTCONV_NUM_IO_PARTITIONS=1
export LBANN_KEEP_ERROR_SIGNALS=1
echo "Started at $(date)"
mpiexec -n 2 --map-by ppr:2:node -wdir /netfs/smain01/scafellpike/local/HT04543/jxc06/jxw92-jxc06/project/lbann_stylES/multiGPU_distconv/distconv_branch/20240509_155702_gan_HW256_nonsaturatingloss_n1_ppn2 /lustre/scafellpike/local/HT04543/jxc06/jxw92-jxc06/spack/opt/spack/linux-rhel7-skylake_avx512/gcc-11.2.0/lbann-develop-qhi2drst2tayjyhwiajajfqtzh7azpnk/bin/lbann --use_data_store --preload_data_store --num_io_threads=1 --prototext=/netfs/smain01/scafellpike/local/HT04543/jxc06/jxw92-jxc06/project/lbann_stylES/multiGPU_distconv/distconv_branch/20240509_155702_gan_HW256_nonsaturatingloss_n1_ppn2/experiment.prototext
status=$?
echo "Finished at $(date)"
exit ${status}Also some other questions on distconv:
|
Beta Was this translation helpful? Give feedback.
DistConv is part of DiHydrogen. In spack, building
dihydrogen+distconvshould get you there -- though make sure you're up to date with LBANN@develop since I merged a PR yesterday addressing a versioning issue. If you build H2 directly with CMake, the option is-D H2_ENABLE_DISTCONV_LEGACY=ON.