how deepspeed can avoid doing all_reduce?

[luuck]

I set something in DeepSpeed in order to bypass being all-reduced on all model parameters.

In order words, below is exactly the parameter list I need to bypass doing allreduce:
bypass_list = [x for x in model.parameters() if getattr(param, '_tutel_expert', False) == True]

how DeepSpeed can avoid doing all_reduce for bypass_list above?

[jomayeri]

There is an outstanding PR that may be of use: #6675