-
Notifications
You must be signed in to change notification settings - Fork 992
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bit mask support for dropout #656
Comments
Hi @eric-haibin-lin, Thank you for your question. Technically nothing prevents us from introducing dropout primitive in the library, including the 1-bit mask support. The main question we need to answer to make this happen is what API and behavior should look like to make the functionality generally useful. For dropout the main source of concern is the fact that it relies on random number generator, which may behave differently in different applications, so having random number generator as part of implementation would be a major source of incompatibility and thread safety issues. A couple of follow up questions so that I can better understand what you are looking for:
|
I think the random number generator is taking a significant part in the execution of dropout. That's why we optimized it in MXNet with
in which, the @eric-haibin-lin @apeforest Could you please share more insights about the random seed distribution in MXNet and reproducibility of the operator? |
The random seed should be taken from MXNet so that if user specify a random seed in mxnet it should guarantee reproducibility. A similar approach has been done for cuDNN library: apache/mxnet#17547 |
I notice there is a RFC opened for this request. You may want to take a look. @eric-haibin-lin @apeforest @pengzhao-intel |
For dropout training, one can save the dropout mask with 1 bit per coordinate. Can we support that in DNNL? Memory is precious.
The text was updated successfully, but these errors were encountered: