Optimize softmax cpu by parallel using openmp. #36
Conversation
|
|
||
| // Put the intermediate result X - max(X) into Y by first copying X to Y, and then subtracting max from each entry | ||
| gsl::copy(gsl::make_span(Xdata, nd), gsl::make_span(Ydata, nd)); | ||
| static const int kGROUP = 8; |
There was a problem hiding this comment.
8 [](start = 28, length = 1)
Why this hardcoded number? Would it be more readable and efficient to use Eigen reduction and broadcast directly for the whole Softmax computation? https://eigen.tuxfamily.org/dox/group__TutorialReductionsVisitorsBroadcasting.html
There was a problem hiding this comment.
This is to control the parallel. I could change it to let openmp decide it dynamically.
Currently no idea on how to use Eigen for such parallel. Could you provide more information?
There was a problem hiding this comment.
There was a problem hiding this comment.
You considered the same thing when I start working on this. I read this before, it have parallel gemm and some other matrix algebra. But no generic support to control the parallel on rowwise, colwise, cwise, etc. Some body raised such issue for years, yet no implementation accepted yet.
So outside control parallel is manually needed now.
Also I changed the group count calculation logic. Please check again.
Thanks, Lei
* Add inference and evaluation for BERT QDQ * Add fp16 and int8 flags * Modify data reader to handle multiple features * Set batch size to 1 due to some qdq model might have issues processing batch size greate than 1 * Add comment and ignore last commit commit * Add evaluation for squad v2.0 * Run evaluate script directly in e2e script
* enable MIGraphX EP on Windows * [MIGraphX EP] Fix provider options * fix formatting * unify the package name for both rocm and migraphx * fix compilation after moving to rocm6.2 * make STREAM_SYNC the default * workaround hip sdk bug on windows * Revert rename of private var for now --------- Co-authored-by: Filip Jankovic <filip.jankovic@amd.com> Co-authored-by: Ted Themistokleous <107195283+TedThemistokleous@users.noreply.github.com> Co-authored-by: Ted Themistokleous <tedthemistokleous@amd.com>
* enable MIGraphX EP on Windows * [MIGraphX EP] Fix provider options * fix formatting * unify the package name for both rocm and migraphx * fix compilation after moving to rocm6.2 * make STREAM_SYNC the default * workaround hip sdk bug on windows * Revert rename of private var for now --------- Co-authored-by: Filip Jankovic <filip.jankovic@amd.com> Co-authored-by: Ted Themistokleous <107195283+TedThemistokleous@users.noreply.github.com> Co-authored-by: Ted Themistokleous <tedthemistokleous@amd.com>
No description provided.