-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for smallint to PrefixSort #10946
Conversation
✅ Deploy Preview for meta-velox canceled.
|
eb61aee
to
a9b4c86
Compare
@mbasmanova would you please take a look ? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kevincmchen Thank you for extending prefix-sort. I noticed you added a benchmark. What did it show?
@@ -54,7 +54,7 @@ class PrefixSortEncoder { | |||
} | |||
|
|||
/// @tparam T Type of value. Supported type are: uint64_t, int64_t, uint32_t, | |||
/// int32_t, float, double, Timestamp. TODO Add support for int16_t, uint16_t. | |||
/// int32_t, int16_t, uint16_t, float, double, Timestamp. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we need support for uint16_t?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just followed the logic of int32/uint32 to support int16/uint16. Its only a specific explanation focused on the function itself.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@skadilover Do we need support for unsigned integer types? Is there any use case for these?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They are used internal, singed ints are always converted to usigned ints
template <>
FOLLY_ALWAYS_INLINE void PrefixSortEncoder::encodeNoNulls(
int64_t value,
char* dest) const {
encodeNoNulls((uint64_t)(value ^ (1ull << 63)), dest);
}
@kevincmchen CI is red. Would you take a look? |
here is small int benchmark: prefixSort vs std:sort |
This is a lot of numbers. Would you summary the results and add the summary to the PR description? |
Here's a table summarizing the benchmark results:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks.
@mbasmanova has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
this is due to hadoop upgrade and #10947 is working on this issue. |
Thanks @mbasmanova |
@kevincmchen Would you rebase to resolve CI failures? |
a9b4c86
to
5009b2e
Compare
Thanks for your reminder. i will rebase and solve the CI failures. |
@mbasmanova has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
@mbasmanova there are some facebook internal CI failures , could you please help me check it ? |
@kevincmchen Seeing merge conflicts. Would you rebase once again? |
5009b2e
to
df61e87
Compare
@mbasmanova has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
@mbasmanova this pr #10965 should be merged first, or it will lead failure when building PyVelox. |
df61e87
to
6e422a9
Compare
@mbasmanova CI is all passed. would you please help me merge this pr? |
@mbasmanova has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
@mbasmanova merged this pull request in 98bbb73. |
Conbench analyzed the 1 benchmark run on commit There were no benchmark performance regressions. 🎉 The full Conbench report has more details. |
According to the benchmark, for data sets larger than 0.5k, PrefixSort outperforms std::sort with performance improvements ranging from approximately 250% to over 500%. Here's a summary of the benchmark results:
follow-up #8350
Part of #6766