-
Notifications
You must be signed in to change notification settings - Fork 770
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Parquet] Add BooleanArray based row selection #6624
base: master
Are you sure you want to change the base?
Conversation
c597e4a
to
42938bf
Compare
42938bf
to
1f017ff
Compare
The benchmark can take a while to run, here're some of my results visualized by ChatGPT. Note that both x and y axies are in log scale. Selection ratio 1e-5 means 0.0001% of the rows are selected, 0.3 means 30% are selected, and so on. |
Perhaps we could model the number of disjoint selections as opposed to raw selectivity, this more accurately models the sorts of selection one gets from evaluation against the page index (which necessarily has page level granularity) Edit: although the methods benchmarked are those used for late materialization, so perhaps it doesn't matter |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've not reviewed in fine detail, but the code looks good to me
I will try and find time to review this today or tomorrow as well -- thank you @XiangpengHao and @tustvold |
Which issue does this PR close?
Part of #5523
Rationale for this change
Implement the row selection based on BooleanArray.
The implementation only covers the core operations like
union
,intersection
,and_then
, and is already quite large. Once we agree with the generally direction, I'll add remaining implementaions in follow up PRs.What changes are included in this PR?
I added more benchmark to cover different selection ratios, as suggested by @tustvold and @alamb
Are there any user-facing changes?