-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[improvement](scan) Support pushdown execute expr ctx #15917
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
clang-tidy made some suggestions
9eb8d18
to
a874c3e
Compare
a874c3e
to
0ca40dc
Compare
TeamCity pipeline, clickbench performance test result: |
c2084b9
to
c7f6225
Compare
c7f6225
to
dcf698e
Compare
dcf698e
to
8f86ee4
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
clang-tidy made some suggestions
60d573d
to
9a5398c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
clang-tidy made some suggestions
176f705
to
8ef09d0
Compare
run p0 |
7f1b9e2
to
72f0935
Compare
run buildall |
run arm |
run p0 |
run buildall |
72f0935
to
8d67ce7
Compare
run buildall |
run clickbench |
1 similar comment
run clickbench |
504c929
to
07a2765
Compare
run buildall |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
PR approved by at least one committer and no changes requested. |
PR approved by anyone and no changes requested. |
07a2765
to
f19a253
Compare
./run buildall |
f19a253
to
267b620
Compare
./run buildall |
1 similar comment
./run buildall |
run buildall |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
clang-tidy made some suggestions
858fd7b
to
de817b8
Compare
run buildall |
de817b8
to
deca217
Compare
run buildall |
function pushdown: apache#10355 NGram BloomFilter Index apply like pushdown: apache#11579 Enabled by default, make sure it stays active. If NGram BloomFilter Index is not used, this like pushdown can be replaced by apache#15917, which can push down all expressions including like.
run buildall |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
In the past, only simple predicates (slot=const), and, like, or (only bitmap index) could be pushed down to the storage layer. scan process: Read part of the column first, and calculate the row ids with a simple push-down predicate. Use row ids to read the remaining columns and pass them to the scanner, and the scanner filters the remaining predicates. This pr will also push-down the remaining predicates (functions, nested predicates...) in the scanner to the storage layer for filtering. scan process: Read part of the column first, and use the push-down simple predicate to calculate the row ids, (same as above) Use row ids to read the columns needed for the remaining predicates, and use the pushed-down remaining predicates to reduce the number of row ids again. Use row ids to read the remaining columns and pass them to the scanner.
Proposed changes
Issue Number: close #xxx
Problem summary
motivation
In the past, only simple predicates (
slot=const
),and
,like
,or
(only bitmap index) could be pushed down to the storage layer. scan process:This pr will also push-down the remaining predicates (functions, nested predicates...) in the scanner to the storage layer for filtering. scan process:
expected
select a,b,c,... from tbl where func(a) = 1
. The performance will be significantly improved, and the effect depends on the number of selected columns and the filtering rate.example: 10 times better performance
select a from tbl where func(a) = 1
. Performance is expected to remain the same, as it behaves the same as before.(In the test, there is a 2% loss in performance. I found that
VExprContext::filter_block
performs differently in different locations, which may be related to cache miss, but the tool did not detect it)clickbench test
default:
set enable_remaining_expr_pushdown = true;
Checklist(Required)
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...