-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor the subquery code and fix outer condition queries #8081
Conversation
@rkuchan is it possible for you to test this out with a few scenarios? I added some tests, but some human eyes would also be good. @jwilder I'm not 100% confident in the performance characteristics of this, but I give my rationale for why I think it's a positive change anyway in the PR. What do you think? |
This isn't a high priority for testing. We're going to push it to the 1.3 release since it's more involved than other bugfixes. |
529d8b9
to
3928786
Compare
I still need to expand the benchmarks to cover more aspects, but the initial numbers are not promising for the simplest case:
That may not be an acceptable speed loss. Especially with 100K points. |
I am so confused, but rerunning the benchmarks that I have locally on my computer gives me this now:
If true, this is definitely acceptable. I'm also trying to use the same old technique for subqueries that I was using, but trying to get the changes I made in this query and I am finding it very difficult to keep track of the code. |
3928786
to
afa1d52
Compare
afa1d52
to
e169aec
Compare
e169aec
to
ff79094
Compare
ff79094
to
3da8fe9
Compare
This change refactors the subquery code into a separate builder class to help allow for more reuse and make the functions smaller and easier to read. The previous function that handled most of the code was too big and impossible to reason through. This also goes and replaces the complicated logic of aggregates that had a subquery source with the simpler IteratorMapper. I think the overhead from the IteratorMapper will be more, but I also believe that the actual code is simpler and more robust to produce more accurate answers. It might be a future project to optimize that section of code, but I don't have any actual numbers for the efficiency of one method and I believe accuracy and code clarity may be more important at the moment since I am otherwise incapable of reading my own code.
3da8fe9
to
df30a4d
Compare
This change refactors the subquery code into a separate builder class to
help allow for more reuse and make the functions smaller and easier to
read.
The previous function that handled most of the code was too big and
impossible to reason through.
This also goes and replaces the complicated logic of aggregates that had
a subquery source with the simpler IteratorMapper. I think the overhead
from the IteratorMapper will be more, but I also believe that the actual
code is simpler and more robust to produce more accurate answers. It
might be a future project to optimize that section of code, but I don't
have any actual numbers for the efficiency of one method and I believe
accuracy and code clarity may be more important at the moment since I am
otherwise incapable of reading my own code.
Fixes #8045.