Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor the subquery code and fix outer condition queries #8081

Merged
merged 1 commit into from
May 1, 2017

Conversation

jsternberg
Copy link
Contributor

@jsternberg jsternberg commented Mar 1, 2017

This change refactors the subquery code into a separate builder class to
help allow for more reuse and make the functions smaller and easier to
read.

The previous function that handled most of the code was too big and
impossible to reason through.

This also goes and replaces the complicated logic of aggregates that had
a subquery source with the simpler IteratorMapper. I think the overhead
from the IteratorMapper will be more, but I also believe that the actual
code is simpler and more robust to produce more accurate answers. It
might be a future project to optimize that section of code, but I don't
have any actual numbers for the efficiency of one method and I believe
accuracy and code clarity may be more important at the moment since I am
otherwise incapable of reading my own code.

Fixes #8045.

  • Rebased/mergable
  • Tests pass
  • CHANGELOG.md updated

@jsternberg
Copy link
Contributor Author

@rkuchan is it possible for you to test this out with a few scenarios? I added some tests, but some human eyes would also be good.

@jwilder I'm not 100% confident in the performance characteristics of this, but I give my rationale for why I think it's a positive change anyway in the PR. What do you think?

@jsternberg
Copy link
Contributor Author

This isn't a high priority for testing. We're going to push it to the 1.3 release since it's more involved than other bugfixes.

@jsternberg jsternberg changed the base branch from 1.2 to master March 7, 2017 19:34
@jsternberg jsternberg force-pushed the js-8045-subqueries-with-conditions branch from 529d8b9 to 3928786 Compare March 7, 2017 19:35
@jsternberg
Copy link
Contributor Author

I still need to expand the benchmarks to cover more aspects, but the initial numbers are not promising for the simplest case:

benchmark                                     old ns/op     new ns/op     delta
BenchmarkSelect_Subquery_Aggregate_1K-8       785914        832085        +5.87%
BenchmarkSelect_Subquery_Aggregate_100K-8     62048569      72915654      +17.51%

benchmark                                     old allocs     new allocs     delta
BenchmarkSelect_Subquery_Aggregate_1K-8       7070           7077           +0.10%
BenchmarkSelect_Subquery_Aggregate_100K-8     700076         700083         +0.00%

benchmark                                     old bytes     new bytes     delta
BenchmarkSelect_Subquery_Aggregate_1K-8       411989        412261        +0.07%
BenchmarkSelect_Subquery_Aggregate_100K-8     40804498      40804770      +0.00%

That may not be an acceptable speed loss. Especially with 100K points.

@jsternberg
Copy link
Contributor Author

I am so confused, but rerunning the benchmarks that I have locally on my computer gives me this now:

jsternberg:[~/g/s/g/i/influxdb] $ (tmp $) benchcmp old.txt new.txt
benchmark                                     old ns/op     new ns/op     delta
BenchmarkSelect_Subquery_Aggregate_1K-8       794322        783705        -1.34%
BenchmarkSelect_Subquery_Aggregate_100K-8     63811348      61522037      -3.59%

benchmark                                     old allocs     new allocs     delta
BenchmarkSelect_Subquery_Aggregate_1K-8       7070           7077           +0.10%
BenchmarkSelect_Subquery_Aggregate_100K-8     700076         700083         +0.00%

benchmark                                     old bytes     new bytes     delta
BenchmarkSelect_Subquery_Aggregate_1K-8       411989        412261        +0.07%
BenchmarkSelect_Subquery_Aggregate_100K-8     40804498      40804770      +0.00%

If true, this is definitely acceptable. I'm also trying to use the same old technique for subqueries that I was using, but trying to get the changes I made in this query and I am finding it very difficult to keep track of the code.

This change refactors the subquery code into a separate builder class to
help allow for more reuse and make the functions smaller and easier to
read.

The previous function that handled most of the code was too big and
impossible to reason through.

This also goes and replaces the complicated logic of aggregates that had
a subquery source with the simpler IteratorMapper. I think the overhead
from the IteratorMapper will be more, but I also believe that the actual
code is simpler and more robust to produce more accurate answers. It
might be a future project to optimize that section of code, but I don't
have any actual numbers for the efficiency of one method and I believe
accuracy and code clarity may be more important at the moment since I am
otherwise incapable of reading my own code.
@jsternberg jsternberg force-pushed the js-8045-subqueries-with-conditions branch from 3da8fe9 to df30a4d Compare April 29, 2017 01:28
@jsternberg jsternberg requested a review from benbjohnson May 1, 2017 18:25
@jsternberg jsternberg merged commit 9bd7fce into master May 1, 2017
@jsternberg jsternberg deleted the js-8045-subqueries-with-conditions branch May 1, 2017 22:11
@121watts 121watts mentioned this pull request Aug 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants