Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add explain verbose with limit push down #751

Merged
merged 1 commit into from
Jul 19, 2021

Conversation

jimexist
Copy link
Member

@jimexist jimexist commented Jul 19, 2021

Which issue does this PR close?

fix explain verbose with limit push down

re #733

Rationale for this change

What changes are included in this PR?

before fix:

> create external table test stored as PARQUET location '/Users/jiayu_liu/Desktop/aggregate_test_100.parquet';
0 rows in set. Query took 0.001 seconds.
> explain verbose select c3, c9 from test limit 3;
+-----------------------------------------+----------------------------------------------------------------------------------------------------------------+
| plan_type                               | plan                                                                                                           |
+-----------------------------------------+----------------------------------------------------------------------------------------------------------------+
| logical_plan                            | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=None                                                                            |
| logical_plan after projection_push_down | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=Some([2, 8])                                                                    |
| logical_plan after simplify_expressions | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=Some([2, 8])                                                                    |
| logical_plan after projection_push_down | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=Some([2, 8])                                                                    |
| logical_plan after simplify_expressions | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=Some([2, 8])                                                                    |
| physical_plan                           | GlobalLimitExec: limit=3                                                                                       |
|                                         |   ProjectionExec: expr=[c3@0 as c3, c9@1 as c9]                                                                |
|                                         |     ParquetExec: batch_size=8192, limit=None, partitions=[/Users/jiayu_liu/Desktop/aggregate_test_100.parquet] |
+-----------------------------------------+----------------------------------------------------------------------------------------------------------------+

after the fix

> create external table test stored as PARQUET location '/Users/jiayu_liu/Desktop/aggregate_test_100.parquet';
0 rows in set. Query took 0.001 seconds.
> explain verbose select c3, c9 from test limit 3;
+-----------------------------------------+----------------------------------------------------------------------------------------------------------------+
| plan_type                               | plan                                                                                                           |
+-----------------------------------------+----------------------------------------------------------------------------------------------------------------+
| logical_plan                            | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=None                                                                            |
| logical_plan after projection_push_down | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=Some([2, 8])                                                                    |
| logical_plan after simplify_expressions | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=Some([2, 8])                                                                    |
| logical_plan after limit_push_down      | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=Some([2, 8]), limit=3                                                           |
| logical_plan after projection_push_down | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=Some([2, 8]), limit=3                                                           |
| logical_plan after simplify_expressions | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=Some([2, 8]), limit=3                                                           |
| logical_plan after limit_push_down      | Limit: 3                                                                                                       |
|                                         |   Projection: #test.c3, #test.c9                                                                               |
|                                         |     TableScan: test projection=Some([2, 8]), limit=3                                                           |
| physical_plan                           | GlobalLimitExec: limit=3                                                                                       |
|                                         |   ProjectionExec: expr=[c3@0 as c3, c9@1 as c9]                                                                |
|                                         |     ParquetExec: batch_size=3, limit=Some(3), partitions=[/Users/jiayu_liu/Desktop/aggregate_test_100.parquet] |
+-----------------------------------------+----------------------------------------------------------------------------------------------------------------+

Are there any user-facing changes?

@github-actions github-actions bot added the datafusion Changes in the datafusion crate label Jul 19, 2021
@alamb alamb changed the title fix explain verbose with limit push down Add explain verbose with limit push down Jul 19, 2021
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @jimexist -- this definitely is an improvement (and relates to #733)

One thing that bothers me about #733 is that each optimizer pass needs to handle explain plans specially, but I think that means we end up missing some of them (and also not catching user defined optimizer passes)

I am thinking it might be better to (eventually) handle explain plan capturing in the planners. I'll try and work on that after #744

@alamb alamb merged commit 6f9681d into apache:master Jul 19, 2021
@jimexist jimexist deleted the fix-explain-limit-push-down branch July 19, 2021 14:19
@Dandandan
Copy link
Contributor

Nice fix @jimexist

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
datafusion Changes in the datafusion crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants