Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support arbitrary expressions in LIMIT clause #9821

Closed
alamb opened this issue Mar 27, 2024 · 5 comments · Fixed by #13028
Closed

Support arbitrary expressions in LIMIT clause #9821

alamb opened this issue Mar 27, 2024 · 5 comments · Fixed by #13028
Assignees

Comments

@alamb
Copy link
Contributor

alamb commented Mar 27, 2024

Follow on to #9506

The idea is to support arbitrary expressions that can be consolidated to a constant in the LIMIT clause. For example

select * from (values (1)) LIMIT 10*100;
Error during planning: Unsupported operator for LIMIT clause

This query should be able to run (and return the single value)

select * from (values (1)) LIMIT 10+100;
+---------+
| column1 |
+---------+
| 1       |
+---------+

#9790 adds support for basic +/- but the general purpose solution that would handle any expr that can be consolidated to a constant would be better

As suggested by @jonahgao this might look like change the Limit logical plan to support arbitrary expressions?

pub struct Limit {
    pub skip: Expr,
    pub fetch: Option<Expr>,
    pub input: Arc<LogicalPlan>,
}

The SimplifyExpressions rule can automatically optimize them into constants. Some optimization rules such as PushDownLimit only run when the limit expression is a constant. We may need to add a cast for the limit expression when planning, only checking if it is a constant of type u64.

When creating the LimitExec physical plan, convert the limit expression into PhysicalExpr and evaluate it.

Originally posted by @jonahgao in #9790 (comment)

@alamb alamb changed the title Could we change the Limit logical plan to support arbitrary expressions? Support arbitrary expressions in LIMIT clause Mar 27, 2024
@Lordworms
Copy link
Contributor

I can do this one since I have been doing a related limit pushdown feature.

@Lordworms
Copy link
Contributor

wait until #9815 (comment) merged

@alamb
Copy link
Contributor Author

alamb commented Oct 15, 2024

This one is probably ready to work on now

@alamb
Copy link
Contributor Author

alamb commented Oct 15, 2024

While re-reading this I think we should start with implementhing limits that can be evaluated to a constant by the time the physical plan is created (aka don't change the physical execution plans)

It is not clear to me what LIMIT 100 + x means

The key usecases are:

@jonahgao
Copy link
Member

While re-reading this I think we should start with implementhing limits that can be evaluated to a constant by the time the physical plan is created (aka don't change the physical execution plans)

I will switch my focus to working on it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants