Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Infinite recursion when calling canonicalized on query plan for TPC-H q8 #576

Closed
andygrove opened this issue Jun 14, 2024 · 3 comments · Fixed by #582
Closed

Infinite recursion when calling canonicalized on query plan for TPC-H q8 #576

andygrove opened this issue Jun 14, 2024 · 3 comments · Fixed by #582
Assignees
@andygrove
Labels
bug Something isn't working
Milestone
0.1.0

Comments

@andygrove
Copy link
Member

Describe the bug

Note: This was originally reported as an issue with xxhash64 in #517 but xxhash64 is not really involved.

When running TPC-H q8 with xxhash64 enabled, the executor becomes unresponsive, with most cores at 100%. Running a kill -QUIT command against the process dumps the current JVM stacks and shows a 500+ line stack performing plan transformation that appears to be the issue.

See attached log.

executor-thread-dump.txt.gz

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response
@andygrove andygrove added the bug Something isn't working label Jun 14, 2024
@andygrove andygrove self-assigned this Jun 14, 2024
@andygrove andygrove mentioned this issue Jun 14, 2024
Closed
@andygrove andygrove added this to the 0.1.0 milestone Jun 14, 2024
@andygrove
Copy link
Member Author

The issue seems to be that doCanonicalized runs forever for this query.

@andygrove andygrove changed the title Spark executor becomes unresponsive when trying to transform query plan for TPC-H q8 Infinite recursion when calling canonicalized on query plan for TPC-H q8 Jun 17, 2024
@andygrove
Copy link
Member Author

The issue goes away if I revert #447 so perhaps this is related to some combination of bloom filters and reused exchanges?

@viirya
Copy link
Member

viirya commented Jun 17, 2024

Hmm, interesting. I will take a look.

@viirya viirya mentioned this issue Jun 17, 2024
Merged
@andygrove andygrove mentioned this issue Jun 17, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andygrove andygrove

Labels
bug Something isn't working
Projects
None yet
Milestone
0.1.0
Development

Successfully merging a pull request may close this issue.

fix: Avoid recursive call to canonicalizePlans viirya/arrow-datafusion-comet
2 participants
@viirya @andygrove