Skip to content
This repository has been archived by the owner on Sep 18, 2023. It is now read-only.

Spark managed spill in columnar sort #417

Open
zhouyuan opened this issue Jul 19, 2021 · 0 comments
Open

Spark managed spill in columnar sort #417

zhouyuan opened this issue Jul 19, 2021 · 0 comments
Labels
enhancement New feature or request performance

Comments

@zhouyuan
Copy link
Collaborator

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Sort should support Spill under lower memory. This could reduce memory footprint and allow to run on bigger dataset

Describe the solution you'd like
As Spark did, Sort should respect the memory size and spill to disk under low memory

Describe alternatives you've considered
N/A

Additional context
N/A

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request performance
Projects
None yet
Development

No branches or pull requests

2 participants