Skip to content
This repository has been archived by the owner on Sep 18, 2023. It is now read-only.

[NSE-45] BHJ memory leak #50

Merged
merged 11 commits into from
Feb 1, 2021
Merged

Conversation

zhztheplayer
Copy link
Collaborator

@zhztheplayer zhztheplayer commented Jan 20, 2021

#45

This is ideally to fix leaks and reduce process RSS during BHJ-related query execution. Still doing a final check for memory segmentation and other minor stuffs.

Checklist:

  1. BHJ: Fix unclosed hash map on driver
  2. BHJ: Fix unclosed input batches on driver
  3. Remove use of unreliable JVM API finalize()
  4. Remove ref-count for java-side hash maps (we should avoid creating more manual ref-count logic in JVM code unless performance getting really worse)
  5. Remove option broadcastCacheTimeout which is no longer needed.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@zhztheplayer zhztheplayer changed the title [NSE-45][DNM] BHJ memory leak [NSE-45] BHJ memory leak Jan 22, 2021
@xuechendi
Copy link
Collaborator

@zhztheplayer , codes are looking good to me, let me verify with a tpcds run.

@zhztheplayer zhztheplayer changed the title [NSE-45] BHJ memory leak [NSE-45][DNM] BHJ memory leak Jan 22, 2021
@xuechendi
Copy link
Collaborator

@zhouyuan, seems this PR will fail in my setup tpcds sf500 dpp test, just synced with @zhztheplayer , he will continue to verify with TPCDS

@zhztheplayer
Copy link
Collaborator Author

Thanks @xuechendi Let's hold off until everything gets stable.

@github-actions
Copy link

@github-actions
Copy link

@github-actions
Copy link

@zhouyuan zhouyuan changed the title [NSE-45][DNM] BHJ memory leak [NSE-45] BHJ memory leak Feb 1, 2021
@zhouyuan zhouyuan merged commit 5c00073 into oap-project:master Feb 1, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants