Skip to content
This repository has been archived by the owner on Sep 18, 2023. It is now read-only.

[NSE-1135] Use spark-3.2.1 shim layer for spark-3.2.2 #1154

Merged
merged 2 commits into from
Nov 3, 2022

Conversation

PHILO-HE
Copy link
Collaborator

@PHILO-HE PHILO-HE commented Nov 1, 2022

What changes were proposed in this pull request?

After cleaning up some code in a previous patch, it looks spark321 shim layer can be used into spark322, i.e., building with spark-3.2 or spark-3.2.1 profile, then using the generated jar for spark322.
This patch is experimental and needs sufficient tests.

How was this patch tested?

Benchmark tests on spark-322.

@github-actions
Copy link

github-actions bot commented Nov 1, 2022

#1135

@PHILO-HE
Copy link
Collaborator Author

PHILO-HE commented Nov 1, 2022

TPC-DS test passed.
@jackylee-ch, please have some tests with this patch. Please note you should use -Pspark-3.2 or -Pspark-3.2.1 to build the project with spark 3.2.1 dependency. It is expected that spark-3.2.1 shim layer is workable for spark-3.2.2 after our previous fixes, which needs more tests to verify.

@PHILO-HE PHILO-HE merged commit 9cbcd7e into oap-project:main Nov 3, 2022
zhouyuan added a commit to zhouyuan/native-sql-engine that referenced this pull request Dec 16, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant