Skip to content
This repository has been archived by the owner on Sep 18, 2023. It is now read-only.

[NSE-758] Upgrade to spark 3.1.3 from 3.1.1 #756

Closed
wants to merge 3 commits into from

Conversation

PHILO-HE
Copy link
Collaborator

@PHILO-HE PHILO-HE commented Mar 9, 2022

According to Jin Hao's feedback, spark-3.1.2 is the version we used in runtime env. for perf. test, even though spark-3.1.1 is the imported dependency version. Maybe, for spark-3.1.1 shim layer, spark-3.1.2 should also be supported.

@github-actions
Copy link

github-actions bot commented Mar 9, 2022

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/native-sql-engine/issues

Then could you also rename commit message and pull request title in the following format?

[NSE-${ISSUES_ID}] ${detailed message}

See also:

@PHILO-HE PHILO-HE changed the title Allow spark 3.1.2 pass the version check in shim layer Add spark-3.1.2 as supported version for spark-3.1.1 shim layer Mar 9, 2022
@PHILO-HE PHILO-HE changed the title Add spark-3.1.2 as supported version for spark-3.1.1 shim layer Add spark-3.1.2 as another supported version for spark-3.1.1 shim layer Mar 9, 2022
@PHILO-HE PHILO-HE marked this pull request as draft March 9, 2022 14:29
val DESCRIPTOR_STRINGS = Seq(s"$DESCRIPTOR")
val SPARK311_DESCRIPTOR = SparkShimDescriptor(3, 1, 1)
val SPARK312_DESCRIPTOR = SparkShimDescriptor(3, 1, 2)
val DESCRIPTOR_STRINGS = Seq(s"$SPARK311_DESCRIPTOR", s"$SPARK312_DESCRIPTOR")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3.1.* should be better

@PHILO-HE PHILO-HE changed the title Add spark-3.1.2 as another supported version for spark-3.1.1 shim layer [NSE-758] Upgrade to spark 3.1.3 from 3.1.1 Mar 10, 2022
@github-actions
Copy link

#758

@PHILO-HE
Copy link
Collaborator Author

There are some compatibility issues in upgrading 3.1.1 to 3.1.3.

gazelle_plugin/shims/spark313/src/main/scala/com/intel/oap/sql/shims/spark313/Spark313Shims.scala:74: error: not enough arguments for constructor ParquetFilters: (schema: org.apache.parquet.schema.MessageType, pushDownDate: Boolean, pushDownTimestamp: Boolean, pushDownDecimal: Boolean, pushDownStartWith: Boolean, pushDownInFilterThreshold: Int, caseSensitive: Boolean, datetimeRebaseMode: org.apache.spark.sql.internal.SQLConf.LegacyBehaviorPolicy.Value)org.apache.spark.sql.execution.datasources.parquet.ParquetFilters.
[ERROR] Unspecified value parameter datetimeRebaseMode.
[ERROR]     new ParquetFilters(parquetSchema, pushDownDate, pushDownTimestamp,
[ERROR]     ^
[ERROR] one error found

@PHILO-HE
Copy link
Collaborator Author

To make gazelle workable on spark 3.1.2 & 3.1.3, we already have a quick fix in #760 by allowing loading 3.1.1 shim layer for 3.1.2 & 3.1.3 at runtime.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants