[SPARK-51348][BUILD][SQL] Upgrade Hive to 4.1 #50213
Conversation
| // Since HIVE-18238(Hive 3.0.0), the Driver.close function's return type changed | ||
| // and the CommandProcessorFactory.clean function removed. | ||
| driver.getClass.getMethod("close").invoke(driver) | ||
| if (version != hive.v3_0 && version != hive.v3_1 && version != hive.v4_0) { |
There was a problem hiding this comment.
This would break backward compatibility when spark tries to connect with older versions of Hive right?
There was a problem hiding this comment.
I don't think it is a connection problem. There will be a problem when 2.x Hive jars are provided at runtime. It is not clear to me if it is still necessary to support such option granted that 2.x is EOL.
There was a problem hiding this comment.
2.x is ancient and should no longer be supported
There was a problem hiding this comment.
Agreed.
Hive 2.x and 3.x is EoL. We should move completely to hive 4+.
Thanks!
| .map { case (k, v) => | ||
| if (v == "NULL") { | ||
| s"$k=${ConfVars.DEFAULTPARTITIONNAME.defaultStrVal}" | ||
| s"$k=${ConfVars.DEFAULTPARTITIONNAME.getDefaultVal}" |
There was a problem hiding this comment.
I think you should use org.apache.hadoop.hive.conf.HiveConf.ConfVars.DEFAULT_PARTITION_NAME
|
Hi, thank you for your work. Will this PR also address the CVE-2019-10202 as described in SPARK-30466 ? |
|
@razvan SPARK-30466 does not seem to affect Spark 4.x and this PR targets 4.x only. If you have more questions, I'd suggest to bring them to dev list and comment on the SPIP or SPARK-52408 |
vrozov
force-pushed
the
SPARK-51348
branch
2 times, most recently
from
514bd5f to
d280720
Compare
…alcite:calcite-core
vrozov
changed the title
vrozov
changed the title
|
Please see #52099 |
What changes were proposed in this pull request?
Upgrade Hive compile time dependency to 4.0.1
Why are the changes needed?
Apache Hive 1.x, 2.x and 3.x are EOL
Does this PR introduce any user-facing change?
Yes, more details to come.
How was this patch tested?
WIP
Was this patch authored or co-authored using generative AI tooling?
No