[SPARK-52408][BUILD][SQL] Upgrade Hive to 4.1 #52099

vrozov · 2025-08-22T15:49:51Z

What changes were proposed in this pull request?

Upgrade Hive dependency to 4.1.0

Why are the changes needed?

Apache Hive 1.x, 2.x and 3.x are EOL and are not longer supported by the Apache Hive community

Does this PR introduce any user-facing change?

Yes, it drops support for EOL versions of Hive 1.x, 2.x and 3.x.

There is no change in the Spark behavior and several changes on Hive 4.x compared to Hive 2.3:

Hive managed tables
Indices are not supported
ORC invalid Gregorian dates are handled differently compared to Spark and prior Hive version

How was this patch tested?

Few tests required modification due to changed Hive behavior, otherwise all existing Spark tests pass.

Was this patch authored or co-authored using generative AI tooling?

No

sarutak · 2025-08-27T00:29:16Z

@vrozov
Thank you for your contribution! I'll leave high level comments.

There seem lots of non trivial changes to upgrade Hive to 4.1 so it's better to explain what changes are needed for what purpose and the reason, and what and how did you tested. These helps reviewers.
I'm concerning whether upgrading Hive to 4.1 and dropping 2.3 causes not small impact for users. Can we switch 2.3 and 4.1 using profile (-P flag) when we build and for the time being, use 2.3 by default?
This PR is large and changes span a wide range of files. To make review easily and avoid conflict with other PRs, can we break this PR into smaller ones somehow? If we can switch 4.1 and 2.3 like I mentioned above, we don't need to complete upgrade to 4.1 at once.

@dongjoon-hyun Do you have other suggestions because you had experience to add 2.3 support.

dongjoon-hyun · 2025-08-27T02:13:14Z

To @sarutak , I have no opinion for the PR because apparently this PR didn't pass the CI yet and what I can say for now is that Apache Spark community doesn't allow this kind of regression (of this PR). I'm not sure why Hive 4.1 upgrade proposal enforces us to downgrade the dependencies. I hope it's a mistake and to make it sure that to avoid this kind of hidden stuff.

- <antlr4.version>4.13.1</antlr4.version>
+ <antlr4.version>4.9.3</antlr4.version>

sarutak · 2025-08-27T18:11:42Z

@dongjoon-hyun
I didn't intend to ask you to look into each part of changes as it didn't pass CI as you mentioned. I wanted to ask you high level comment If you had because upgrading Hive should be done carefully .

dongjoon-hyun · 2025-08-27T18:20:58Z

Sorry but I'm not using Apache Hive in these days in all cases including even a HiveMetaStore. So, it's a little hard for me to help this PR.

According to my previous experience, I can say that this PR requires significant verification efforts even after CI passes.

May I ask if @vrozov and @sarutak uses Hive 4.1 or this patch internally in the production level with AWS Glue? If then, it would be a great reference to the community.

sarutak · 2025-08-28T09:53:05Z

@dongjoon-hyun
Thank you for your comment. Yes, I agree that we need to verify very carefully so I'm wondering if we can minimize the risk by allowing to build with both 2.3(as default) and 4.1 transitionally like as we supported to build with Scala 2.12 and 2.13 before.

May I ask if @vrozov and @sarutak uses Hive 4.1 or this patch internally in the production level with AWS Glue? If then, it would be a great reference to the community.

@vrozov Can you share the background if possible?

vrozov · 2025-08-29T19:46:48Z

To @sarutak , I have no opinion for the PR because apparently this PR didn't pass the CI yet and what I can say for now is that Apache Spark community doesn't allow this kind of regression (of this PR). I'm not sure why Hive 4.1 upgrade proposal enforces us to downgrade the dependencies. I hope it's a mistake and to make it sure that to avoid this kind of hidden stuff.
- <antlr4.version>4.13.1</antlr4.version>
+ <antlr4.version>4.9.3</antlr4.version>

@dongjoon-hyun @sarutak Unfortunately it is not a mistake. Hive uses 4.9.3 that is not compatible with 4.10.x and above: "Mixing ANTLR 4.9.3 and 4.10 can lead to errors that point to a version mismatch. A very common Java error looks like this: Caused by: java.io.InvalidClassException: org.antlr.v4.runtime.atn.ATN; Could not deserialize ATN with version 3 (expected 4)." and this is exactly the error I got without changing antlr4.version.

One possible solution is to shade antlr4 in Spark, so there will be no conflict between Spark and Hive version. Please let me know what do you think and if that sounds reasonable, I'll open a separate PR for shading.

…e:calcite-core

sarutak

@vrozov
I have not yet looked into all the changes deeply but I'll leave some comments.
There seems some compatibility break in this PR. I think we should make consensus before we make critical changes.

sarutak · 2025-09-09T07:08:30Z

dev/test-dependencies.sh

 MVN="build/mvn"
 HADOOP_HIVE_PROFILES=(
-    hadoop-3-hive-2.3
+    hadoop-3-hive-4.0


sarutak · 2025-09-09T07:08:53Z

dev/test-dependencies.sh

 # Generate manifests for each Hadoop profile:
 for HADOOP_HIVE_PROFILE in "${HADOOP_HIVE_PROFILES[@]}"; do
-  if [[ $HADOOP_HIVE_PROFILE == **hadoop-3-hive-2.3** ]]; then
+  if [[ $HADOOP_HIVE_PROFILE == **hadoop-3-hive-4.0** ]]; then


sarutak · 2025-09-09T07:10:24Z

dev/test-jars.txt

 sql/core/src/test/resources/SPARK-33084.jar
 sql/core/src/test/resources/artifact-tests/udf_noA.jar
 sql/hive-thriftserver/src/test/resources/TestUDTF.jar
-sql/hive/src/test/noclasspath/hive-test-udfs.jar


Why this and related change are needed? If there is a reason, it should be explained in the PR description.

sarutak · 2025-09-09T07:15:37Z

pom.xml

    <jaxb.version>2.2.11</jaxb.version>
    <libthrift.version>0.16.0</libthrift.version>
-    <antlr4.version>4.13.1</antlr4.version>
+    <antlr4.version>4.9.3</antlr4.version>


Basically, we don't allow downgrading dependencies as @dongjoon-hyun mentioned.
If there is dependency conflict between Spark and Hive mentioned here, we should discuss first how to resolve it first.
Also, please don't make this kind of change without explanation.

sarutak · 2025-09-09T07:17:02Z

pom.xml

      -Djdk.reflect.useDirectMethodHandle=false
      -Dio.netty.tryReflectionSetAccessible=true
      --enable-native-access=ALL-UNNAMED
+      -Dmvn.executable=${maven.multiModuleProjectDirectory}/build/mvn


Why is this change needed?

sarutak · 2025-09-09T08:04:00Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala

        }
        DeserializerLock.synchronized {
-          deserializer.initialize(hconf, props)
+          deserializer.initialize(hconf, props, partProps)


Why do we need pass partProps as third parameter while null is passed to other initialize.

sarutak · 2025-09-09T08:11:03Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala


  def listFunctions(hive: Hive, db: String, pattern: String): Seq[String]

-  def dropIndex(hive: Hive, dbName: String, tableName: String, indexName: String): Unit


This kind of change seems to break compatibility.
Did you test that old hive metastores work with this change?

sarutak · 2025-09-09T08:27:03Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientVersions.scala

  protected val versions = if (testVersions.nonEmpty) {
    testVersions.get.split(",").map(_.trim).filter(_.nonEmpty).toIndexedSeq
  } else {
-    IndexedSeq("2.0", "2.1", "2.2", "2.3", "3.0", "3.1", "4.0", "4.1")


Why should we drop tests with old clients? Upgrading builtin Hive and supported metastore version should be discussed separately.

sarutak · 2025-09-09T08:28:10Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveVersionSuite.scala

    hadoopConf.set("hive.metastore.schema.verification", "false")
    // Since Hive 3.0, HIVE-19310 skipped `ensureDbInit` if `hive.in.test=false`.
-    if (version == "3.0" || version == "3.1" || version == "4.0") {
+    if (version == "3.0" || version == "3.1" || version == "4.1") {


Why is 4.0 excluded?

sarutak · 2025-09-09T08:31:54Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveVersionSuite.scala

-    if (version == "3.0" || version == "3.1" || version == "4.0") {
+    if (version == "3.0" || version == "3.1" || version == "4.1") {
      hadoopConf.set("hive.in.test", "true")
-      hadoopConf.set("hive.query.reexecution.enabled", "false")


Why is this change needed?

vrozov · 2025-10-01T13:40:38Z

@dongjoon-hyun #52490 WDYT?

github-actions bot added SQL BUILD SPARK SHELL labels Aug 22, 2025

vrozov mentioned this pull request Aug 22, 2025

[SPARK-51348][BUILD][SQL] Upgrade Hive to 4.1 #50213

Closed

vrozov added 20 commits September 1, 2025 16:12

[SPARK-52408][BUILD][SQL] Upgrade Hive to 4.1

21e9647

[SPARK-52408][BUILD][SQL] Add missing dependency on org.apache.calcit…

9f4544e

…e:calcite-core

[SPARK-52408][BUILD][SQL] More changes

26fb0a9

[SPARK-52408][BUILD][SQL] More changes

4498b8c

[SPARK-52020][TEST] Build hive-test-udfs.jar from source

964c69e

remove from test-jars.txt

b6e4bc4

[SPARK-52408][BUILD][SQL] 15 tests fails

79f3a0c

fixes for Java 17 and more

46376ce

more fixes

e7a2cb4

more fixes

53586eb

Hive 4.1.0

f9e4fe9

Revert MavenUtils

a1d3a5c

calcite dependency fix

62bd861

dependencies

3442244

dependencies and unused imports

ccd6a15

restrict to 4.1 for now

cc0e539

exclude orc-core and orc-format from SBT build

a05d777

fix

de674fb

fix

8e3ee52

exclude sqlline

a535b0a

more dependencies explicitly excluded for SBT build

4815e8e

vrozov force-pushed the SPARK-52408 branch from a8172d8 to 4815e8e Compare September 1, 2025 23:13

sarutak reviewed Sep 9, 2025

View reviewed changes

vrozov changed the title ~~[SPARK-51348][BUILD][SQL] Upgrade Hive to 4.1~~ [SPARK-52408][BUILD][SQL] Upgrade Hive to 4.1 Sep 23, 2025


		def listFunctions(hive: Hive, db: String, pattern: String): Seq[String]

		def dropIndex(hive: Hive, dbName: String, tableName: String, indexName: String): Unit

[SPARK-52408][BUILD][SQL] Upgrade Hive to 4.1 #52099

Are you sure you want to change the base?

[SPARK-52408][BUILD][SQL] Upgrade Hive to 4.1 #52099

Conversation

vrozov commented Aug 22, 2025

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

sarutak commented Aug 27, 2025

Uh oh!

dongjoon-hyun commented Aug 27, 2025

Uh oh!

sarutak commented Aug 27, 2025

Uh oh!

dongjoon-hyun commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sarutak commented Aug 28, 2025

Uh oh!

vrozov commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sarutak left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vrozov commented Oct 1, 2025

Uh oh!

Uh oh!

dongjoon-hyun commented Aug 27, 2025 •

edited

Loading

vrozov commented Aug 29, 2025 •

edited

Loading