[SPARK-50107][SQL][BUILD] Upgrade `protobuf-java` to 4.28.3 #48646

dongjoon-hyun · 2024-10-24T21:18:41Z

What changes were proposed in this pull request?

This PR aims to upgrade protobuf-java to 4.28.3 for Apache Spark 4.0.0 on February 2025.

Why are the changes needed?

protobuf-java 4.28.3 is the latest version which is released two days ago (Oct 23, 2024).

https://mvnrepository.com/artifact/com.google.protobuf/protobuf-java/4.28.3

This is a part of v28.3.

https://github.com/protocolbuffers/protobuf/releases/tag/v28.3 (2024-10-22)

Note that v26 introduced a breaking change to Java API last year.

https://protobuf.dev/news/2023-12-05/#breaking-compatibility-with-old-generated-code

v26.x will break compatibility with generated code from older major versions. Users should regenerate old generated code to be from the same version.
For example, GeneratedMessageV3, which was originally introduced for backwards compatibility with generated code from v2.x.x against v3.x.x runtime, will be renamed to GeneratedMessage. Runtimes will be updated to support Editions, which will not be compatible with old generated code.

Does this PR introduce any user-facing change?

How was this patch tested?

Pass the CIs.

Was this patch authored or co-authored using generative AI tooling?

No.

dongjoon-hyun · 2024-10-24T21:26:13Z

To @HyukjinKwon and @grundprinzip , WDYT?

I believe Apache Spark 4 is the last chance to take this kind of change because it's hard to image Apache Spark 5 in next 4 years.

dongjoon-hyun · 2024-10-25T08:25:23Z

All tests passed.

HyukjinKwon · 2024-10-25T10:55:51Z

cc @hvanhovell too

LuciferYang · 2024-10-25T13:11:00Z

sql/core/src/main/java/org/apache/spark/sql/execution/streaming/state/StateMessage.java

@@ -217,12 +217,12 @@ public interface StateRequestOrBuilder extends
   * Protobuf type {@code org.apache.spark.sql.execution.streaming.state.StateRequest}
   */
  public static final class StateRequest extends
-      com.google.protobuf.GeneratedMessageV3 implements
+      com.google.protobuf.GeneratedMessage implements


I submitted a PR to change to use maven/sbt plugin to generation this file: #48654

grundprinzip · 2024-10-25T13:13:48Z

I think that makes sense, is this a recent version or do we need to upgrade even more?

dongjoon-hyun · 2024-10-25T14:02:34Z

To @grundprinzip , this is the latest version which is released two days ago (Oct 23, 2024).

https://mvnrepository.com/artifact/com.google.protobuf/protobuf-java/4.28.3

To @LuciferYang , thanks. Let's review and handle your PR after merging this.

dongjoon-hyun · 2024-10-25T14:06:17Z

Thank you, @HyukjinKwon , @LuciferYang , @grundprinzip .

Merged to master for Apache Spark 4.0.0 on February 2025.

LuciferYang · 2024-10-28T03:30:38Z

@dongjoon-hyun It seems that after this pr, the spark-protobuf module will fail to compile using Maven:

Before:

git reset --hard b0dc6546a0b720f86ddbaa87e4d055557160af54
build/mvn -DskipTests clean install -pl connector/protobuf -e -am

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 4.0.0-SNAPSHOT:
[INFO] 
[INFO] Spark Project Parent POM ........................... SUCCESS [  2.854 s]
[INFO] Spark Project Tags ................................. SUCCESS [  5.257 s]
[INFO] Spark Project Sketch ............................... SUCCESS [  5.023 s]
[INFO] Spark Project Common Utils ......................... SUCCESS [ 16.851 s]
[INFO] Spark Project Local DB ............................. SUCCESS [  8.222 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 11.172 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  9.049 s]
[INFO] Spark Project Variant .............................. SUCCESS [  3.260 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 10.125 s]
[INFO] Spark Project Connect Shims ........................ SUCCESS [  2.640 s]
[INFO] Spark Project Launcher ............................. SUCCESS [  6.882 s]
[INFO] Spark Project Core ................................. SUCCESS [01:43 min]
[INFO] Spark Project SQL API .............................. SUCCESS [ 29.428 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [02:02 min]
[INFO] Spark Project SQL .................................. SUCCESS [02:33 min]
[INFO] Spark Protobuf ..................................... SUCCESS [ 24.383 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  08:34 min
[INFO] Finished at: 2024-10-28T11:12:10+08:00
[INFO] ------------------------------------------------------------------------

After:

git reset --hard b0dc6546a0b720f86ddbaa87e4d055557160af54
build/mvn -DskipTests clean install -pl connector/protobuf -e -am

[INFO] --- protoc-jar:3.11.4:run (default) @ spark-protobuf_2.13 ---
[INFO] Resolving artifact: com.google.protobuf:protoc:4.28.3, platform: osx-aarch_64
protoc-jar: executing: [/var/folders/j2/cfn7w6795538n_416_27rkqm0000gn/T/protoc17161577217517635203.exe, --version]
libprotoc 28.3
[INFO] Protoc command: /var/folders/j2/cfn7w6795538n_416_27rkqm0000gn/T/protoc17161577217517635203.exe
[INFO] Additional include types: /var/folders/j2/cfn7w6795538n_416_27rkqm0000gn/T/protocjar1650670058665477047/include
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 4.0.0-SNAPSHOT:
[INFO] 
[INFO] Spark Project Parent POM ........................... SUCCESS [  5.220 s]
[INFO] Spark Project Tags ................................. SUCCESS [  5.724 s]
[INFO] Spark Project Sketch ............................... SUCCESS [  6.586 s]
[INFO] Spark Project Common Utils ......................... SUCCESS [ 17.162 s]
[INFO] Spark Project Local DB ............................. SUCCESS [  7.086 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 13.517 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  8.659 s]
[INFO] Spark Project Variant .............................. SUCCESS [  3.383 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [  9.294 s]
[INFO] Spark Project Connect Shims ........................ SUCCESS [  2.865 s]
[INFO] Spark Project Launcher ............................. SUCCESS [  5.692 s]
[INFO] Spark Project Core ................................. SUCCESS [01:39 min]
[INFO] Spark Project SQL API .............................. SUCCESS [ 26.080 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [01:56 min]
[INFO] Spark Project SQL .................................. SUCCESS [02:27 min]
[INFO] Spark Protobuf ..................................... FAILURE [  7.295 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  08:02 min
[INFO] Finished at: 2024-10-28T11:20:52+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (default) on project spark-protobuf_2.13: Execution default of goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run failed: Cannot read the array length because "<local6>" is null -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (default) on project spark-protobuf_2.13: Execution default of goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run failed: Cannot read the array length because "<local6>" is null
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:333)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:569)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:255)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:201)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:361)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:314)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution default of goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run failed: Cannot read the array length because "<local6>" is null
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:133)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:569)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:255)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:201)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:361)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:314)
Caused by: java.lang.NullPointerException: Cannot read the array length because "<local6>" is null
    at com.github.os72.protocjar.Protoc.extractStdTypes (Protoc.java:372)
    at com.github.os72.protocjar.maven.ProtocJarMojo.performProtoCompilation (ProtocJarMojo.java:397)
    at com.github.os72.protocjar.maven.ProtocJarMojo.execute (ProtocJarMojo.java:374)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:569)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:255)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:201)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:361)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:314)
[ERROR] 
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :spark-protobuf_2.13

The Maven daily tests for the past two days have all failed to compile, and I haven't found the root cause yet:

dongjoon-hyun · 2024-10-28T03:39:36Z

Thank you for reporting.

Let me check, @LuciferYang .

dongjoon-hyun · 2024-10-28T03:41:01Z

According to the log message string local6, it seems Maven plugin issue, protoc-jar-maven-plugin, because SBT works fine.

NPE while executing plugin on unchanged proto files os72/protoc-jar-maven-plugin#104

dongjoon-hyun · 2024-10-28T03:41:44Z

Let me try the workaround, optimizeCodegen=true, in the above link.

dongjoon-hyun · 2024-10-28T03:52:56Z

Here is the PR, @LuciferYang .

[SPARK-50143][BUILD] Fix protobuf module Maven compilation #48673

[SPARK-50107][SQL][BUILD] Upgrade protobuf-java to 4.28.3

2910b85

github-actions bot added SQL STRUCTURED STREAMING BUILD CONNECT labels Oct 24, 2024

dongjoon-hyun marked this pull request as ready for review October 25, 2024 03:22

HyukjinKwon approved these changes Oct 25, 2024

View reviewed changes

LuciferYang reviewed Oct 25, 2024

View reviewed changes

dongjoon-hyun closed this in 250f8af Oct 25, 2024

dongjoon-hyun deleted the SPARK-50107 branch October 25, 2024 14:06

dongjoon-hyun mentioned this pull request Oct 25, 2024

[SPARK-50015][BUILD] Upgrade grpcio* to 1.67.0 and grpc-java to 1.67.1 #48524

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-50107][SQL][BUILD] Upgrade `protobuf-java` to 4.28.3 #48646

[SPARK-50107][SQL][BUILD] Upgrade `protobuf-java` to 4.28.3 #48646

dongjoon-hyun commented Oct 24, 2024 •

edited

Loading

dongjoon-hyun commented Oct 24, 2024

dongjoon-hyun commented Oct 25, 2024

HyukjinKwon commented Oct 25, 2024

LuciferYang Oct 25, 2024 •

edited

Loading

grundprinzip commented Oct 25, 2024

dongjoon-hyun commented Oct 25, 2024

dongjoon-hyun commented Oct 25, 2024

LuciferYang commented Oct 28, 2024 •

edited

Loading

dongjoon-hyun commented Oct 28, 2024

dongjoon-hyun commented Oct 28, 2024

dongjoon-hyun commented Oct 28, 2024

dongjoon-hyun commented Oct 28, 2024 •

edited

Loading

[SPARK-50107][SQL][BUILD] Upgrade protobuf-java to 4.28.3 #48646

[SPARK-50107][SQL][BUILD] Upgrade protobuf-java to 4.28.3 #48646

Conversation

dongjoon-hyun commented Oct 24, 2024 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

dongjoon-hyun commented Oct 24, 2024

dongjoon-hyun commented Oct 25, 2024

HyukjinKwon commented Oct 25, 2024

LuciferYang Oct 25, 2024 • edited Loading

Choose a reason for hiding this comment

grundprinzip commented Oct 25, 2024

dongjoon-hyun commented Oct 25, 2024

dongjoon-hyun commented Oct 25, 2024

LuciferYang commented Oct 28, 2024 • edited Loading

dongjoon-hyun commented Oct 28, 2024

dongjoon-hyun commented Oct 28, 2024

dongjoon-hyun commented Oct 28, 2024

dongjoon-hyun commented Oct 28, 2024 • edited Loading

[SPARK-50107][SQL][BUILD] Upgrade `protobuf-java` to 4.28.3 #48646

[SPARK-50107][SQL][BUILD] Upgrade `protobuf-java` to 4.28.3 #48646

dongjoon-hyun commented Oct 24, 2024 •

edited

Loading

LuciferYang Oct 25, 2024 •

edited

Loading

LuciferYang commented Oct 28, 2024 •

edited

Loading

dongjoon-hyun commented Oct 28, 2024 •

edited

Loading