HADOOP-19065. Improve Protocol Buffers installation #6526

huangzhaobo99 · 2024-02-03T09:04:01Z

Description of PR

JIRA: https://issues.apache.org/jira/browse/HADOOP-19065

When downloading 'Protocol Buffers', there is no progress bar, Not very friendly for user.
The '-s' param is Silent mode, and if the network is blocked, without prompt, Not very friendly either.
Now, remove the -S -s param, Consistent with the install steps of CMake 3.19, concise and easy to understand.
reference: CMake 3.19

  CMake 3.19
  $ curl -L https://cmake.org/files/v3.19/cmake-3.19.0.tar.gz > cmake-3.19.0.tar.gz
  $ tar -zxvf cmake-3.19.0.tar.gz && cd cmake-3.19.0
  $ ./bootstrap
  $ make -j$(nproc)
  $ sudo make install

Additional: Update all versions of protobuf to 3.21.12, We have started using protobuf 3.21 by HADOOP-19069.

hadoop-yetus · 2024-02-03T10:37:59Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 48s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+1 💚	shadedclient	47m 8s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	shadedclient	39m 18s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 43s		The patch does not generate ASF License warnings.
		92m 37s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/1/artifact/out/Dockerfile
GITHUB PR	#6526
Optional Tests	dupname asflicense codespell detsecrets
uname	Linux 0f6f6c4adbb3 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `88d2d8c`
Max. process+thread count	531 (vs. ulimit of 5500)
modules	C: . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/1/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2024-02-03T14:33:56Z

someone who knows curl better than me will need to review this

huangzhaobo99 · 2024-02-19T11:59:30Z

Hi @tasanuma Can you help me review it? Thanks.

Thanks @steveloughran.

tasanuma · 2024-02-21T14:11:59Z

Although I'm not an expert in curl, I believe this change is fine as it aligns with the instruction for CMake. However, in the first place, protobuf 3.7.1 could be outdated. We have started using protobuf 3.21 by HADOOP-19069.

huangzhaobo99 · 2024-02-21T14:28:48Z

Although I'm not an expert in curl, I believe this change is fine as it aligns with the instruction for CMake. However, in the first place, protobuf 3.7.1 could be outdated. We have started using protobuf 3.21 by HADOOP-19069.

@tasanuma Thanks for your reply and review. After the updated version was tested locally, it can be downloaded normally.

tasanuma · 2024-02-21T15:00:35Z

@huangzhaobo99 Thanks for updating it, but there are still many mentions of Protocol Buffers 3.7.1 in BUILDING.txt. If we want to update all them to version 3.21.12, we firstly need to update the Dockerfile and install-protobuf.sh to install protobuf 3.21.12, and we must make sure that Hadoop can still be built successfully.

huangzhaobo99 · 2024-02-21T15:05:39Z

@tasanuma Thanks for the reminder. I will complete it after.

hadoop-yetus · 2024-02-21T15:51:09Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 39s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+1 💚	shadedclient	41m 24s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	shadedclient	34m 51s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		82m 39s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/2/artifact/out/Dockerfile
GITHUB PR	#6526
Optional Tests	dupname asflicense codespell detsecrets
uname	Linux 4adb15f13af3 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `f470b40`
Max. process+thread count	712 (vs. ulimit of 5500)
modules	C: . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/2/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

huangzhaobo99 · 2024-02-21T15:52:31Z

@tasanuma Have completed, Whether or not need to modify the PR title?

Currently, The description of PR has been modified at point 5.

hadoop-yetus · 2024-02-21T15:53:16Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/3/console in case of problems.

hadoop-yetus · 2024-02-21T16:06:19Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	docker	13m 2s		Docker failed to build run-specific yetus/hadoop:tp-26055}.

Subsystem	Report/Notes
GITHUB PR	#6526
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/3/console
versions	git=2.34.1
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-02-21T16:31:46Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/4/console in case of problems.

hadoop-yetus · 2024-02-21T17:33:41Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	52m 45s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	hadolint	0m 0s		hadolint was not available.
+0 🆗	shellcheck	0m 0s		Shellcheck was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
-1 ❌	mvninstall	4m 31s	/branch-mvninstall-root.txt	root in trunk failed.
-1 ❌	mvnsite	0m 9s	/branch-mvnsite-root.txt	root in trunk failed.
-1 ❌	shadedclient	6m 10s		branch has errors when building and testing our client artifacts.
			_ Patch Compile Tests _
-1 ❌	mvninstall	0m 29s	/patch-mvninstall-root.txt	root in the patch failed.
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-1 ❌	mvnsite	0m 24s	/patch-mvnsite-root.txt	root in the patch failed.
+1 💚	shadedclient	1m 18s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	0m 24s	/patch-unit-root.txt	root in the patch failed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		67m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/4/artifact/out/Dockerfile
GITHUB PR	#6526
Optional Tests	dupname asflicense codespell detsecrets hadolint shellcheck shelldocs mvnsite unit
uname	Linux 4f263dea2f4f 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `b0f9f31`
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/4/testReport/
Max. process+thread count	56 (vs. ulimit of 5500)
modules	C: . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/4/console
versions	git=2.9.5 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-02-21T18:30:31Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/5/console in case of problems.

hadoop-yetus · 2024-02-21T18:37:18Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 45s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	hadolint	0m 0s		hadolint was not available.
+0 🆗	shellcheck	0m 0s		Shellcheck was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
-1 ❌	mvninstall	0m 24s	/branch-mvninstall-root.txt	root in trunk failed.
-1 ❌	mvnsite	0m 24s	/branch-mvnsite-root.txt	root in trunk failed.
-1 ❌	shadedclient	2m 14s		branch has errors when building and testing our client artifacts.
			_ Patch Compile Tests _
-1 ❌	mvninstall	0m 9s	/patch-mvninstall-root.txt	root in the patch failed.
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-1 ❌	mvnsite	0m 29s	/patch-mvnsite-root.txt	root in the patch failed.
-1 ❌	shadedclient	2m 50s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	0m 24s	/patch-unit-root.txt	root in the patch failed.
+0 🆗	asflicense	0m 25s		ASF License check generated no output?
		11m 52s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/5/artifact/out/Dockerfile
GITHUB PR	#6526
Optional Tests	dupname asflicense codespell detsecrets hadolint shellcheck shelldocs mvnsite unit
uname	Linux e1a4968923ae 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `62390b2`
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/5/testReport/
Max. process+thread count	51 (vs. ulimit of 5500)
modules	C: . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/5/console
versions	git=2.9.5 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

huangzhaobo99 · 2024-02-22T02:41:22Z

Pipeline failed, but the local env launched docker image was successfully built and compiled. cc @tasanuma

Install Docker and run this command:

./start-build-env.sh

trigger compile:

xxx@886b0e5bc37b:~/hadoop$ mvn package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [  3.984 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [  0.016 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  16:56 min
[INFO] Finished at: 2024-02-22T02:39:26Z
[INFO] ------------------------------------------------------------------------

check docker env protobuf version:

xxx@886b0e5bc37b:~/hadoop$ protoc --version

libprotoc 3.21.12

hadoop-yetus · 2024-02-22T12:56:42Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/6/console in case of problems.

hadoop-yetus · 2024-02-22T16:13:06Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 35s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	hadolint	0m 0s		hadolint was not available.
+0 🆗	shellcheck	0m 0s		Shellcheck was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	44m 53s		trunk passed
+1 💚	mvnsite	19m 31s		trunk passed
+1 💚	shadedclient	99m 9s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	30m 56s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	16m 3s		the patch passed
+1 💚	shadedclient	48m 1s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	16m 12s		root in the patch passed.
+1 💚	asflicense	1m 5s		The patch does not generate ASF License warnings.
		200m 22s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/6/artifact/out/Dockerfile
GITHUB PR	#6526
Optional Tests	dupname asflicense codespell detsecrets hadolint shellcheck shelldocs mvnsite unit
uname	Linux ee637f0b52b8 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `62390b2`
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/6/testReport/
Max. process+thread count	560 (vs. ulimit of 5500)
modules	C: . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6526/6/console
versions	git=2.9.5 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran

LGTM
+1

…#6526) Instructions and docker script for installing protobuf updated from 3.7.1 to 3.21.12 Contributed by huangzhaobo

slfan1989 · 2024-02-23T05:20:14Z

@huangzhaobo99 Do you have any actual environment to use these dockerfiles? This involves many operating systems, and I am not sure whether it can really be executed normally.

I will try to use this change on the Centos7.9 I prepared, including x86 & arm processors, and hope it goes well.

cc: @tasanuma @steveloughran

huangzhaobo99 · 2024-02-23T05:23:43Z

@huangzhaobo99 Do you have any actual environment to use these dockerfiles? This involves many operating systems, and I am not sure whether it can really be executed normally.

cc: @tasanuma @steveloughran

@slfan1989 It has been verified, including manual installation.

slfan1989 · 2024-02-23T05:34:14Z

@huangzhaobo99 Do you have any actual environment to use these dockerfiles? This involves many operating systems, and I am not sure whether it can really be executed normally.
cc: @tasanuma @steveloughran

@slfan1989 It has been verified, including manual installation.

Thanks for the feedback, this is good news.

steveloughran · 2024-02-23T15:57:50Z

@slfan1989 docker worked on the 3.3.6 release for me with macbook m1 for arm and AMD x86 on EC2 for x86 release. doing that in US-west-2 built pretty fast as that is where the maven central repo is

github-actions bot added the trunk label Feb 3, 2024

HADOOP-19065. Improve Protocol Buffers install instructions.

bd494fe

huangzhaobo99 force-pushed the HADOOP-19065 branch from f470b40 to bd494fe Compare February 21, 2024 15:47

github-actions bot added the Infra label Feb 21, 2024

fix

b0f9f31

fix

62390b2

steveloughran approved these changes Feb 22, 2024

View reviewed changes

steveloughran changed the title ~~HADOOP-19065. Improve Protocol Buffers install instructions.~~ HADOOP-19065. Improve Protocol Buffers installation Feb 22, 2024

steveloughran merged commit 6dfebdb into apache:trunk Feb 22, 2024

huangzhaobo99 deleted the HADOOP-19065 branch February 23, 2024 02:52

slfan1989 mentioned this pull request Feb 23, 2024

HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. #6537

Merged

4 tasks

steveloughran mentioned this pull request Mar 1, 2024

Commit to remove vulnerabilities raised for: HADOOP-19074 #6586

Closed

4 tasks

HADOOP-19065. Improve Protocol Buffers installation #6526

HADOOP-19065. Improve Protocol Buffers installation #6526

Uh oh!

Conversation

huangzhaobo99 commented Feb 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Uh oh!

hadoop-yetus commented Feb 3, 2024

Uh oh!

steveloughran commented Feb 3, 2024

Uh oh!

huangzhaobo99 commented Feb 19, 2024

Uh oh!

tasanuma commented Feb 21, 2024

Uh oh!

huangzhaobo99 commented Feb 21, 2024

Uh oh!

tasanuma commented Feb 21, 2024

Uh oh!

huangzhaobo99 commented Feb 21, 2024

Uh oh!

hadoop-yetus commented Feb 21, 2024

Uh oh!

huangzhaobo99 commented Feb 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hadoop-yetus commented Feb 21, 2024

Uh oh!

hadoop-yetus commented Feb 21, 2024

Uh oh!

hadoop-yetus commented Feb 21, 2024

Uh oh!

hadoop-yetus commented Feb 21, 2024

Uh oh!

hadoop-yetus commented Feb 21, 2024

Uh oh!

hadoop-yetus commented Feb 21, 2024

Uh oh!

huangzhaobo99 commented Feb 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hadoop-yetus commented Feb 22, 2024

Uh oh!

hadoop-yetus commented Feb 22, 2024

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

slfan1989 commented Feb 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

huangzhaobo99 commented Feb 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

slfan1989 commented Feb 23, 2024

Uh oh!

steveloughran commented Feb 23, 2024

Uh oh!

Uh oh!

huangzhaobo99 commented Feb 3, 2024 •

edited

Loading

huangzhaobo99 commented Feb 21, 2024 •

edited

Loading

huangzhaobo99 commented Feb 22, 2024 •

edited

Loading

slfan1989 commented Feb 23, 2024 •

edited

Loading

huangzhaobo99 commented Feb 23, 2024 •

edited

Loading