[Do not commit] Modularize docker images - debug #3077

GauthamBanasandra · 2021-06-04T17:03:50Z

Debugging why the CI gets stuck.

hadoop-yetus · 2021-06-04T17:06:29Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3077/1/console in case of problems.

hadoop-yetus · 2021-06-04T17:07:13Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	docker	0m 41s		Docker failed to build yetus/hadoop:ad923ad5642.

Subsystem	Report/Notes
GITHUB PR	#3077
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3077/1/console
versions	git=2.17.1
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

* Added the necessary permissions to the files in pkg-resolver.

* Formatted the file and renamed it.

hadoop-yetus · 2021-06-04T17:13:49Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3077/2/console in case of problems.

hadoop-yetus · 2021-06-04T18:46:28Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	21m 50s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 50s		Maven dependency ordering for branch
+1 💚	mvninstall	24m 22s		trunk passed
+1 💚	mvnsite	0m 0s		trunk passed
+1 💚	shadedclient	16m 8s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 25s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 0s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-1 ❌	hadolint	0m 5s	/results-hadolint.txt	The patch generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0)
+1 💚	mvnsite	0m 0s		the patch passed
-1 ❌	pylint	0m 3s	/results-pylint.txt	The patch generated 10 new + 0 unchanged - 0 fixed = 10 total (was 0)
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	shadedclient	15m 53s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	asflicense	0m 29s	/results-asflicense.txt	The patch generated 3 ASF License warnings.
		94m 0s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3077/2/artifact/out/Dockerfile
GITHUB PR	#3077
Optional Tests	dupname asflicense codespell hadolint shellcheck shelldocs mvnsite unit pylint
uname	Linux a89759c7f570 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ee688f7`
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3077/2/testReport/
Max. process+thread count	564 (vs. ulimit of 5500)
modules	C: U:
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3077/2/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0 hadolint=1.11.1-0-g0e692dd pylint=2.6.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

* Added necessary warning suppressions. * Fixed warnings reported by pylint.

* Added a trailing / for continuation of the command.

* Added a quote as per the reported warning in CI run to prevent the splitting of word.

* Quoting $(...) causes the command execution to fail. Thus, ignoring the warning instead.

hadoop-yetus · 2021-06-05T04:58:44Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3077/3/console in case of problems.

hadoop-yetus · 2021-06-05T06:26:52Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	20m 1s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 43s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 52s		trunk passed
+1 💚	mvnsite	0m 0s		trunk passed
+1 💚	shadedclient	15m 30s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 22s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 0s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	hadolint	0m 5s		No new issues.
+1 💚	mvnsite	0m 0s		the patch passed
+1 💚	pylint	0m 2s		No new issues.
+1 💚	shellcheck	0m 1s		No new issues.
+1 💚	shadedclient	15m 34s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	asflicense	0m 29s	/results-asflicense.txt	The patch generated 3 ASF License warnings.
		89m 30s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3077/3/artifact/out/Dockerfile
GITHUB PR	#3077
Optional Tests	dupname asflicense codespell hadolint shellcheck shelldocs mvnsite unit pylint
uname	Linux 5078b0569e4f 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `2513a12`
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3077/3/testReport/
Max. process+thread count	518 (vs. ulimit of 5500)
modules	C: U:
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3077/3/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0 hadolint=1.11.1-0-g0e692dd pylint=2.6.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

* Using pkg-resolver for resolving the dependencies on Ubuntu Focal arch64, Centos 7 and Centos 8.

* The centos-release-scl package must be installed as a prerequisite before installing devtoolset-9 for gcc.

* Intel ISA-L library has dependencies on yasm, shasum which needs to be installed prior to installing it.

* Moved the build and installation of separate packages into individual scripts.

* Forgot to negate the platform check which caused a bug.

* shasum must be installed through other deps.

hadoop-yetus · 2021-06-05T06:58:41Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3077/4/console in case of problems.

hadoop-yetus · 2021-06-05T08:27:05Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	20m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 37s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 48s		trunk passed
+1 💚	mvnsite	0m 0s		trunk passed
+1 💚	shadedclient	15m 47s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 21s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 0s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	hadolint	0m 9s		No new issues.
+1 💚	mvnsite	0m 0s		the patch passed
+1 💚	pylint	0m 3s		No new issues.
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	shadedclient	15m 17s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	asflicense	0m 28s	/results-asflicense.txt	The patch generated 3 ASF License warnings.
		89m 47s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3077/4/artifact/out/Dockerfile
GITHUB PR	#3077
Optional Tests	dupname asflicense codespell hadolint shellcheck shelldocs pylint mvnsite unit
uname	Linux 65f0fefc1d51 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `191368f`
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3077/4/testReport/
Max. process+thread count	604 (vs. ulimit of 5500)
modules	C: U:
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3077/4/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0 hadolint=1.11.1-0-g0e692dd pylint=2.6.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

* Moved the packages' installation to separate scripts.

* Replaced explicit installation of packages with script invocations.

* Replaced explicit installations of packages with script invocations.

hadoop-yetus · 2021-06-05T09:14:22Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3077/5/console in case of problems.

hadoop-yetus · 2021-06-05T10:24:38Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 7s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	13m 3s		Maven dependency ordering for branch
+1 💚	mvninstall	23m 4s		trunk passed
+1 💚	mvnsite	0m 0s		trunk passed
+1 💚	shadedclient	15m 47s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 22s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 0s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	hadolint	0m 9s		No new issues.
+1 💚	mvnsite	0m 0s		the patch passed
+1 💚	pylint	0m 3s		No new issues.
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	shadedclient	15m 44s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	asflicense	0m 29s	/results-asflicense.txt	The patch generated 3 ASF License warnings.
		71m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3077/5/artifact/out/Dockerfile
GITHUB PR	#3077
Optional Tests	dupname asflicense codespell hadolint shellcheck shelldocs pylint mvnsite unit
uname	Linux c5d6a35d8ad7 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `325b2dd`
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3077/5/testReport/
Max. process+thread count	522 (vs. ulimit of 5500)
modules	C: U:
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3077/5/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0 hadolint=1.11.1-0-g0e692dd pylint=2.6.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

* Added a readme file for describing the package management module.

hadoop-yetus · 2021-06-05T10:30:38Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3077/6/console in case of problems.

hadoop-yetus · 2021-06-05T11:39:07Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 50s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 34s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 36s		trunk passed
+1 💚	mvnsite	0m 0s		trunk passed
+1 💚	shadedclient	15m 26s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 23s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 0s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	hadolint	0m 8s		No new issues.
+1 💚	mvnsite	0m 0s		the patch passed
+1 💚	pylint	0m 3s		No new issues.
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	shadedclient	15m 28s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	asflicense	0m 29s	/results-asflicense.txt	The patch generated 4 ASF License warnings.
		69m 53s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3077/6/artifact/out/Dockerfile
GITHUB PR	#3077
Optional Tests	dupname asflicense codespell hadolint shellcheck shelldocs pylint mvnsite unit markdownlint
uname	Linux 869c765bb1cf 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `2169f81`
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3077/6/testReport/
Max. process+thread count	566 (vs. ulimit of 5500)
modules	C: U:
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3077/6/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0 hadolint=1.11.1-0-g0e692dd pylint=2.6.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

* RAT check fails on JSON files since ASF license header can't be added, since there's no comment in JSON file. Thus, we're excluding JSON files from RAT checks.

hadoop-yetus · 2021-06-05T11:43:29Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3077/7/console in case of problems.

HADOOP-17727. Modularize docker images

777700e

GauthamBanasandra added 3 commits June 4, 2021 22:40

Add license headers

456cc76

Add the required permissions

e0f99d6

* Added the necessary permissions to the files in pkg-resolver.

Refactor install-common-pkgs

ee688f7

* Formatted the file and renamed it.

GauthamBanasandra added 4 commits June 5, 2021 10:23

Fix warnings reported in CI run

a229946

* Added necessary warning suppressions. * Fixed warnings reported by pylint.

Add trailing slash

ecc6dbf

* Added a trailing / for continuation of the command.

Add quotes to prevent word split

43a7946

* Added a quote as per the reported warning in CI run to prevent the splitting of word.

Unquote and ignore warning

2513a12

* Quoting $(...) causes the command execution to fail. Thus, ignoring the warning instead.

GauthamBanasandra added 6 commits June 5, 2021 12:22

Use pkg-resolver for other platforms

d58e295

* Using pkg-resolver for resolving the dependencies on Ubuntu Focal arch64, Centos 7 and Centos 8.

Install centos-release-scl as prerequisite

692169f

* The centos-release-scl package must be installed as a prerequisite before installing devtoolset-9 for gcc.

Fix installation of Intel ISA-L library

50eff69

* Intel ISA-L library has dependencies on yasm, shasum which needs to be installed prior to installing it.

Add checks for package installation

701ef5e

* Moved the build and installation of separate packages into individual scripts.

Fix negation bug

a9bf9e2

* Forgot to negate the platform check which caused a bug.

Remove shasum for Ubuntu

191368f

* shasum must be installed through other deps.

GauthamBanasandra added 3 commits June 5, 2021 14:39

Modularize Dockerfile for Centos 7

e0a5eca

* Moved the packages' installation to separate scripts.

Modularize Dockerfile for Centos 8

8728912

* Replaced explicit installation of packages with script invocations.

Modularize Dockerfile for aarch64

325b2dd

* Replaced explicit installations of packages with script invocations.

GauthamBanasandra added 2 commits June 5, 2021 15:56

Add README.md

d97a07a

* Added a readme file for describing the package management module.

Trigger CI

2169f81

Exclude JSON from RAT check

49e578d

* RAT check fails on JSON files since ASF license header can't be added, since there's no comment in JSON file. Thus, we're excluding JSON files from RAT checks.

GauthamBanasandra closed this Jun 7, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Do not commit] Modularize docker images - debug #3077

[Do not commit] Modularize docker images - debug #3077

Uh oh!

GauthamBanasandra commented Jun 4, 2021

Uh oh!

hadoop-yetus commented Jun 4, 2021

Uh oh!

hadoop-yetus commented Jun 4, 2021

Uh oh!

hadoop-yetus commented Jun 4, 2021

Uh oh!

hadoop-yetus commented Jun 4, 2021

Uh oh!

hadoop-yetus commented Jun 5, 2021

Uh oh!

hadoop-yetus commented Jun 5, 2021

Uh oh!

hadoop-yetus commented Jun 5, 2021

Uh oh!

hadoop-yetus commented Jun 5, 2021

Uh oh!

hadoop-yetus commented Jun 5, 2021

Uh oh!

hadoop-yetus commented Jun 5, 2021

Uh oh!

hadoop-yetus commented Jun 5, 2021

Uh oh!

hadoop-yetus commented Jun 5, 2021

Uh oh!

hadoop-yetus commented Jun 5, 2021

Uh oh!

Uh oh!

[Do not commit] Modularize docker images - debug #3077

[Do not commit] Modularize docker images - debug #3077

Uh oh!

Conversation

GauthamBanasandra commented Jun 4, 2021

Uh oh!

hadoop-yetus commented Jun 4, 2021

Uh oh!

hadoop-yetus commented Jun 4, 2021

Uh oh!

hadoop-yetus commented Jun 4, 2021

Uh oh!

hadoop-yetus commented Jun 4, 2021

Uh oh!

hadoop-yetus commented Jun 5, 2021

Uh oh!

hadoop-yetus commented Jun 5, 2021

Uh oh!

hadoop-yetus commented Jun 5, 2021

Uh oh!

hadoop-yetus commented Jun 5, 2021

Uh oh!

hadoop-yetus commented Jun 5, 2021

Uh oh!

hadoop-yetus commented Jun 5, 2021

Uh oh!

hadoop-yetus commented Jun 5, 2021

Uh oh!

hadoop-yetus commented Jun 5, 2021

Uh oh!

hadoop-yetus commented Jun 5, 2021

Uh oh!

Uh oh!