HADOOP-16797. Add Dockerfile for ARM builds #1801

vinayakumarb · 2020-01-09T12:08:59Z

What does this change do?

Added a Dockerfile, which will be used by scripts during build and test in aarch64 platform.

Following are the differences between x86 and aarch64 Dockerfile

CMake 3.1.0 compilation from source as no binary package readily available for aarch64
Skips installation of ghc package which does not have any package for aarch64 from repo.
Skips installation of hadolint as no package is available for aarch64
x86_64 specific packages were replaced with aarch64 specific packages.
phantomjs is being installed from prebuilt package and made available in the system path to avoid failure during maven build.

Additional changes

Updated the dev-support/bin/createrelease and start-build-env.sh script to load platform specific docker files.
Updated dev-support/bin/createrelease to make use of gpg-agent for signing artifacts.

Testing ?

Yes, Tested well!!
Verified creation of release package using dev-support/bin/createrelease itself in aarch64 platform.

hadoop-yetus · 2020-01-09T12:11:20Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://builds.apache.org/job/hadoop-multibranch/job/PR-1801/1/console in case of problems.

hadoop-yetus · 2020-01-09T14:48:41Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 13s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+0 🆗	shelldocs	0m 0s	Shelldocs was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+0 🆗	mvndep	1m 15s	Maven dependency ordering for branch
+1 💚	mvninstall	18m 35s	trunk passed
+1 💚	compile	16m 43s	trunk passed
+1 💚	mvnsite	16m 14s	trunk passed
+1 💚	shadedclient	12m 39s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	7m 11s	trunk passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 32s	Maven dependency ordering for patch
+1 💚	mvninstall	19m 9s	the patch passed
+1 💚	compile	16m 21s	the patch passed
+1 💚	javac	16m 21s	the patch passed
-1 ❌	hadolint	0m 2s	The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	mvnsite	16m 15s	the patch passed
-1 ❌	shellcheck	0m 1s	The patch generated 3 new + 2 unchanged - 0 fixed = 5 total (was 2)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
-1 ❌	shadedclient	3m 13s	patch has errors when building and testing our client artifacts.
+1 💚	javadoc	7m 3s	the patch passed
		_ Other Tests _
-1 ❌	unit	17m 42s	root in the patch failed.
+1 💚	asflicense	0m 58s	The patch does not generate ASF License warnings.
		158m 34s

Subsystem	Report/Notes
Docker	Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1801/1/artifact/out/Dockerfile
GITHUB PR	#1801
Optional Tests	dupname asflicense shellcheck shelldocs hadolint compile javac javadoc mvninstall mvnsite unit shadedclient xml
uname	Linux dd0e18049ed0 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `a40dc9e`
Default Java	1.8.0_232
hadolint	https://builds.apache.org/job/hadoop-multibranch/job/PR-1801/1/artifact/out/diff-patch-hadolint.txt
shellcheck	https://builds.apache.org/job/hadoop-multibranch/job/PR-1801/1/artifact/out/diff-patch-shellcheck.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1801/1/artifact/out/patch-unit-root.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1801/1/testReport/
Max. process+thread count	1373 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs . U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1801/1/console
versions	git=2.7.4 maven=3.3.9 shellcheck=0.3.7 hadolint=1.11.1-0-g0e692dd
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

ayushtkn · 2020-01-09T17:01:30Z

dev-support/docker/Dockerfile_aarch64

+# Install hadolint
+####
+#RUN curl -L -s -S \
+#        https://github.com/hadolint/hadolint/releases/download/v1.11.1/hadolint-Linux-x86_64 \


Does the X86 one works for ARM too?

These lines are copied from x86 docker file and since there was no alternative available for aarch64 just commented the line.
This will be required when precommit builds are configured to run on aarch64.
So these lines will be ingored while creating the image.

ayushtkn · 2020-01-09T17:15:13Z

dev-support/docker/Dockerfile_aarch64

+# YETUS CUT HERE
+###
+
+# Hugo static website generator (for new hadoop site and Ozone docs)


Need not to mention ozone here.

ayushtkn · 2020-01-09T17:19:23Z

Yetus, has complains, Pls give a check.

Would it be possible for you to share one Build report from an Arm machine?

aajisaka · 2020-01-10T06:03:39Z

dev-support/docker/Dockerfile_aarch64

+    && cd /opt/cmake && rm -rf /opt/cmake/src
+ENV CMAKE_HOME /opt/cmake
+ENV PATH "${PATH}:/opt/cmake/bin"
+


Can we use cmake 3.5.1 that Xenial ships?

I am not sure about it. I will give a check.
x86 docker also uses 3.1.0 CMake itself.
Could be fixed in both places in a separate jira?

Could be fixed in both places in a separate jira?

Yes, it could be fixed in a separate jira. Thanks.

hadoop-yetus · 2020-01-10T07:16:41Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://builds.apache.org/job/hadoop-multibranch/job/PR-1801/2/console in case of problems.

vinayakumarb · 2020-01-10T08:22:04Z

@ayushtkn Here is the final output summary of the below command

$ dev-support/bin/create-release --native --rc-label=aarch64-rc1 --docker
..
..
============================================================================
============================================================================
                          Hadoop Release Creator
============================================================================
============================================================================


Version to create      : 3.3.0-SNAPSHOT
Release Candidate Label: aarch64-rc1
Source Version         : 3.3.0-SNAPSHOT

Attached the complete output.
create-release-output.log

hadoop-yetus · 2020-01-10T08:26:27Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://builds.apache.org/job/hadoop-multibranch/job/PR-1801/3/console in case of problems.

ayushtkn · 2020-01-10T15:13:34Z

Thanx @vinayakumarb seems the build got aborted, Pls check once

vinayakumarb · 2020-01-10T16:45:19Z

Yetus detected that change done in root of the project and chose to run unit tests of all modules. This needs approx 15 hours instead of current time-out of 5 hours.
So build got aborted.
But other than this every thing else should be fine.

ayushtkn

Should be fine then.
LGTM

vinayakumarb · 2020-01-13T05:11:24Z

Merged.
Thanks @ayushtkn and @aajisaka for reviews.

…ar B. (apache#1801)

HADOOP-16797. Add Dockerfile for ARM builds

994661c

ayushtkn reviewed Jan 9, 2020

View reviewed changes

aajisaka reviewed Jan 10, 2020

View reviewed changes

HADOOP-16797. Add Dockerfile for ARM builds. Addendum

1e20e9c

vinayakumarb force-pushed the HADOOP-16723/HADOOP-16797 branch from 76741e5 to 1e20e9c Compare January 10, 2020 08:24

ayushtkn approved these changes Jan 10, 2020

View reviewed changes

vinayakumarb merged commit 52b360a into apache:trunk Jan 13, 2020

RogPodge pushed a commit to RogPodge/hadoop that referenced this pull request Mar 25, 2020

HADOOP-16797. Add Dockerfile for ARM builds. Contributed by Vinayakum…

81640db

…ar B. (apache#1801)

HADOOP-16797. Add Dockerfile for ARM builds #1801

HADOOP-16797. Add Dockerfile for ARM builds #1801

Uh oh!

Conversation

vinayakumarb commented Jan 9, 2020

What does this change do?

Following are the differences between x86 and aarch64 Dockerfile

Additional changes

Testing ?

Uh oh!

hadoop-yetus commented Jan 9, 2020

Uh oh!

hadoop-yetus commented Jan 9, 2020

Uh oh!

ayushtkn Jan 9, 2020

Choose a reason for hiding this comment

Uh oh!

vinayakumarb Jan 10, 2020

Choose a reason for hiding this comment

Uh oh!

ayushtkn Jan 9, 2020

Choose a reason for hiding this comment

Uh oh!

vinayakumarb Jan 10, 2020

Choose a reason for hiding this comment

Uh oh!

ayushtkn commented Jan 9, 2020

Uh oh!

aajisaka Jan 10, 2020

Choose a reason for hiding this comment

Uh oh!

vinayakumarb Jan 10, 2020

Choose a reason for hiding this comment

Uh oh!

aajisaka Jan 10, 2020

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Jan 10, 2020

Uh oh!

vinayakumarb commented Jan 10, 2020

Uh oh!

hadoop-yetus commented Jan 10, 2020

Uh oh!

ayushtkn commented Jan 10, 2020

Uh oh!

vinayakumarb commented Jan 10, 2020

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

vinayakumarb commented Jan 13, 2020

Uh oh!

Uh oh!