HADOOP-18305. Release Hadoop 3.3.4: minor update of hadoop-3.3.3

[steveloughran]

This is a cherrypick from branch-3.3 of security related changes applied to hadoop branch-3.3.

this lets yetus check they are all good

Tests known to fail

This branch isn't going to worry about test failures which have been fixed in PRs which we aren't going to backport.

Instead it is going to track what those known failures are, to help triage them.

Test	Fix
yarn.applications.distributedshell.TestDistributedShell	YARN-10553
yarn.csi.client.TestCsiClient	YARN-10788
yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation	YARN-10506
hdfs.server.datanode.TestDataNodeRollingUpgrade	HDFS-15580
fs.s3a.tools.ITestMarkerTool	HADOOP-18168

[steveloughran]

four test failures

hadoop.yarn.applications.distributedshell.TestDistributedShell

  | hadoop.yarn.csi.client.TestCsiClient
  | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation
  | hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade

need to see if these existed before the changes when in, in which case we can start that list of known test failures to ignore

[slfan1989]

hadoop.yarn.csi.client.TestCsiClient

hadoop.yarn.csi.client.TestCsiClient has been fixed, I submit a backport pr(#4380).

hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation

TestCapacitySchedulerAutoQueueCreation#testDynamicAutoQueueCreationWithTags need backport YARN-10506
The main changes are as follows:

 // This test we will reinitialize mockRM, so stop the previous initialized
    // mockRM to avoid issues like MetricsSystem
    if (mockRM != null) {
      mockRM.stop();
    }

hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade

TestDataNodeRollingUpgrade#testWithLayoutChangeAndRollback need backport HDFS-15580.

hadoop.yarn.applications.distributedshell.TestDistributedShell

this class has been removed, need backport YARN-10553.

I can help submit a backport pr, is this required?

[steveloughran]

Thanks for the offer, but it's not needed.

I'm not going to backport any test fixes here, simply note that they are broken and ignore. this really is a security only branch where keeping the #of changes to a minimum reduces the regression risk

[slfan1989]

Thanks for the offer, but it's not needed.

I'm not going to backport any test fixes here, simply note that they are broken and ignore. this really is a security only branch where keeping the #of changes to a minimum reduces the regression risk

Thanks for the explanation, YARN-11166 has been closed.

[steveloughran]

thanks. I've created a table listing those failures so we know what not to worry about. if a new failure surfaces, that's an issue.

[steveloughran]

reverted the cherrypick of HDFS-16563. Namenode WebUI prints sensitive information on Token expiry (#4241), aas it broke other tests.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	10m 58s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	jshint	0m 0s		jshint was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ branch-3.3.3 Compile Tests _
+0 🆗	mvndep	13m 12s		Maven dependency ordering for branch
+1 💚	mvninstall	27m 7s		branch-3.3.3 passed
+1 💚	compile	18m 32s		branch-3.3.3 passed
+1 💚	checkstyle	3m 16s		branch-3.3.3 passed
+1 💚	mvnsite	25m 38s		branch-3.3.3 passed
+1 💚	javadoc	8m 6s		branch-3.3.3 passed
+0 🆗	spotbugs	0m 27s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+0 🆗	spotbugs	0m 28s		branch/hadoop-client-modules/hadoop-client no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	59m 41s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 35s		Maven dependency ordering for patch
+1 💚	mvninstall	27m 43s		the patch passed
+1 💚	compile	17m 54s		the patch passed
+1 💚	javac	17m 54s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	3m 6s		the patch passed
+1 💚	mvnsite	21m 19s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	xml	0m 8s		The patch has no ill-formed XML file.
+1 💚	javadoc	7m 56s		the patch passed
+0 🆗	spotbugs	0m 25s		hadoop-project has no data from spotbugs
+0 🆗	spotbugs	0m 28s		hadoop-client-modules/hadoop-client has no data from spotbugs
+1 💚	shadedclient	59m 11s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	717m 33s	/patch-unit-root.txt	root in the patch passed.
-1 ❌	asflicense	2m 1s	/results-asflicense.txt	The patch generated 1 ASF License warnings.
		1045m 56s

Reason	Tests
Failed junit tests	hadoop.yarn.applications.distributedshell.TestDistributedShell
	hadoop.yarn.csi.client.TestCsiClient
	hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
	hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
	hadoop.hdfs.server.balancer.TestBalancer

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4345/5/artifact/out/Dockerfile
GITHUB PR	#4345
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml spotbugs checkstyle jshint shellcheck shelldocs
uname	Linux 50a0baaae283 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3.3 / 3a8ed3da58e57bbc7e1841a0f529852d9ec9fe2c
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4345/5/testReport/
Max. process+thread count	2192 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-hdfs-project/hadoop-hdfs-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-client-modules/hadoop-client hadoop-tools/hadoop-azure-datalake . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4345/5/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.