HDFS-16740. Mini cluster test flakiness #4835

snmvaughan · 2022-08-30T19:33:42Z

Description of PR

Mini clusters used during HDFS unit tests are reporting test failures that do not appear to be directly related to submitted changes. The failures are the result of either interactions between tests run in parallel, or tests which share common disk space for tests. In all cases, the tests can be run individually serially without any errors. Addressing this issue will simplify future submissions by eliminating the confusion introduced by these unrelated test failures.

We can apply lessons recently from TestRollingUpgrade, which was recently patched to unblock a recent submission. The fixes involved changing the HDFS configuration to use temporary disk space for each individual tests, and using try-with-resources to ensure that clusters were shutdown cleanly.

Add a constructor to the mini cluster constructors and builders that can take a directory from a TemporaryFolder. This encourages the use of managed temporary folders that are unique to the test and automatically cleaned up. The field is named "baseDir" to communicate that this folder acts as the base directory for test data. Using the new constructor encourages the consistent use of the managed temporary folder.

Allow the same baseDir to be provided multiple times, simplifying the coding of cluster restarts. This simplifies these tests as the use of the same TemporaryFolder will always work (even for restarts)

This change also reduces the number of tests that make assumptions about the mini clusters using encapsulation. These changes avoid configurations that directly use HDFS_MINIDFS_BASEDIR.

How was this patch tested?

Both Maven Surefire and an IDE were used to demonstrate that the tests operate as expected.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

snmvaughan · 2022-09-20T19:55:44Z

@steveloughran You mentioned that the test system could use some attention. The set of changes in this pull request, using an updated MiniDFSCluster that leverages temporary folders and ensuring that the cluster is shutdown at the end of the test, could be applied to the remaining tests.

steveloughran

really needs hdfs reviewer i'm afraid -but i've had a look

We can't add junit dependencies to the miniclusters as they get used in different places outside our codebase and adding a new dependency could break them. sorry

hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java

...fs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/MiniJournalCluster.java

hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java

hadoop-yetus · 2022-10-25T04:33:59Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 54s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 13 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	39m 23s		trunk passed
+1 💚	compile	1m 32s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	1m 29s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 19s		trunk passed
+1 💚	mvnsite	1m 39s		trunk passed
+1 💚	javadoc	1m 25s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 46s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 32s		trunk passed
+1 💚	shadedclient	23m 4s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 23s		the patch passed
+1 💚	compile	1m 23s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	1m 23s		the patch passed
+1 💚	compile	1m 18s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 18s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 0s		hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 257 unchanged - 1 fixed = 257 total (was 258)
+1 💚	mvnsite	1m 27s		the patch passed
+1 💚	javadoc	0m 58s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 25s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 23s		the patch passed
+1 💚	shadedclient	22m 45s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	248m 31s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 57s		The patch does not generate ASF License warnings.
		358m 37s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4835/4/artifact/out/Dockerfile
GITHUB PR	#4835
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux cb0727a11da3 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `a5ed6af`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4835/4/testReport/
Max. process+thread count	3513 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4835/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran

I'm happy. yes, it's a bit more verbose -but less likely to cause problems for others.

I still think we need an hdfs developer to review though -can you ask on hdfs-dev list?

xinglin

Overall, looks good. Appreciate your efforts on improving this! This would be really helpful!

...fs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommissionWithStriped.java

hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java

GuoPhilipse · 2023-02-10T01:47:20Z

indeed, restartNameNode faces timeout from time to time, for now we have #5377. it will be very nice if we can eliminate this test case errors .
do we need to change all places related new MiniDFSCluster.Builder to the new constructor ?

snmvaughan · 2023-02-10T02:01:17Z

From what I've seen, many of the timeouts are related to:

Resources that aren't shutdown appropriately, leaving background threads that aren't daemon threads
Interactions between tests when run concurrently, where the tests operate using the same space

This fix is focussed on both issues, as it uses a TemporaryFolder to avoid sharing space between tests and ensures that the cluster and its resources are always shutdown. I'd hope that we would expand this to the remaining uses of the mini clusters. Originally I had proposed new calls that actually took a TemporaryFolder as a parameter, thinking we'd eventually phase out the File-based calls and drop them entirely to ensure consistency.

In addition, I've been submitting PRs that address some of the threading issues (e.g. HDFS-16904 and HADOOP-18279.

GuoPhilipse · 2023-02-10T03:01:53Z

Thanks @snmvaughan for you nice idea, comments from @xinglin seems make sense ,Could you review and resubmit new changes ?

hadoop-yetus · 2023-02-11T02:27:09Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 29s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 1s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 13 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	46m 54s		trunk passed
+1 💚	compile	1m 35s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	compile	1m 27s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	checkstyle	1m 9s		trunk passed
+1 💚	mvnsite	1m 39s		trunk passed
+1 💚	javadoc	1m 11s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javadoc	1m 37s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	3m 44s		trunk passed
+1 💚	shadedclient	28m 26s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 34s		the patch passed
+1 💚	compile	1m 34s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javac	1m 34s		the patch passed
+1 💚	compile	1m 24s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	javac	1m 24s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 58s		hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 257 unchanged - 1 fixed = 257 total (was 258)
+1 💚	mvnsite	1m 31s		the patch passed
+1 💚	javadoc	0m 57s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javadoc	1m 42s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	3m 50s		the patch passed
+1 💚	shadedclient	30m 5s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	304m 33s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
-1 ❌	asflicense	0m 46s	/results-asflicense.txt	The patch generated 2 ASF License warnings.
		435m 23s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestFileCorruption
	hadoop.hdfs.TestBlocksScheduledCounter
	hadoop.hdfs.TestDataTransferProtocol
	hadoop.hdfs.TestSetrepIncreasing
	hadoop.hdfs.TestReadStripedFileWithDecodingDeletedData
	hadoop.hdfs.TestErasureCodingPolicyWithSnapshotWithRandomECPolicy
	hadoop.hdfs.server.namenode.ha.TestObserverNode
	hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor
	hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
	hadoop.hdfs.TestDFSOutputStream
	hadoop.hdfs.TestDistributedFileSystemWithECFile

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4835/5/artifact/out/Dockerfile
GITHUB PR	#4835
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 6ff65efba137 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `2072939`
Default Java	Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4835/5/testReport/
Max. process+thread count	2805 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4835/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

snmvaughan · 2023-02-12T01:46:10Z

There are 5 other test classes which create a build directory directly in the module. 4 of them are looking for a system property test.cache.data, but default to build/test/cache.

src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
src/test/java/org/apache/hadoop/hdfs/util/TestCombinedHostsFileReader.java
src/test/java/org/apache/hadoop/hdfs/TestFileAppendRestart.java
src/test/java/org/apache/hadoop/hdfs/TestDFSShellGenericOptions.java
src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
src/test/java/org/apache/hadoop/hdfs/TestPersistBlocks.java

hadoop-yetus · 2023-02-12T07:14:07Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 44s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 13 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	49m 44s		trunk passed
+1 💚	compile	1m 29s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	compile	1m 21s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	checkstyle	1m 10s		trunk passed
+1 💚	mvnsite	1m 31s		trunk passed
+1 💚	javadoc	1m 9s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javadoc	1m 33s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	3m 30s		trunk passed
+1 💚	shadedclient	27m 0s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 34s		the patch passed
+1 💚	compile	1m 25s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javac	1m 25s		the patch passed
+1 💚	compile	1m 14s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	javac	1m 14s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 54s		hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 257 unchanged - 1 fixed = 257 total (was 258)
+1 💚	mvnsite	1m 26s		the patch passed
+1 💚	javadoc	0m 51s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javadoc	1m 27s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	3m 34s		the patch passed
+1 💚	shadedclient	27m 9s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	206m 18s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 41s		The patch does not generate ASF License warnings.
		333m 12s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.blockmanagement.TestBlockTokenWithShortCircuitRead

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4835/6/artifact/out/Dockerfile
GITHUB PR	#4835
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 7d7fc2526c95 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `495b92d`
Default Java	Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4835/6/testReport/
Max. process+thread count	3969 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4835/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@after

Add a constructor to the mini cluster constructors and builders that takes a TemporaryFolder. This encourages the use of managed temporary folders that are unique to the test and automatically cleaned up. The field is named "baseDir" to communicate that this folder acts as the base directory for test data. These subset of tests recently demonstrated flaky behavior for multiple pull requests. Switch to using a temporary folder managed by the test class, and ensure that the test cluster is shutdown using either @after or try-with-resources. Using the new constructor encourages the consistent use of the managed temporary folder. Allow the same baseDir to be provided multiple times, simplifying cluster restarts. This simplifies these tests as the use of the same TemporaryFolder will always work (even for restarts) This change also reduces the number of tests that make assumptions about the mini clusters using encapsulation. These changes avoid configurations that directly use HDFS_MINIDFS_BASEDIR.

Replace TemporaryFolder parameter with File parameter by using ".getRoot()". Replace JUnit Assert.fail with AssertionError.

The outcome when compared to FileSystem.get is the same, but this makes it clear that it is a LocalFileSystem instance.

hadoop-yetus · 2023-09-15T01:14:13Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 28s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 13 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	40m 23s		trunk passed
+1 💚	compile	1m 6s		trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 56s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	checkstyle	0m 51s		trunk passed
+1 💚	mvnsite	1m 4s		trunk passed
+1 💚	javadoc	0m 53s		trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 19s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	2m 38s		trunk passed
+1 💚	shadedclient	30m 2s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 53s		the patch passed
+1 💚	compile	0m 55s		the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 55s		the patch passed
+1 💚	compile	0m 49s		the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	javac	0m 49s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 45s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 257 unchanged - 1 fixed = 258 total (was 258)
+1 💚	mvnsite	0m 55s		the patch passed
+1 💚	javadoc	0m 39s		the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 20s		the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	2m 22s		the patch passed
+1 💚	shadedclient	28m 15s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	210m 37s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		329m 21s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestFileChecksum
	hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4835/7/artifact/out/Dockerfile
GITHUB PR	#4835
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux b16bcd1b7bd5 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e9103cf`
Default Java	Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4835/7/testReport/
Max. process+thread count	3550 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4835/7/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2023-09-16T00:44:15Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 41s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 13 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	47m 30s		trunk passed
+1 💚	compile	1m 39s		trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	compile	1m 24s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	checkstyle	1m 10s		trunk passed
+1 💚	mvnsite	1m 39s		trunk passed
+1 💚	javadoc	1m 19s		trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 50s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	3m 20s		trunk passed
+1 💚	shadedclient	40m 19s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 18s		the patch passed
+1 💚	compile	1m 21s		the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	javac	1m 21s		the patch passed
+1 💚	compile	1m 15s		the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	javac	1m 15s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 5s		hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 257 unchanged - 1 fixed = 257 total (was 258)
+1 💚	mvnsite	1m 23s		the patch passed
+1 💚	javadoc	0m 56s		the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 40s		the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	3m 47s		the patch passed
+1 💚	shadedclient	41m 29s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	217m 58s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 52s		The patch does not generate ASF License warnings.
		374m 29s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4835/8/artifact/out/Dockerfile
GITHUB PR	#4835
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux ea159205f3e0 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `12bcff4`
Default Java	Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4835/8/testReport/
Max. process+thread count	3234 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4835/8/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran

LGTM -couple of comments

...fs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/MiniJournalCluster.java

...hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/MiniQJMHACluster.java

...ect/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java

xinglin

Thanks for working on this! This improvement is much needed!

I still remember that TestRollingUpgrade failed frequently/non-deterministically for my previous PRs.

xinglin · 2023-09-28T04:27:14Z

@steveloughran Is there anything you are waining for or are we ready to merge this PR? I want it to be merged soon! Thanks,

xinglin · 2023-10-07T17:02:01Z

@steveloughran/ @goiri could we merge this PR? Thanks,

goiri · 2023-10-10T20:51:40Z

I tested this a bit and it seems safe.
Let's merge.

xinglin · 2023-10-10T21:04:57Z

Thanks @goiri for moving this PR forward!

snmvaughan · 2023-10-11T16:39:57Z

Thanks!

* Steve VaughanSteve Vaughan <email@stevevaughan.me> Co-authored-by: Tom McCormick <tmccormi@linkedin.com>

This reverts commit 9f34593.

Contributed by Steve Vaughan <email@stevevaughan.me> Co-authored-by: Tom McCormick <tmccormi@linkedin.com>

steveloughran requested changes Oct 20, 2022

View reviewed changes

apache deleted a comment from hadoop-yetus Oct 25, 2022

steveloughran approved these changes Oct 25, 2022

View reviewed changes

xinglin suggested changes Oct 25, 2022

View reviewed changes

...fs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommissionWithStriped.java Outdated Show resolved Hide resolved

hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java Show resolved Hide resolved

snmvaughan mentioned this pull request Feb 9, 2023

HDFS-16913. Fix flaky some unit tests since they offen timeout #5377

Merged

4 tasks

Steve Vaughan Jr added 8 commits September 14, 2023 15:41

Wrap long-lines

280c1ad

Avoid hidden field message

d15643c

Replace iterator for-loop

9d19720

Remove blanks at the end of lines

ecbaae1

Remove junit dependencies

3ccfae7

Replace TemporaryFolder parameter with File parameter by using ".getRoot()". Replace JUnit Assert.fail with AssertionError.

Use FileSystem.getLocal(conf) to make local use clear

7cf86e7

The outcome when compared to FileSystem.get is the same, but this makes it clear that it is a LocalFileSystem instance.

Use setWorkingDirectory to specify the baseDir

e9103cf

snmvaughan force-pushed the feature/HDFS-16740 branch from 495b92d to e9103cf Compare September 14, 2023 19:43

github-actions bot added HDFS trunk labels Sep 14, 2023

snmvaughan requested a review from xinglin September 14, 2023 19:47

Drop unused import introduced in rebase

12bcff4

steveloughran reviewed Sep 19, 2023

View reviewed changes

xinglin approved these changes Sep 21, 2023

View reviewed changes

goiri approved these changes Oct 10, 2023

View reviewed changes

goiri merged commit 73eccd6 into apache:trunk Oct 10, 2023

snmvaughan deleted the feature/HDFS-16740 branch October 11, 2023 16:40

steveloughran mentioned this pull request Dec 1, 2023

HDFS-16791. Add getEnclosingRoot() API to filesystem interface and im… #6262

Merged

4 tasks

mccormickt12 pushed a commit to mccormickt12/hadoop that referenced this pull request Dec 6, 2023

HDFS-16740. Mini cluster test flakiness (apache#4835)

4536205

steveloughran pushed a commit that referenced this pull request Jan 1, 2024

HDFS-16740. Mini cluster test flakiness (#4835) (#6325)

9f34593

* Steve VaughanSteve Vaughan <email@stevevaughan.me> Co-authored-by: Tom McCormick <tmccormi@linkedin.com>

asfgit pushed a commit that referenced this pull request Jan 1, 2024

Revert "HDFS-16740. Mini cluster test flakiness (#4835) (#6325)"

f3eab22

This reverts commit 9f34593.

asfgit pushed a commit that referenced this pull request Jan 1, 2024

HDFS-16740. Mini cluster test flakiness (#4835) (#6325)

4c08ca2

Contributed by Steve Vaughan <email@stevevaughan.me> Co-authored-by: Tom McCormick <tmccormi@linkedin.com>

jiajunmao pushed a commit to jiajunmao/hadoop-MLEC that referenced this pull request Feb 6, 2024

HDFS-16740. Mini cluster test flakiness (apache#4835)

3a1a772

HDFS-16740. Mini cluster test flakiness #4835

HDFS-16740. Mini cluster test flakiness #4835

Uh oh!

Conversation

snmvaughan commented Aug 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

snmvaughan commented Sep 20, 2022

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Oct 25, 2022

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

xinglin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

GuoPhilipse commented Feb 10, 2023

Uh oh!

snmvaughan commented Feb 10, 2023

Uh oh!

GuoPhilipse commented Feb 10, 2023

Uh oh!

hadoop-yetus commented Feb 11, 2023

Uh oh!

snmvaughan commented Feb 12, 2023

Uh oh!

hadoop-yetus commented Feb 12, 2023

Uh oh!

hadoop-yetus commented Sep 15, 2023

Uh oh!

hadoop-yetus commented Sep 16, 2023

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xinglin left a comment

Choose a reason for hiding this comment

Uh oh!

xinglin commented Sep 28, 2023

Uh oh!

xinglin commented Oct 7, 2023

Uh oh!

goiri commented Oct 10, 2023

Uh oh!

xinglin commented Oct 10, 2023

Uh oh!

snmvaughan commented Oct 11, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

snmvaughan commented Aug 30, 2022 •

edited

Loading