HDDS-1553. Add metrics in rack aware container placement policy. #1361

ChenSammi · 2019-08-28T02:51:29Z

https://issues.apache.org/jira/browse/HDDS-1553.

Initial patch

hadoop-yetus · 2019-08-28T10:48:35Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	59	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 6 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	68	Maven dependency ordering for branch
+1	mvninstall	719	trunk passed
+1	compile	443	trunk passed
+1	checkstyle	77	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	861	branch has no errors when building and testing our client artifacts.
+1	javadoc	173	trunk passed
0	spotbugs	435	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	638	trunk passed
		_ Patch Compile Tests _
0	mvndep	36	Maven dependency ordering for patch
+1	mvninstall	567	the patch passed
+1	compile	398	the patch passed
+1	javac	398	the patch passed
-0	checkstyle	40	hadoop-hdds: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	763	patch has no errors when building and testing our client artifacts.
+1	javadoc	189	the patch passed
+1	findbugs	776	the patch passed
		_ Other Tests _
-1	unit	343	hadoop-hdds in the patch failed.
-1	unit	1345	hadoop-ozone in the patch failed.
+1	asflicense	44	The patch does not generate ASF License warnings.
		7761

Reason	Tests
Failed junit tests	hadoop.hdds.scm.container.placement.algorithms.TestSCMContainerPlacementRackAware
	hadoop.ozone.TestMiniChaosOzoneCluster
	hadoop.ozone.TestMiniOzoneCluster
	hadoop.ozone.client.rpc.TestContainerStateMachine

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1361/1/artifact/out/Dockerfile
GITHUB PR	#1361
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux b28d7db4c212 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `55cc115`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1361/1/artifact/out/diff-checkstyle-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1361/1/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1361/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1361/1/testReport/
Max. process+thread count	1537 (vs. ulimit of 5500)
modules	C: hadoop-hdds/server-scm hadoop-ozone/integration-test U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1361/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-08-29T09:45:31Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	75	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 6 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	71	Maven dependency ordering for branch
+1	mvninstall	643	trunk passed
+1	compile	362	trunk passed
+1	checkstyle	71	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	933	branch has no errors when building and testing our client artifacts.
+1	javadoc	170	trunk passed
0	spotbugs	421	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	628	trunk passed
		_ Patch Compile Tests _
0	mvndep	31	Maven dependency ordering for patch
+1	mvninstall	549	the patch passed
+1	compile	378	the patch passed
+1	javac	378	the patch passed
+1	checkstyle	77	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	757	patch has no errors when building and testing our client artifacts.
+1	javadoc	173	the patch passed
+1	findbugs	656	the patch passed
		_ Other Tests _
+1	unit	356	hadoop-hdds in the patch passed.
-1	unit	2379	hadoop-ozone in the patch failed.
+1	asflicense	44	The patch does not generate ASF License warnings.
		8523

Reason	Tests
Failed junit tests	hadoop.ozone.TestOzoneConfigurationFields
	hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
	hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures

Subsystem	Report/Notes
Docker	Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1361/2/artifact/out/Dockerfile
GITHUB PR	#1361
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 1fe3ca2fca57 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `371c9eb`
Default Java	1.8.0_212
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1361/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1361/2/testReport/
Max. process+thread count	5092 (vs. ulimit of 5500)
modules	C: hadoop-hdds/server-scm hadoop-ozone/integration-test U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1361/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-09-03T00:25:06Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	39	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 6 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	64	Maven dependency ordering for branch
+1	mvninstall	582	trunk passed
+1	compile	379	trunk passed
+1	checkstyle	82	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	879	branch has no errors when building and testing our client artifacts.
+1	javadoc	178	trunk passed
0	spotbugs	415	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	613	trunk passed
		_ Patch Compile Tests _
0	mvndep	40	Maven dependency ordering for patch
+1	mvninstall	541	the patch passed
+1	compile	388	the patch passed
+1	javac	388	the patch passed
+1	checkstyle	86	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	686	patch has no errors when building and testing our client artifacts.
+1	javadoc	174	the patch passed
+1	findbugs	631	the patch passed
		_ Other Tests _
+1	unit	269	hadoop-hdds in the patch passed.
-1	unit	1886	hadoop-ozone in the patch failed.
+1	asflicense	54	The patch does not generate ASF License warnings.
		7747

Reason	Tests
Failed junit tests	hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion
	hadoop.ozone.om.TestSecureOzoneManager

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1361/3/artifact/out/Dockerfile
GITHUB PR	#1361
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux c5dc564ae7bd 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `915cbc9`
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1361/3/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1361/3/testReport/
Max. process+thread count	5395 (vs. ulimit of 5500)
modules	C: hadoop-hdds/server-scm hadoop-ozone/integration-test U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1361/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

xiaoyuyao · 2019-09-03T21:30:28Z

...main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMPlacementMetrics.java

NIT: datanodeAllocationTryCount -> datanodeChooseAttemptCount

xiaoyuyao · 2019-09-03T21:31:30Z

...main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMPlacementMetrics.java

datanodeAllocationSuccessCount -> datanodeChooseSuccessCount

xiaoyuyao · 2019-09-03T21:31:49Z

...main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMPlacementMetrics.java

datanodeAllocationCompromiseCount -> datanodeChooseFallbackCount

xiaoyuyao

LGTM, just some minor comments on metrics naming.

ChenSammi · 2019-09-04T12:21:24Z

Thanks Xiaoyu for the comments.

hadoop-yetus · 2019-09-04T19:31:27Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	41	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 6 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	27	Maven dependency ordering for branch
+1	mvninstall	588	trunk passed
+1	compile	405	trunk passed
+1	checkstyle	73	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	913	branch has no errors when building and testing our client artifacts.
+1	javadoc	179	trunk passed
0	spotbugs	471	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	690	trunk passed
-0	patch	521	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
0	mvndep	41	Maven dependency ordering for patch
+1	mvninstall	581	the patch passed
+1	compile	406	the patch passed
+1	javac	406	the patch passed
+1	checkstyle	89	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	776	patch has no errors when building and testing our client artifacts.
+1	javadoc	192	the patch passed
+1	findbugs	689	the patch passed
		_ Other Tests _
+1	unit	286	hadoop-hdds in the patch passed.
-1	unit	192	hadoop-ozone in the patch failed.
+1	asflicense	47	The patch does not generate ASF License warnings.
		6408

Reason	Tests
Failed junit tests	hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1361/4/artifact/out/Dockerfile
GITHUB PR	#1361
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 897d688e963c 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `337e9b7`
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1361/4/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1361/4/testReport/
Max. process+thread count	1346 (vs. ulimit of 5500)
modules	C: hadoop-hdds/server-scm hadoop-ozone/integration-test U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1361/4/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

xiaoyuyao · 2019-09-07T00:12:27Z

Thanks @ChenSammi for the contribution. +1 for the latest change, I merged the change to trunk.

ChenSammi · 2019-09-07T08:04:16Z

Thanks @xiaoyuyao for the review.

…che#1361)

ChenSammi added the ozone label Aug 28, 2019

xiaoyuyao reviewed Sep 3, 2019

View reviewed changes

ChenSammi added 3 commits September 4, 2019 19:09

HDDS-1553. Add metrics in rack aware container placement policy.

514b2a3

fix check stype and failed UT

afbd70e

Change metrics name per Comments; Refatctor class name

ac08519

ChenSammi force-pushed the HDDS-1553 branch from 73861ff to ac08519 Compare September 4, 2019 12:21

xiaoyuyao merged commit c46d43a into apache:trunk Sep 7, 2019

amahussein pushed a commit to amahussein/hadoop that referenced this pull request Oct 29, 2019

HDDS-1553. Add metrics in rack aware container placement policy. (apa…

f35ebf1

…che#1361)

RogPodge pushed a commit to RogPodge/hadoop that referenced this pull request Mar 25, 2020

HDDS-1553. Add metrics in rack aware container placement policy. (apa…

506371c

…che#1361)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-1553. Add metrics in rack aware container placement policy. #1361

HDDS-1553. Add metrics in rack aware container placement policy. #1361

Uh oh!

ChenSammi commented Aug 28, 2019

Uh oh!

hadoop-yetus commented Aug 28, 2019

Uh oh!

hadoop-yetus commented Aug 29, 2019

Uh oh!

hadoop-yetus commented Sep 3, 2019

Uh oh!

xiaoyuyao Sep 3, 2019 •

edited

Loading

Uh oh!

xiaoyuyao Sep 3, 2019

Uh oh!

xiaoyuyao Sep 3, 2019

Uh oh!

xiaoyuyao left a comment

Uh oh!

ChenSammi commented Sep 4, 2019

Uh oh!

hadoop-yetus commented Sep 4, 2019

Uh oh!

xiaoyuyao commented Sep 7, 2019

Uh oh!

ChenSammi commented Sep 7, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HDDS-1553. Add metrics in rack aware container placement policy. #1361

HDDS-1553. Add metrics in rack aware container placement policy. #1361

Uh oh!

Conversation

ChenSammi commented Aug 28, 2019

Uh oh!

hadoop-yetus commented Aug 28, 2019

Uh oh!

hadoop-yetus commented Aug 29, 2019

Uh oh!

hadoop-yetus commented Sep 3, 2019

Uh oh!

xiaoyuyao Sep 3, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xiaoyuyao Sep 3, 2019

Choose a reason for hiding this comment

Uh oh!

xiaoyuyao Sep 3, 2019

Choose a reason for hiding this comment

Uh oh!

xiaoyuyao left a comment

Choose a reason for hiding this comment

Uh oh!

ChenSammi commented Sep 4, 2019

Uh oh!

hadoop-yetus commented Sep 4, 2019

Uh oh!

xiaoyuyao commented Sep 7, 2019

Uh oh!

ChenSammi commented Sep 7, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

xiaoyuyao Sep 3, 2019 •

edited

Loading