HDFS-16672. Fix lease interval comparison in BlockReportLeaseManager #4598

cxzl25 · 2022-07-20T13:50:37Z

Description of PR

monotonicNowMs is generated by System.nanoTime(), direct comparison is not recommended.

org.apache.hadoop.hdfs.server.blockmanagement.BlockReportLeaseManager#pruneIfExpired

if (monotonicNowMs < node.leaseTimeMs + leaseExpiryMs) {
  return false;
}

How was this patch tested?

exist UT

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2022-07-20T19:51:40Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 37s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	38m 36s		trunk passed
+1 💚	compile	1m 43s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	1m 40s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 24s		trunk passed
+1 💚	mvnsite	1m 48s		trunk passed
+1 💚	javadoc	1m 26s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	1m 48s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 43s		trunk passed
+1 💚	shadedclient	23m 56s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 27s		the patch passed
+1 💚	compile	1m 29s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	1m 29s		the patch passed
+1 💚	compile	1m 17s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 17s		the patch passed
+1 💚	blanks	0m 1s		The patch has no blanks issues.
+1 💚	checkstyle	0m 56s		the patch passed
+1 💚	mvnsite	1m 20s		the patch passed
+1 💚	javadoc	0m 55s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	1m 28s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 18s		the patch passed
+1 💚	shadedclient	23m 7s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	248m 21s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 15s		The patch does not generate ASF License warnings.
		359m 40s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4598/1/artifact/out/Dockerfile
GITHUB PR	#4598
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 4e1fa9a68002 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `4781ea4`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4598/1/testReport/
Max. process+thread count	3295 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4598/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

cxzl25 · 2022-07-29T07:58:36Z

Help to review when you are free. Thanks.
@Hexiaoqiao

Hexiaoqiao · 2022-08-01T06:12:37Z

@cxzl25 Thanks involve me here. I don't get this improvement. It seems that System.nanoTime() in single thread then compare. Do you meet any issue here? Thanks.

cxzl25 · 2022-08-01T06:37:11Z

I don't get this improvement. It seems that System.nanoTime() in single thread then compare. Do you meet any issue here? Thanks.

System.nanoTime()may return a negative number, the comparison here may be wrong.

In LeaseManager.Lease#expiredHardLimit() we use two nanotimes to subtract. This is consistent with the usage of nanoTime recommended by JDK.

Hexiaoqiao · 2022-08-26T08:49:58Z

Sorry for the late response. But I did not get why System.nanoTime() may return a negative number. Do you mind to offer any jdk information? Thanks.

System.nanoTime()may return a negative number, the comparison here may be wrong.

cxzl25 · 2022-08-26T09:06:43Z

Sorry for the late response. But I did not get why System.nanoTime() may return a negative number. Do you mind to offer any jdk information? Thanks.

System.nanoTime()may return a negative number, the comparison here may be wrong.

https://docs.oracle.com/javase/8/docs/api/index.html

public static long nanoTime()

(perhaps in the future, so values may be negative)

one should use t1 - t0 < 0, not t1 < t0, because of the possibility of numerical overflow.

Hexiaoqiao · 2022-08-26T09:24:06Z

OK, Got it, Thanks. LGTM. +1 from me.

ayushtkn

Changes LGTM.

@cxzl25 If you rebase this and if the build comes green I can commit this

hadoop-yetus · 2023-04-25T08:16:49Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 47s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	40m 2s		trunk passed
+1 💚	compile	1m 26s		trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	1m 24s		trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	checkstyle	1m 8s		trunk passed
+1 💚	mvnsite	1m 30s		trunk passed
+1 💚	javadoc	1m 8s		trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	1m 35s		trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	spotbugs	3m 28s		trunk passed
+1 💚	shadedclient	23m 1s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 16s		the patch passed
+1 💚	compile	1m 19s		the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	1m 19s		the patch passed
+1 💚	compile	1m 14s		the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	javac	1m 14s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 51s		the patch passed
+1 💚	mvnsite	1m 21s		the patch passed
+1 💚	javadoc	0m 50s		the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	1m 26s		the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	spotbugs	3m 15s		the patch passed
+1 💚	shadedclient	22m 21s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	205m 38s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 46s		The patch does not generate ASF License warnings.
		314m 31s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4598/2/artifact/out/Dockerfile
GITHUB PR	#4598
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux c73b387aac1d 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `64e4dfd`
Default Java	Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4598/2/testReport/
Max. process+thread count	3056 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4598/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2023-04-25T19:34:33Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 36s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	43m 14s		trunk passed
+1 💚	compile	1m 23s		trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	1m 18s		trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	checkstyle	1m 5s		trunk passed
+1 💚	mvnsite	1m 26s		trunk passed
+1 💚	javadoc	1m 6s		trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	1m 33s		trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	spotbugs	3m 29s		trunk passed
+1 💚	shadedclient	23m 21s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 17s		the patch passed
+1 💚	compile	1m 18s		the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	1m 18s		the patch passed
+1 💚	compile	1m 13s		the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	javac	1m 13s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 50s		the patch passed
+1 💚	mvnsite	1m 19s		the patch passed
+1 💚	javadoc	0m 50s		the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	1m 27s		the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	spotbugs	3m 16s		the patch passed
+1 💚	shadedclient	24m 57s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	209m 16s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 50s		The patch does not generate ASF License warnings.
		323m 21s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4598/3/artifact/out/Dockerfile
GITHUB PR	#4598
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux b7684163632e 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `f212fbc`
Default Java	Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4598/3/testReport/
Max. process+thread count	3529 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4598/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

…4598). Contributed by dzcxzl. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org

ayushtkn approved these changes Apr 24, 2023

View reviewed changes

fix

64e4dfd

cxzl25 force-pushed the HDFS-16672 branch from 4781ea4 to 64e4dfd Compare April 25, 2023 03:00

trigger test

f212fbc

ayushtkn merged commit 5af0845 into apache:trunk Apr 25, 2023

asfgit pushed a commit that referenced this pull request Apr 25, 2023

HDFS-16672. Fix lease interval comparison in BlockReportLeaseManager (#…

4d052a2

…4598). Contributed by dzcxzl. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-16672. Fix lease interval comparison in BlockReportLeaseManager #4598

HDFS-16672. Fix lease interval comparison in BlockReportLeaseManager #4598

Uh oh!

cxzl25 commented Jul 20, 2022 •

edited

Loading

Uh oh!

hadoop-yetus commented Jul 20, 2022

Uh oh!

cxzl25 commented Jul 29, 2022

Uh oh!

Hexiaoqiao commented Aug 1, 2022

Uh oh!

cxzl25 commented Aug 1, 2022

Uh oh!

Hexiaoqiao commented Aug 26, 2022

Uh oh!

cxzl25 commented Aug 26, 2022

Uh oh!

Hexiaoqiao commented Aug 26, 2022

Uh oh!

ayushtkn left a comment

Uh oh!

hadoop-yetus commented Apr 25, 2023

Uh oh!

hadoop-yetus commented Apr 25, 2023

Uh oh!

Uh oh!

HDFS-16672. Fix lease interval comparison in BlockReportLeaseManager #4598

HDFS-16672. Fix lease interval comparison in BlockReportLeaseManager #4598

Uh oh!

Conversation

cxzl25 commented Jul 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

hadoop-yetus commented Jul 20, 2022

Uh oh!

cxzl25 commented Jul 29, 2022

Uh oh!

Hexiaoqiao commented Aug 1, 2022

Uh oh!

cxzl25 commented Aug 1, 2022

Uh oh!

Hexiaoqiao commented Aug 26, 2022

Uh oh!

cxzl25 commented Aug 26, 2022

Uh oh!

Hexiaoqiao commented Aug 26, 2022

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Apr 25, 2023

Uh oh!

hadoop-yetus commented Apr 25, 2023

Uh oh!

Uh oh!

cxzl25 commented Jul 20, 2022 •

edited

Loading