HDDS-2175. Propagate stack trace for OM Exceptions to the Client. Contributed by Supratim Deka

[supratimdeka]

https://issues.apache.org/jira/browse/HDDS-2175

complete stack trace added into the message field of OM response.
SCM exceptions(allocate block) are wrapped into an IOException on the OM and propagated to the client.

Intent is to make debugging more convenient.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	77	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
-1	mvninstall	49	hadoop-hdds in trunk failed.
-1	mvninstall	28	hadoop-ozone in trunk failed.
-1	compile	18	hadoop-hdds in trunk failed.
-1	compile	12	hadoop-ozone in trunk failed.
+1	checkstyle	58	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	951	branch has no errors when building and testing our client artifacts.
-1	javadoc	18	hadoop-hdds in trunk failed.
-1	javadoc	16	hadoop-ozone in trunk failed.
0	spotbugs	1035	Used deprecated FindBugs config; considering switching to SpotBugs.
-1	findbugs	28	hadoop-hdds in trunk failed.
-1	findbugs	17	hadoop-ozone in trunk failed.
		_ Patch Compile Tests _
-1	mvninstall	31	hadoop-hdds in the patch failed.
-1	mvninstall	25	hadoop-ozone in the patch failed.
-1	compile	20	hadoop-hdds in the patch failed.
-1	compile	15	hadoop-ozone in the patch failed.
-1	javac	20	hadoop-hdds in the patch failed.
-1	javac	15	hadoop-ozone in the patch failed.
+1	checkstyle	50	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	791	patch has no errors when building and testing our client artifacts.
-1	javadoc	17	hadoop-hdds in the patch failed.
-1	javadoc	16	hadoop-ozone in the patch failed.
-1	findbugs	27	hadoop-hdds in the patch failed.
-1	findbugs	16	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	24	hadoop-hdds in the patch failed.
-1	unit	20	hadoop-ozone in the patch failed.
+1	asflicense	29	The patch does not generate ASF License warnings.
		2463

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1520/1/artifact/out/Dockerfile
GITHUB PR	#1520
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 8bb46246a1d7 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 3f89084
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1520/1/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1520/1/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1520/1/artifact/out/branch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1520/1/artifact/out/branch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1520/1/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1520/1/artifact/out/branch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1520/1/artifact/out/branch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1520/1/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1520/1/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1520/1/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1520/1/artifact/out/patch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1520/1/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1520/1/artifact/out/patch-compile-hadoop-hdds.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1520/1/artifact/out/patch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1520/1/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1520/1/artifact/out/patch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1520/1/artifact/out/patch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1520/1/artifact/out/patch-findbugs-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1520/1/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1520/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1520/1/testReport/
Max. process+thread count	308 (vs. ulimit of 5500)
modules	C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1520/1/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[elek]

Thank you very much to file this issue @supratimdeka. The big question (for me) is the type of the exceptions.

There are two kind of exception:

• Business exceptions (eg. VolumeNotFound) and
• System exceptions (eg. connection refused).

This particular code is about propagating the business exceptions. I am not sure if we need to
propagate the strack trace there. For example a VOLUME_NOT_FOUND doesn't require any exception message to be propagated.

It's enough to propagate the error type and the message. (This is what we do.)

In case of a real system exception the stack trace is is propagated with Hadoop RPC (inside an RpcException If I remember well... The full stack trace is propagated there...)

[elek]

For the records. We discussed it offline:

The most easy but powerful way can be an improvement to convert only the OM/SCM exceptions to message + status code and throw all the other exceptions (to be caucht by Hadoop RPC and stack trace will be propagated by hadoop rpc)

[anuengineer]

Also are these call stacks something that the end user should ever see? I have always found as user a call stack useless, it might be useful for the developer for debugging purposes, but clients are generally things used by real users. Maybe if these stacks are not logged in the ozone.log, we can log them, provided we can guard them via a config key and by default we do not do that.