AlreadyBeingCreatedException when appending the bookkeeper file on HDFS

[yruslan]

Describe the bug

On HDFS which support atomic appends, the following exception is observed:

 AlreadyBeingCreatedException: Failed to APPEND_FILE
 because DFSClient_NONMAPREDUCE is already the current lease holder.
 This is rare, but happens. This is why retries are introduced.

Such an exception is rare, and seems like completely random.

Expected behavior

The solution is not use unsupported bookkeeping HDFS storage.
But for users that rely on it, the workaround should be in place so that such exceptions won't occur.

Proposed Solution

Add a retry-mechanism for appends.