[leo_storage][leo_manager] delete-bucket can stop proceeding for a long time

[mocchira]

Once this problem happened, logs like below started to appear

### on manager
leofs@cat2neat:leofs.1.3.5$ cat package/leo_manager_1/log/app/error.20171019.13.1
[E]     manager_1@127.0.0.1     2017-10-19 13:32:34.555274 +0900        1508387554      leo_manager_del_bucket_handler:notify_fun/3     286     [{cause,timeout}]
[E]     manager_1@127.0.0.1     2017-10-19 13:33:04.558549 +0900        1508387584      leo_manager_del_bucket_handler:notify_fun/3     286     [{cause,timeout}]
[E]     manager_1@127.0.0.1     2017-10-19 13:33:34.561387 +0900        1508387614      leo_manager_del_bucket_handler:notify_fun/3     286     [{cause,timeout}]
[E]     manager_1@127.0.0.1     2017-10-19 13:34:04.562930 +0900        1508387644      leo_manager_del_bucket_handler:notify_fun/3     286     [{cause,{'EXIT',{timeout,{gen_server,call,[leo_storage_handler_del_directory,{enqueue,0,<<"test_1">>,false},30000]}}}}]
[E]     manager_1@127.0.0.1     2017-10-19 13:34:34.564992 +0900        1508387674      leo_manager_del_bucket_handler:notify_fun/3     286     [{cause,timeout}]
[E]     manager_1@127.0.0.1     2017-10-19 13:35:04.567261 +0900        1508387704      leo_manager_del_bucket_handler:notify_fun/3     286     [{cause,timeout}]
[E]     manager_1@127.0.0.1     2017-10-19 13:35:34.569891 +0900        1508387734      leo_manager_del_bucket_handler:notify_fun/3     286     [{cause,timeout}]
[E]     manager_1@127.0.0.1     2017-10-19 13:36:04.571432 +0900        1508387764      leo_manager_del_bucket_handler:notify_fun/3     286     [{cause,timeout}]
[E]     manager_1@127.0.0.1     2017-10-19 13:36:34.572910 +0900        1508387794      leo_manager_del_bucket_handler:notify_fun/3     286     [{cause,timeout}]
[E]     manager_1@127.0.0.1     2017-10-19 13:37:04.574207 +0900        1508387824      leo_manager_del_bucket_handler:notify_fun/3     286     [{cause,timeout}]
[E]     manager_1@127.0.0.1     2017-10-19 13:37:34.576652 +0900        1508387854      leo_manager_del_bucket_handler:notify_fun/3     286     [{cause,timeout}]
[E]     manager_1@127.0.0.1     2017-10-19 13:38:04.580294 +0900        1508387884      leo_manager_del_bucket_handler:notify_fun/3     286     [{cause,timeout}]

### on storage
[E]     storage_0@127.0.0.1     2017-10-19 13:32:09.486342 +0900        1508387529      leo_storage_handler_del_directory:notify_current_state_to_manager/4     758     [{cause,{badrpc,timeout}}]
[E]     storage_0@127.0.0.1     2017-10-19 13:32:39.487967 +0900        1508387559      leo_storage_handler_del_directory:notify_current_state_to_manager/4     758     [{cause,{badrpc,timeout}}]
[E]     storage_0@127.0.0.1     2017-10-19 13:33:14.495068 +0900        1508387594      leo_storage_handler_del_directory:notify_current_state_to_manager/4     758     [{cause,{badrpc,timeout}}]
[E]     storage_0@127.0.0.1     2017-10-19 13:33:44.498368 +0900        1508387624      leo_storage_handler_del_directory:notify_current_state_to_manager/4     758     [{cause,{badrpc,timeout}}]
[E]     storage_0@127.0.0.1     2017-10-19 13:34:19.502181 +0900        1508387659      leo_storage_handler_del_directory:notify_current_state_to_manager/4     758     [{cause,{badrpc,timeout}}]
[E]     storage_0@127.0.0.1     2017-10-19 13:34:49.503000 +0900        1508387689      leo_storage_handler_del_directory:notify_current_state_to_manager/4     758     [{cause,{badrpc,timeout}}]
[E]     storage_0@127.0.0.1     2017-10-19 13:35:24.505199 +0900        1508387724      leo_storage_handler_del_directory:notify_current_state_to_manager/4     758     [{cause,{badrpc,timeout}}]
[E]     storage_0@127.0.0.1     2017-10-19 13:35:54.506750 +0900        1508387754      leo_storage_handler_del_directory:notify_current_state_to_manager/4     758     [{cause,{badrpc,timeout}}]
[E]     storage_0@127.0.0.1     2017-10-19 13:36:29.509413 +0900        1508387789      leo_storage_handler_del_directory:notify_current_state_to_manager/4     758     [{cause,{badrpc,timeout}}]
[E]     storage_0@127.0.0.1     2017-10-19 13:36:59.510998 +0900        1508387819      leo_storage_handler_del_directory:notify_current_state_to_manager/4     758     [{cause,{badrpc,timeout}}]
[E]     storage_0@127.0.0.1     2017-10-19 13:37:34.514059 +0900        1508387854      leo_storage_handler_del_directory:notify_current_state_to_manager/4     758     [{cause,{badrpc,timeout}}]
[E]     storage_0@127.0.0.1     2017-10-19 13:38:04.516246 +0900        1508387884      leo_storage_handler_del_directory:notify_current_state_to_manager/4     758     [{cause,{badrpc,timeout}}]

and keep dumping the same error. if you have a luck then those errors get disappeared and delete-bucket get to proceed again.
The reason why this happens is
gen_server on leo_storage and gen_server on leo_manager call the rpc each other in the blocking mode so depending on the timing, each rpc calls can be blocked and subsequently timeout will happen.
Decreasing the timeout value for rpc calls would make this problem less likely to happen.