[leo_gateway][leo_storage] Abort multipart upload may not delete parts objects

[mocchira]

How to reproduce

1. Apply the below patch to make multipart uploads fail

diff --git a/apps/leo_gateway/src/leo_gateway_s3_api.erl b/apps/leo_gateway/src/leo_gateway_s3_api.erl
index e91ccff..64f2201 100644
--- a/apps/leo_gateway/src/leo_gateway_s3_api.erl
+++ b/apps/leo_gateway/src/leo_gateway_s3_api.erl
@@ -1193,7 +1193,7 @@ aws_chunk_decode({ok, Acc}, Buffer, read_chunk, Offset,
                             TransferDecodeFun::function(),
                             TransferDecodeState::term(),
                             BucketInfo::#?BUCKET{}).
-handle_multi_upload_1(true, Req, Path, UploadId,
+handle_multi_upload_1(false, Req, Path, UploadId,
                       ChunkedLen, TransferDecodeFun, TransferDecodeState, BucketInfo) ->
     Path4Conf = << Path/binary, ?STR_NEWLINE, UploadId/binary >>,

@@ -1212,7 +1212,7 @@ handle_multi_upload_1(true, Req, Path, UploadId,
         _ ->
             ?reply_service_unavailable_error([?SERVER_HEADER], Path, <<>>, Req)
     end;
-handle_multi_upload_1(false, Req, Path,_UploadId,_ChunkedLen,_,_,_) ->
+handle_multi_upload_1(true, Req, Path,_UploadId,_ChunkedLen,_,_,_) ->
     ?reply_service_unavailable_error([?SERVER_HEADER], Path, <<>>, Req).

 %% @private

2. start a local cluster
3. do the following procedure

$ s3cmd mb s3://s3cmd-bucket
$ s3cmd put README s3://s3cmd-bucket/README # README > 15MB to get it upload through multipart
$ s3cmd abortmp s3://s3cmd-bucket/README ${UPLOAD_ID}
$ leofs-adm compact-start storage_0@127.0.0.1 all
$ leofs-adm du storage_0@127.0.0.1 # show the parts remained in AVS

The result of diagnose before doing compaction

• package/leo_storage/avs/log/leo_object_storage_0.20170614.16.1

194     208189250993278787566252984640442669250 s3cmd-bucket/README     2       5242880 1497423530358933        2017-06-14 15:58:50 +0900       0

• package/leo_storage/avs/log/leo_object_storage_1.20170614.16.1

194     115852115608871076605454137544171509657 s3cmd-bucket/README     1       5242880 1497423530109683        2017-06-14 15:58:50 +0900       0

• package/leo_storage/avs/log/leo_object_storage_2.20170614.16.1

194     234300140694019747126440857722929295612 s3cmd-bucket/README     1       5242880 1497423530070895        2017-06-14 15:58:50 +0900       0
5243233 105661447320521378063464477682844642699 s3cmd-bucket/README     3       5242880 1497423530475018        2017-06-14 15:58:50 +0900       0

• package/leo_storage/avs/log/leo_object_storage_3.20170614.16.1

• package/leo_storage/avs/log/leo_object_storage_4.20170614.16.1

194     322929472505169677583706861778642657406 s3cmd-bucket/README     1       5242880 1497423530136702        2017-06-14 15:58:50 +0900       0

• package/leo_storage/avs/log/leo_object_storage_5.20170614.16.1

194     73111601076547138798092882010366013697  s3cmd-bucket/README     1       15728640        1497423530151350        2017-06-14 15:58:50 +0900       0
351     260946931409785347806887096047221189678 s3cmd-bucket/README     2       5242880 1497423530251371        2017-06-14 15:58:50 +0900       0

• package/leo_storage/avs/log/leo_object_storage_6.20170614.16.1

194     287235268004064704375586318067723307313 s3cmd-bucket/README     2       15728640        1497423530376292        2017-06-14 15:58:50 +0900       0
351     163265313498694697692563896186071436794 s3cmd-bucket/README     0       0       1497423603228365        2017-06-14 16:00:03 +0900       0
506     163265313498694697692563896186071436794 s3cmd-bucket/README     0       0       1497423603229426        2017-06-14 16:00:03 +0900       1
661     163265313498694697692563896186071436794 s3cmd-bucket/README     0       0       1497423633540822        2017-06-14 16:00:33 +0900       0
816     163265313498694697692563896186071436794 s3cmd-bucket/README     0       0       1497423633542037        2017-06-14 16:00:33 +0900       1

• package/leo_storage/avs/log/leo_object_storage_7.20170614.16.1

194     44734086203145785534651236070075864743  s3cmd-bucket/README     0       0       1497423529968551        2017-06-14 15:58:49 +0900       0
563     168600817319643678105772643001232010938 s3cmd-bucket/README     2       5242880 1497423530284103        2017-06-14 15:58:50 +0900       0

That said, there are only tombstones for parent objects. (No tombstones for children)

Now we are vetting when this bug introduced.

[mocchira]

The culprit here

• https://github.com/leo-project/leofs/blob/1.3.4/apps/leo_storage/src/leo_storage_handler_object.erl#L623-L624

Now deleting multiple objects doesn't work for children of a large object but only for children of a (virtual) directory whose name is separated by '/'.

Also there are some bugs found through vetting this issue.

• https://github.com/leo-project/leofs/blob/1.3.4/apps/leo_storage/src/leo_storage_mq.erl#L424
  • Not returning 'ok' but any other type like '{error, cause}' in order to preserve the item and retry later again

@yosukehara I'm going to send PR if fixing the above issues don't conflict with your ongoing work so please check when you can spare time.

[yosukehara]

@mocchira Since there is no problem to me, please send PR.

[mocchira]

@yosukehara Got it. I will go forward to test those fixes then PR.

[mocchira]

Close as the fix already got merged.