Replace uploader & downloader implementations with rclone and compress before upload

[aylei]

@xiaojingchen @onlymellb @tennix @weekface PTAL
Signed-off-by: Aylei rayingecho@gmail.com

Benchmark:

Environment: container runs in an AWS m4.xlarge instance (4C 16G) in tokyo with no resources limit. The data is uploaded to and download from S3 in US East Ohio.

Data size: 30.2 GB

rclone implementation:

Upload Time:

• Compress: 17m 26s
• Upload: 15m 05s
• Total: 32m 32.74s

(Compressed size 14.5 GB)

Download Time:

• Download: 7m 42s
• Decompress: 10m 07s
• Total: 17m 40.09s

Resources Usage:

• Compress: the container stably consumes 400% CPU and 11MB memory
• Upload: the container stably consumes 30% memory and 23MB memory, avg network tx 16.4MB/s

Go implementation:

Upload Time: 38m 10s
Download Time: 54m 47s

Tested upon s3 & ceph manually.

s3 log:

docker run -it --entrypoint sh aylei:test
/ # mkdir -p /data/backup/
/ # cd /data/backup
/data/backup # echo 111 > sql1
/data/backup # echo 222 > sql2
/data/backup # cd /
/ # uploader  --cloud=aws --region=us-east-2 --bucket=new-bucket-aylei --backup-dir=/data/backup
+ basename /usr/local/bin/uploader
+ export 'RUN_MODE=uploader'
+ export 'CLOUD=aws'
+ shift
+ export 'REGION=us-east-2'
+ shift
+ export 'BUCKET=new-bucket-aylei'
+ shift
+ export 'BACKUP_DIR=/data/backup'
+ shift
+ '[' aws '=' gcp ]
+ '[' aws '=' ceph ]
+ '[' aws '=' aws ]
+ '[' -z aaaa ]
+ '[' -z xxxxx ]
+ cat
+ '[' uploader '=' uploader ]
+ dirname /data/backup
+ export 'BACKUP_BASE_DIR=/data'
+ basename /data/backup
+ tar cvzf /data/backup.tgz -C /data backup
backup/
backup/sql1
backup/sql2
+ rclone --config /tmp/rclone.conf copyto /data/backup.tgz aws:new-bucket-aylei//data/backup/backup.tgz
/usr/local/bin/uploader: line 1: rclone: not found
/ # mv rclone /usr/local/bin/rclone
/ # uploader  --cloud=aws --region=us-east-2 --bucket=new-bucket-aylei --backup-dir=/data/backup
+ basename /usr/local/bin/uploader
+ export 'RUN_MODE=uploader'
+ export 'CLOUD=aws'
+ shift
+ export 'REGION=us-east-2'
+ shift
+ export 'BUCKET=new-bucket-aylei'
+ shift
+ export 'BACKUP_DIR=/data/backup'
+ shift
+ '[' aws '=' gcp ]
+ '[' aws '=' ceph ]
+ '[' aws '=' aws ]
+ '[' -z aaaaa ]
+ '[' -z ccccc ]
+ cat
+ '[' uploader '=' uploader ]
+ dirname /data/backup
+ export 'BACKUP_BASE_DIR=/data'
+ basename /data/backup
+ tar cvzf /data/backup.tgz -C /data backup
backup/
backup/sql1
backup/sql2
+ rclone --config /tmp/rclone.conf copyto /data/backup.tgz aws:new-bucket-aylei//data/backup/backup.tgz
/ # downloader --cloud=aws --region=us-east-2 --bucket=new-bucket-aylei --srcDir=/data/backup --destDir=/data
+ basename /usr/local/bin/downloader
+ export 'RUN_MODE=downloader'
+ export 'CLOUD=aws'
+ shift
+ export 'REGION=us-east-2'
+ shift
+ export 'BUCKET=new-bucket-aylei'
+ shift
+ export 'BACKUP_DIR=/data/backup'
+ shift
+ export 'DEST_DIR=/data'
+ shift
+ '[' aws '=' gcp ]
+ '[' aws '=' ceph ]
+ '[' aws '=' aws ]
+ '[' -z aaaaa ]
+ '[' -z ccccc ]
+ cat
+ '[' downloader '=' uploader ]
+ '[' downloader '=' downloader ]
+ rclone --config /tmp/rclone.conf copyto aws:new-bucket-aylei//data/backup /data
+ '[' -f /data/backup.tgz ]
+ tar xzvf /data/backup.tgz -C /data
backup/
backup/sql1
backup/sql2
+ rm /data/backup.tgz
/ # cd /data
/data # cd backup/
/data/backup # ls
sql1  sql2

Note that the new image can read the backup taken before, but the backup data taken by the new image cannot be properly de-compressed by the old image.

GCP test has not been done yet.

[CLAassistant]

All committers have signed the CLA.

[aylei]

@onlymellb @xiaojingchen @tennix @weekface All comments addressed, PTAL

[gregwebs]

Did you benchmark the two implementations?

[aylei]

@gregwebs Not yet, I've found a data set with compressed size ~15 GiB, hopefully I can do a simple benchmark tomorrow.

[onlymellb]

LGTM

[tennix]

Rest LGTM

[aylei]

@tennix Addressed, PTAL again

[aylei]

@gregwebs I've posted a simple benchmark but I'm really sorry that I don't have enough time these days to do a detailed benchmark

[gregwebs]

benchmark is good enough for me