Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Uploading Empty file when piping with compression #1378

Open
Cosmicoppai opened this issue May 16, 2024 · 1 comment
Open

Uploading Empty file when piping with compression #1378

Cosmicoppai opened this issue May 16, 2024 · 1 comment

Comments

@Cosmicoppai
Copy link

Cosmicoppai commented May 16, 2024

Uploading Empty file when piping with compression

tar -cf - -C "00" . | pzstd > ./test_bkp.tar.zst
[root@docker-desktop:/xbackup# ls
00  02  final  latest_backup.txt  test_bkp.tar.zst
root@docker-desktop:/xbackup# du ish test_bkp.tar.zst
du: cannot access 'ish': No such file or directory
2214520 test_bkp.tar.zst
root@docker-desktop:/xbackup# du -sh test_bkp.tar.zst
2.2G    test_bkp.tar.zst
root@docker-desktop:/xbackup# ^C
root@docker-desktop:/xbackup# s3cmd --config=/root/.s3cfg put ./test_bkp.tar.zst s3://pl-rails-db/xbackup/test_bkp.tar.zst --encrypt --verbose
INFO: No cache file found, creating it.
INFO: Compiling list of local files...
INFO: Running stat() and reading/calculating MD5 values on 1 files, this may take some time...
INFO: Summary: 1 local files to upload
INFO: Encrypting file ./test_bkp.tar.zst to /tmp/tmpfile-Z4t4eDelPWMHo06lYV7W...
... truncated logs( here it's working correctly)

root@docker-desktop:/xbackup# ls
00  02  final  latest_backup.txt  test_bkp.tar.zst
root@docker-desktop:/xbackup# rm test_bkup.tar.zst
rm: cannot remove 'test_bkup.tar.zst': No such file or directory
root@docker-desktop:/xbackup# rm test_bkp.tar.zst
root@docker-desktop:/xbackup# tar -cf - -C "00" . | pzstd | tee /tmp/test_bkp.tar.zst | s3cmd --config=/root/.s3cfg put - s3://pl-rails-db/xbackup/test_bkup.tar.zst --encrypt --verbose
INFO: No cache file found, creating it.
INFO: Compiling list of local files...
INFO: Running stat() and reading/calculating MD5 values on 1 files, this may take some time...
INFO: Summary: 1 local files to upload
INFO: Encrypting file - to /tmp/tmpfile-bsWEPE5NPfYLDHBgxVg1...
upload: '/tmp/tmpfile-bsWEPE5NPfYLDHBgxVg1' -> 's3://pl-rails-db/xbackup/test_bkup.tar.zst'  [1 of 1]
 70 of 70   100% in    0s   109.63 B/s  done

When I compress and upload separately it's getting uploaded correctly else just an empty file (70 B of data getting uploaded).
Found no error in logs or by running even with verbose mode

@Cosmicoppai
Copy link
Author

Cosmicoppai commented May 17, 2024

tar -cf - -C "00" . | pzstd | s3cmd --config=/root/.s3cfg put - s3://pl-rails-db/xbackup/test_bkup.tar.zst --verbose

It's working without the --encrypt option

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant