Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Storage] sky storage delete for Public Buckets #434

Closed
michaelzhiluo opened this issue Feb 24, 2022 · 1 comment · Fixed by #893
Closed

[Storage] sky storage delete for Public Buckets #434

michaelzhiluo opened this issue Feb 24, 2022 · 1 comment · Fixed by #893
Assignees
@michaelzhiluo
Labels
P1

Comments

@michaelzhiluo
Copy link
Collaborator

Deleting public buckets on S3 will hang if the bucket is super big (it tries to delete all objects and doesn't stop halfway even detecting permission error).

Hence it is difficult to delete sky storage logs in sky.db in some cases.
@michaelzhiluo michaelzhiluo self-assigned this Feb 24, 2022
@Michaelvll Michaelvll added the P1 label Mar 7, 2022
@michaelzhiluo
Copy link
Collaborator Author

It turns out aws s3 sync an empty directory onto a huge bucket is very fast! (10-100x faster than aws s3 rm)

@michaelzhiluo michaelzhiluo mentioned this issue Jun 2, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@michaelzhiluo michaelzhiluo

Labels
P1
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fast removal of S3 Storage buckets with 10k-1 million objects skypilot-org/skypilot
2 participants
@Michaelvll @michaelzhiluo