Delete csv #47
Delete csv #47
Conversation
delete csv file which create from df_to_s3()
|
I was just about to create an issue for this; I think this is a near-must-have feature as I can't imagine users want these files to stay around. It's useful for debugging, so an arg that defaults to deleting the file with the option to disable that feature is the way to go, as this PR is doing. |
| @@ -237,11 +237,17 @@ def s3_to_redshift(redshift_table_name, csv_name, delimiter=',', quotechar='"', | |||
| raise | |||
|
|
|||
|
|
|||
| def delete_csv(csv_name): | |||
| print("Delete s3 bucket's csv file: {0}/{1}".format(s3_subdirectory_var, csv_name)) | |||
| s3.Bucket(s3_bucket_var).objects.filter(Prefix=s3_subdirectory_var+csv_name) | |||
There was a problem hiding this comment.
Don't think this actually does a delete - I did this instead:
s3.Object(s3_bucket_var, s3_subdirectory_var+csv_name).delete()
@agawronski Are you still performing updates to this project? I'd be happy to submit a new PR that provides this "optionally delete S3 file at end" feature. My team would definitely want a 2.1 release that includes this enhancement.
There was a problem hiding this comment.
I won't be able to review and release until the weekend, but sure, we can add this feature. In the meantime I would suggest that you consider checking out https://github.com/jucyai/red-panda which is a very similar package created by @yaojiach who also has been helping maintain pandas_redshift. I believe that it already has this functionality and more, along with several improvements.
There was a problem hiding this comment.
Thanks, we'll use red-panda, no need to address this PR then.
remove csv file whick create from bucket