Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

batch deletions on azure / limited to 256 - Closes #509 #508

Merged

Conversation

alikefia
Copy link

@alikefia alikefia commented Mar 7, 2025

One of the implementations of delete folder on azure lists all the blobs and delete them directly. The delete request on azure blobs is limited to 256 objects per request (https://learn.microsoft.com/en-us/python/api/azure-storage-blob/azure.storage.blob.containerclient?view=azure-python#azure-storage-blob-containerclient-delete-blobs)
The proposal is to batch the call


Contributor checklist:

  • I have read and understood CONTRIBUTING.md
  • Confirmed an issue exists for the PR, and the text Closes #issue appears in the PR summary (e.g., Closes #123).
  • Confirmed PR is rebased onto the latest base
  • Confirmed failure before change and success after change
  • Any generic new functionality is replicated across cloud providers if necessary
  • Tested manually against live server backend for at least one provider
  • Added tests for any new functionality
  • Linting passes locally
  • Tests pass locally
  • Updated HISTORY.md with the issue that is addressed and the PR you are submitting. If the top section is not `## UNRELEASED``, then you need to add a new section to the top of the document for your change.

@pjbull
Copy link
Member

pjbull commented Mar 7, 2025

Thanks @alikefia, looks good in principal. Can you follow the other checklist items to lint, add changelog entry, and add a test?

@alikefia
Copy link
Author

alikefia commented Mar 10, 2025

Test running on master branch (fixed fail)

image

@alikefia alikefia changed the title batch deletions on azure / limited to 256 batch deletions on azure / limited to 256 - Closes #509 Mar 10, 2025
@pjbull
Copy link
Member

pjbull commented Mar 10, 2025

@alikefia Looks like batched isn't available in all the versions of Python we support. We can add a backport with the implementation from the docs: https://docs.python.org/3/library/itertools.html#itertools.batched

Copy link

codecov bot commented Mar 12, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.3%. Comparing base (d3522c7) to head (f7e2dc4).
Report is 1 commits behind head on 508-live-tests.

Additional details and impacted files
@@               Coverage Diff                @@
##           508-live-tests    #508     +/-   ##
================================================
- Coverage            93.7%   92.3%   -1.4%     
================================================
  Files                  23      23             
  Lines                1798    1800      +2     
================================================
- Hits                 1685    1663     -22     
- Misses                113     137     +24     
Files with missing lines Coverage Δ
cloudpathlib/azure/azblobclient.py 84.8% <100.0%> (-9.5%) ⬇️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@pjbull pjbull changed the base branch from master to 508-live-tests March 12, 2025 18:19
@pjbull
Copy link
Member

pjbull commented Mar 12, 2025

Docs failure is unrelated; pushing to local branch to run live tests.

@pjbull pjbull merged commit 8f99698 into drivendataorg:508-live-tests Mar 12, 2025
22 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants