Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix failures handling #95

Merged
merged 6 commits into from
Feb 29, 2024
Merged

Fix failures handling #95

merged 6 commits into from
Feb 29, 2024

Conversation

benoit74
Copy link
Collaborator

@benoit74 benoit74 commented Feb 27, 2024

Rationale

Fix #93
Fix #94
Fix #89

Changes

  • raise an Exception which clearly indicates which node failed to be processed
  • do not remove the future from the list of all futures (we need them for stats + display exceptions)
  • add more details about the number of failures among done tasks
  • log all exceptions which occurred in all tasks so that it is possible to debug all of them at once
  • upgrade to zimscraperlib 3.3.1 for Many errors Cannot set the number of threads, already set #89

@benoit74 benoit74 self-assigned this Feb 27, 2024
@benoit74 benoit74 added this to the 1.3.0 milestone Feb 27, 2024
Copy link

codecov bot commented Feb 27, 2024

Codecov Report

Attention: Patch coverage is 23.52941% with 13 lines in your changes are missing coverage. Please review.

Project coverage is 29.94%. Comparing base (a5b0165) to head (18c7034).

Files Patch % Lines
src/kolibri2zim/scraper.py 23.52% 13 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #95      +/-   ##
==========================================
+ Coverage   29.69%   29.94%   +0.25%     
==========================================
  Files           6        6              
  Lines         751      758       +7     
  Branches      130      131       +1     
==========================================
+ Hits          223      227       +4     
- Misses        522      525       +3     
  Partials        6        6              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@benoit74 benoit74 modified the milestones: 1.3.0, 1.2.1 Feb 27, 2024
@benoit74 benoit74 marked this pull request as ready for review February 27, 2024 14:31
src/kolibri2zim/scraper.py Outdated Show resolved Hide resolved
src/kolibri2zim/scraper.py Outdated Show resolved Hide resolved
@benoit74 benoit74 force-pushed the fix_failures_handling branch 2 times, most recently from 1c8a027 to 1111b36 Compare February 29, 2024 10:55
Copy link
Member

@rgaudin rgaudin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM ; please change the var name before merging

src/kolibri2zim/scraper.py Outdated Show resolved Hide resolved
src/kolibri2zim/scraper.py Outdated Show resolved Hide resolved
@benoit74 benoit74 merged commit 393bcf2 into main Feb 29, 2024
6 of 7 checks passed
@benoit74 benoit74 deleted the fix_failures_handling branch February 29, 2024 13:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants