Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: 🐛 don't check if dataset is supported when we know it is #720

Merged
merged 6 commits into from
Jan 30, 2023
Merged

fix: 🐛 don't check if dataset is supported when we know it is #720

merged 6 commits into from
Jan 30, 2023

Conversation

severo
Copy link
Collaborator

@severo severo commented Jan 30, 2023 •
edited
Loading

As we are running a loop of updates on supported datasets, it's useless to check if the dataset is supported inside the update_dataset method.

@severo severo mentioned this pull request Jan 30, 2023
Merged
@severo
Copy link
Collaborator Author

severo commented Jan 30, 2023 •
edited
Loading

codecov upload is failing... it's not related to the PR - fixed (hidden) with #721

@codecov-commenter
Copy link

Codecov Report

Base: 92.01% // Head: 91.56% // Decreases project coverage by -0.46% ⚠️

Coverage data is based on head (a6c2968) compared to base (4414d57).
Patch coverage: 0.00% of modified lines in pull request are covered.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #720      +/-   ##
==========================================
- Coverage   92.01%   91.56%   -0.46%     
==========================================
  Files          33       65      +32     
  Lines        2205     3865    +1660     
==========================================
+ Hits         2029     3539    +1510     
- Misses        176      326     +150
Flag Coverage Δ
libs_libcommon 91.40% <0.00%> (?)
services_api 90.13% <ø> (?)
workers_datasets_based 92.01% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
libs/libcommon/src/libcommon/dataset.py 53.33% <0.00%> (ø)
services/api/src/api/app.py 92.59% <0.00%> (ø)
libs/libcommon/tests/test_queue.py 100.00% <0.00%> (ø)
libs/libcommon/src/libcommon/storage.py 87.50% <0.00%> (ø)
libs/libcommon/src/libcommon/processing_graph.py 91.66% <0.00%> (ø)
services/api/src/api/utils.py 94.00% <0.00%> (ø)
services/api/src/api/routes/processing_step.py 60.00% <0.00%> (ø)
libs/libcommon/src/libcommon/config.py 80.21% <0.00%> (ø)
services/api/tests/test_prometheus.py 93.75% <0.00%> (ø)
... and 23 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

AndreaFrancis
AndreaFrancis approved these changes Jan 30, 2023
View reviewed changes
Copy link
Contributor

@AndreaFrancis AndreaFrancis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

albertvillanova
albertvillanova approved these changes Jan 30, 2023
View reviewed changes
Copy link
Member

@albertvillanova albertvillanova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for removing the unnecessary check.

.github/workflows/_unit-tests-python.yml
@@ -59,5 +59,5 @@ jobs:
with:
working-directory: ${{ inputs.working-directory }}
files: ./coverage.xml
fail_ci_if_error: true
fail_ci_if_error: false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this change is unrelated to this PR... ;)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I used a separate PR for that: #720 (comment), and merged it here...
Maybe it would have been better to send to main... Well, next time

libs/libcommon/src/libcommon/dataset.py Outdated Show resolved Hide resolved
libs/libcommon/src/libcommon/operations.py Outdated
@@ -26,6 +26,7 @@ def update_dataset(
hf_token: Optional[str] = None,
force: bool = False,
priority: Priority = Priority.NORMAL,
skip_check_support: bool = False,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have recently had a discussion in datasets about parameters with reversed semantic meaning...

What about reversing the logic? Something like do_check_support and being True by default?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or support_checking...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, you're right

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done with 52d2cf6

severo and others added 3 commits January 30, 2023 14:08
@severo @albertvillanova
Update libs/libcommon/src/libcommon/dataset.py
479a6f6 
Co-authored-by: Albert Villanova del Moral <8515462+albertvillanova@users.noreply.github.com>
@severo
refactor: 💡 don't use double negations :P
52d2cf6
@severo
style: 💄 please black
411f7a8
albertvillanova
albertvillanova approved these changes Jan 30, 2023
View reviewed changes
Copy link
Member

@albertvillanova albertvillanova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@severo
Copy link
Collaborator Author

severo commented Jan 30, 2023

Sorry, didn't see you already approved... Merging

@severo severo merged commit 2f38593 into main Jan 30, 2023
@severo severo deleted the reduce-duration-backfill branch January 30, 2023 13:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AndreaFrancis AndreaFrancis AndreaFrancis approved these changes

@lhoestq lhoestq Awaiting requested review from lhoestq

@albertvillanova albertvillanova Awaiting requested review from albertvillanova

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@severo @codecov-commenter @AndreaFrancis @albertvillanova