Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: Allow DatasetMetadataExtractor.get_required_content() to return a generator #361

Merged
merged 8 commits into from
Mar 7, 2023

Conversation

jsheunis
Copy link
Member

@jsheunis jsheunis commented Mar 2, 2023

This PR:

  • Adds logic to extract.py to allow either a returned bool or a yieldedGenerator to be interpreted correctly: If an extractor implements logic where any result record is yielded with a failure 'status' (i.e. equal to 'impossible' or 'error') the extractor infrastructure will signal an error and the extractor's extract method will not be called.
  • Updates the docstring of DatasetMetadataExtractor.get_required_content() in extractors/base.by to indicate this change, especially to developers implementing extractors
  • Adds missing return statements in extract.py where failures are yielded
  • Updates related documentation
  • Closes DatasetMetadataExtractor.get_required_content() should be able to yield datalad result dicts #356

@jsheunis jsheunis requested review from christian-monch and mslw March 2, 2023 13:10
Copy link
Collaborator

@christian-monch christian-monch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks a lot. A few comments below

datalad_metalad/extract.py Outdated Show resolved Hide resolved
datalad_metalad/extract.py Outdated Show resolved Hide resolved
datalad_metalad/extract.py Outdated Show resolved Hide resolved
datalad_metalad/extractors/base.py Show resolved Hide resolved
datalad_metalad/extractors/base.py Show resolved Hide resolved
docs/source/user_guide/writing-extractors.rst Outdated Show resolved Hide resolved
docs/source/user_guide/writing-extractors.rst Outdated Show resolved Hide resolved
docs/source/user_guide/writing-extractors.rst Outdated Show resolved Hide resolved
jsheunis and others added 4 commits March 2, 2023 16:40
Co-authored-by: Christian Mönch <christian.moench@web.de>
Co-authored-by: Christian Mönch <christian.moench@web.de>
Co-authored-by: Christian Mönch <christian.moench@web.de>
Co-authored-by: Christian Mönch <christian.moench@web.de>
@codecov-commenter
Copy link

codecov-commenter commented Mar 2, 2023

Codecov Report

Patch coverage: 30.76% and project coverage change: -0.14 ⚠️

Comparison is base (7a60107) 86.42% compared to head (932d211) 86.28%.

❗ Current head 932d211 differs from pull request most recent head 58c51e5. Consider uploading reports for the commit 58c51e5 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #361      +/-   ##
==========================================
- Coverage   86.42%   86.28%   -0.14%     
==========================================
  Files          88       88              
  Lines        4831     4841      +10     
==========================================
+ Hits         4175     4177       +2     
- Misses        656      664       +8     
Impacted Files Coverage Δ
datalad_metalad/extract.py 88.28% <25.00%> (-2.78%) ⬇️
datalad_metalad/extractors/base.py 86.66% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Copy link
Collaborator

@christian-monch christian-monch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @jsheunis

@jsheunis
Copy link
Member Author

jsheunis commented Mar 3, 2023

Great! @christian-monch feel free to merge if you're happy.

@christian-monch christian-monch changed the title ENH: Allow DatasetMetadataExtractor.get_required_content() to yield a generator ENH: Allow DatasetMetadataExtractor.get_required_content() to return a generator Mar 7, 2023
@christian-monch christian-monch merged commit c741d49 into master Mar 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

DatasetMetadataExtractor.get_required_content() should be able to yield datalad result dicts
3 participants