Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Gather Inputs pipeline step #33

Merged
5 commits merged into from
Sep 20, 2024
Merged

Gather Inputs pipeline step #33

5 commits merged into from
Sep 20, 2024

Conversation

pgierz
Copy link
Member

@pgierz pgierz commented Sep 11, 2024

Summary

  • fix: small mistake in gather inputs
  • fix(gather_inputs.py): deprecates the gather_inputs function
  • feat: new API for inputs in Rule
  • doc: fix too long docstring

Description

This changes the way the Rule API works. You now need to give Rule objects a list of dictionaries:

rules:
  - name: my_rule
    inputs:
      - path: "/some/location/with/files"
        pattern: ".*nc"
      - path: "/some/location/with/more/files"
        pattern: "EXPID_fesom_thetao_.*nc"

The old API, using input_patterns now raises deprecation warnings. I have not tested if it still work technically, but it should (maybe).

In any case, this is the way to go for the future.

@pgierz
fix: small mistake in gather inputs
e05c201
@pgierz
fix(gather_inputs.py): deprecates the gather_inputs function
7647db8 
This will deprecate the gather_inputs function in favour of a pipeline step. At the moment, you just get a deprecation warning if you try to use the function.
@pgierz
feat: new API for inputs in Rule
8b68241 
Rule has a new API. Instead of input patterns as a list of strings, you
now provide inputs, as a list of dictionaries. These should be
key/values with path and pattern as keys, later used to get the
appropriate file(s) from each path. Tests are also updated to reflect the
new way of loading files into the Rules.

BREAKING CHANGE: Any signature constructing a Rule object will need to
be changed!
@pgierz
doc: fix too long docstring
0e1ce57
@pgierz pgierz mentioned this pull request Sep 17, 2024
Merged
@siligam siligam closed this pull request by merging all changes into main in 587820f Sep 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mandresm mandresm Awaiting requested review from mandresm

@siligam siligam Awaiting requested review from siligam

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@pgierz @siligam