Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated narg parsing. Documentation. Testing. #36

Merged
merged 24 commits into from
Nov 4, 2021

Conversation

cigani
Copy link
Contributor

@cigani cigani commented Sep 11, 2021

  • Modified nargs in the following ways:
  1. nargs not explicitly provided are not passed to the generated kwargs dictionary
  2. nargs can optionally have their type specified
  3. types of nargs are checked against a simple lookup dictionary
  4. tried to increase the feedback (error messages) where behavior might not be immediately transparent to the user
  • Updated the documentation to be a bit more clear about how default arguments are parsed
  • Added a few more tests to ensure functionality of default and type specified arguments

Copy link
Owner

@nat-n nat-n left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey thanks again for working on this. It's good to see progress!

I guess by nargs you mean "named args" rather than the argparse nargs option?

I've left a first round of comments. I still need to check it out and play around with it, maybe think through a few more scenarios for script args, and properly review the tests.

poethepoet/task/args.py Outdated Show resolved Hide resolved
poethepoet/task/args.py Outdated Show resolved Hide resolved
poethepoet/task/args.py Outdated Show resolved Hide resolved
poethepoet/task/args.py Outdated Show resolved Hide resolved
README.rst Show resolved Hide resolved
README.rst Outdated Show resolved Hide resolved
README.rst Outdated Show resolved Hide resolved
tests/fixtures/dummy_project/pyproject.toml Outdated Show resolved Hide resolved
poethepoet/task/script.py Outdated Show resolved Hide resolved
@cigani
Copy link
Contributor Author

cigani commented Sep 14, 2021

Thanks for the feedback nat. I'll try to realign the PR. Hopefully I'll get you something updated this weekend since I have to work this weekend anyway :D

Copy link
Owner

@nat-n nat-n left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we shouldn't require the use of kwargs (script.py:48), since it creates a coupling between the task definition and the function definition, whereby even if you define the call params using positional arguments, you actually have to make the variables names line up with the names from the function def anyway or it won't work. Also if you want to call a function that accepts *args you're screwed...

For context, a use-case I'd like to encourage is using poe to create a quick and convenient CLI for a python function from any random library with just a couple lines of config! But this means the user might not control the function signature they're interfacing with.


Also from seeing the default_value example in your test case and playing around I realise we have a bigger problem with how we handle call_params and need to rethink a bit. The way it works now (since my original implementation for named args) will cause a breaking change and loss of functionality vs 0.10.0.

That default_value in script = "pkg:fun(foo=default_value)" gets interpreted as a string is weird!

For consistency the following should work as one would naively expect:

[tool.poe.tasks.call]
script = "pkg:fun(foo='foo', bar=1, baz=1.0, qux=true, vroom=CLI_ARG, shell_path=environ.get('PATH'))"
args = ["CLI_ARG"]

Note this is assuming we put add from os import environ to the setup code which I think makes sense to enable this kind of usage, though that's probably a feature for after 0.11.0.

I think we should start with a failing test case like that one, and coordinate a bit so we can both push some changes to this branch without stepping on each others toes.

Thinking on solutions; One alternative approach would be to inject all the named argument values as locals in the scope where the function is called. Then maybe we don't need to edit the call_params string at all! Instead we just expect it to be valid python (maybe with a couple of checks against code injection shenanigans). The downside of this is that we need to think about how to deal with collisions. Eg. if a user wants a cli arg called locals. etc

Another option would be to inject the named args like is done for cmd tasks, e.g.

[tool.poe.tasks.call]
script = "pkg:fun(foo='foo', bar=1, baz=1.0, qux=true, vroom=${CLI_ARGS}, shell_path=environ.get('PATH'))"
args = ["CLI_ARG"]

Though this seems kind of weird...

Anything else probably requires some level of parsing python syntax to modify the call_args string with confidence; at least to handle primitive types, (which means quote matching etc, not just naively splitting on , as a first step 🙀 ). Maybe there is an easy way to achieve this... I think it's my preferred solution but more research is required.

What do you think?

Of course we can break this problem up, and leave parts of it to solve in follow up PRs/releases, as long as we don't box ourselves in.

tests/fixtures/dummy_project/pyproject.toml Outdated Show resolved Hide resolved
tests/test_task_running.py Outdated Show resolved Hide resolved
@cigani
Copy link
Contributor Author

cigani commented Sep 20, 2021

[tool.poe.tasks.call]
script = "pkg:fun(foo='foo', bar=1, baz=1.0, qux=true, vroom=${CLI_ARGS}, shell_path=environ.get('PATH'))"
args = ["CLI_ARG"]

As far as I can tell this doesn't work natively. All those "Default Values" will be interpreted as strings. This would be a good base for a failing test but I think to make it function properly we'd need to exploit TOML types a bit more efficiently

script = "pkg:fun", { foo='foo', bar=1, baz=1.0, qux=true, vroom=${CLI_ARGS}, shell_path=environ.get('PATH')) }
args = ["CLI_ARG"]

Basically taking the arguments outside of the " " so the TOML can make sense of the type when generating the underling dictionary utilized. An example of this working well is

  [tool.poe.tasks.greet-rekeyed.args.value]
    default=false

Here false will be translated by tomli to False as expected.

nat-n and others added 18 commits September 21, 2021 23:20
- validate that the script content is just a function call
- validate that only known arguments are referenced

Also
- reorganise related tests and fixtures
- reorganise filters

In progress
- cover a failing edge case or two
- revisit script task validation logic
- clean up
WIP - Parse script task content as python using ast
- remove support for falling back to default function kwarg values when arguments
  are explicitly passed to the function (because it's python)
- simplify test case a bit
- add simple support for boolean flag type args
- fix some test code broken by the previous WIP refactor
- fix some type annotations

Still to do:
- finishing fixing existing tests
- add more tests
WIP fixing named arg parsing for scripts
- also fix misconfiguration of argparse
And fix bug in substitution of cli args into script tasks
Implement missing ast logic for python <3.8
.vscode/launch.json Outdated Show resolved Hide resolved
assert result.stderr == ""


def test_shell_task(run_poe_subproc):
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not all of these tests are focused on cmd task logic.

tsimoshka and others added 2 commits November 2, 2021 08:25
…ence (nat-n#40)

Allow two addition option value for `ingore_fail` option:
- `return_zero`: works like true
- `return_non_zero`: run all the subtasks regardless theirs exit status,
but returns non-zero exit status if any of subtasks failed
@nat-n nat-n merged commit cfffd23 into nat-n:development Nov 4, 2021
@cigani cigani deleted the development branch November 11, 2021 12:32
ThatXliner pushed a commit to ThatXliner/poethepoet that referenced this pull request Dec 24, 2021
…at-n#36)

- Parse script task content as python using ast
- Add support for arg types, including special behaviour for boolean type
- Add documentation for  named arguments
- Improve error handling when parsing CLI arguments

Testing & quality
- improve organisation of test fixtures
- reorganise tests by
- add extensive tests for named arguments on script tasks
- improve use of type anotations like Mapping

Co-authored-by: NEXTHINK\mjaquier <michael.jaquier@nexthink.com>
Co-authored-by: Nat Noordanus <n@natn.me>
ThatXliner pushed a commit to ThatXliner/poethepoet that referenced this pull request Dec 24, 2021
…at-n#36)

- Parse script task content as python using ast
- Add support for arg types, including special behaviour for boolean type
- Add documentation for  named arguments
- Improve error handling when parsing CLI arguments

Testing & quality
- improve organisation of test fixtures
- reorganise tests by
- add extensive tests for named arguments on script tasks
- improve use of type anotations like Mapping

Co-authored-by: NEXTHINK\mjaquier <michael.jaquier@nexthink.com>
Co-authored-by: Nat Noordanus <n@natn.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants