-
-
Notifications
You must be signed in to change notification settings - Fork 63
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Updated narg parsing. Documentation. Testing. #36
Conversation
cigani
commented
Sep 11, 2021
- Modified nargs in the following ways:
- nargs not explicitly provided are not passed to the generated kwargs dictionary
- nargs can optionally have their type specified
- types of nargs are checked against a simple lookup dictionary
- tried to increase the feedback (error messages) where behavior might not be immediately transparent to the user
- Updated the documentation to be a bit more clear about how default arguments are parsed
- Added a few more tests to ensure functionality of default and type specified arguments
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey thanks again for working on this. It's good to see progress!
I guess by nargs you mean "named args" rather than the argparse nargs
option?
I've left a first round of comments. I still need to check it out and play around with it, maybe think through a few more scenarios for script args, and properly review the tests.
Thanks for the feedback nat. I'll try to realign the PR. Hopefully I'll get you something updated this weekend since I have to work this weekend anyway :D |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we shouldn't require the use of kwargs (script.py:48), since it creates a coupling between the task definition and the function definition, whereby even if you define the call params using positional arguments, you actually have to make the variables names line up with the names from the function def anyway or it won't work. Also if you want to call a function that accepts *args
you're screwed...
For context, a use-case I'd like to encourage is using poe to create a quick and convenient CLI for a python function from any random library with just a couple lines of config! But this means the user might not control the function signature they're interfacing with.
Also from seeing the default_value example in your test case and playing around I realise we have a bigger problem with how we handle call_params and need to rethink a bit. The way it works now (since my original implementation for named args) will cause a breaking change and loss of functionality vs 0.10.0.
That default_value
in script = "pkg:fun(foo=default_value)"
gets interpreted as a string is weird!
For consistency the following should work as one would naively expect:
[tool.poe.tasks.call]
script = "pkg:fun(foo='foo', bar=1, baz=1.0, qux=true, vroom=CLI_ARG, shell_path=environ.get('PATH'))"
args = ["CLI_ARG"]
Note this is assuming we put add from os import environ
to the setup code which I think makes sense to enable this kind of usage, though that's probably a feature for after 0.11.0.
I think we should start with a failing test case like that one, and coordinate a bit so we can both push some changes to this branch without stepping on each others toes.
Thinking on solutions; One alternative approach would be to inject all the named argument values as locals in the scope where the function is called. Then maybe we don't need to edit the call_params string at all! Instead we just expect it to be valid python (maybe with a couple of checks against code injection shenanigans). The downside of this is that we need to think about how to deal with collisions. Eg. if a user wants a cli arg called locals. etc
Another option would be to inject the named args like is done for cmd tasks, e.g.
[tool.poe.tasks.call]
script = "pkg:fun(foo='foo', bar=1, baz=1.0, qux=true, vroom=${CLI_ARGS}, shell_path=environ.get('PATH'))"
args = ["CLI_ARG"]
Though this seems kind of weird...
Anything else probably requires some level of parsing python syntax to modify the call_args string with confidence; at least to handle primitive types, (which means quote matching etc, not just naively splitting on ,
as a first step 🙀 ). Maybe there is an easy way to achieve this... I think it's my preferred solution but more research is required.
What do you think?
Of course we can break this problem up, and leave parts of it to solve in follow up PRs/releases, as long as we don't box ourselves in.
As far as I can tell this doesn't work natively. All those "Default Values" will be interpreted as strings. This would be a good base for a failing test but I think to make it function properly we'd need to exploit TOML types a bit more efficiently
Basically taking the arguments outside of the " " so the TOML can make sense of the type when generating the underling dictionary utilized. An example of this working well is
Here false will be translated by tomli to False as expected. |
- validate that the script content is just a function call - validate that only known arguments are referenced Also - reorganise related tests and fixtures - reorganise filters In progress - cover a failing edge case or two - revisit script task validation logic - clean up
WIP - Parse script task content as python using ast
- remove support for falling back to default function kwarg values when arguments are explicitly passed to the function (because it's python) - simplify test case a bit - add simple support for boolean flag type args - fix some test code broken by the previous WIP refactor - fix some type annotations Still to do: - finishing fixing existing tests - add more tests
WIP fixing named arg parsing for scripts
- also fix misconfiguration of argparse
Cigani dev
And fix bug in substitution of cli args into script tasks
Implement missing ast logic for python <3.8
More tests
tests/test_cmd_tasks.py
Outdated
assert result.stderr == "" | ||
|
||
|
||
def test_shell_task(run_poe_subproc): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not all of these tests are focused on cmd task logic.
…ence (nat-n#40) Allow two addition option value for `ingore_fail` option: - `return_zero`: works like true - `return_non_zero`: run all the subtasks regardless theirs exit status, but returns non-zero exit status if any of subtasks failed
…at-n#36) - Parse script task content as python using ast - Add support for arg types, including special behaviour for boolean type - Add documentation for named arguments - Improve error handling when parsing CLI arguments Testing & quality - improve organisation of test fixtures - reorganise tests by - add extensive tests for named arguments on script tasks - improve use of type anotations like Mapping Co-authored-by: NEXTHINK\mjaquier <michael.jaquier@nexthink.com> Co-authored-by: Nat Noordanus <n@natn.me>
…at-n#36) - Parse script task content as python using ast - Add support for arg types, including special behaviour for boolean type - Add documentation for named arguments - Improve error handling when parsing CLI arguments Testing & quality - improve organisation of test fixtures - reorganise tests by - add extensive tests for named arguments on script tasks - improve use of type anotations like Mapping Co-authored-by: NEXTHINK\mjaquier <michael.jaquier@nexthink.com> Co-authored-by: Nat Noordanus <n@natn.me>