Make command parsing support operations on param expansions (#266)

Specifically cmd tasks may now use the following operations like in bash
- `:-` fallback value, e.g. ${AWS_REGION:-us-east-1}
- `:+` value replacement, e.g. ${AWESOME:+--awesome-mode}

This was done by adding AST parser support for Param Expansion operators

Also:
- Fix bug in param expansion logic for pure whitespace param values
- Add env paramater to PoeThePoet class to override os.environ for testing

.github/workflows/ci.yml

@@ -118,14 +118,14 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Trigger update of homebrew formula
-        run: >
+        run: |
           sleep 10 # some delay seems to be necessary
-          curl -L -X POST
-          -H "Accept: application/vnd.github+json"
-          -H "Authorization: Bearer ${{ secrets.homebrew_pat }}"
-          -H "X-GitHub-Api-Version: 2022-11-28"
-          https://api.github.com/repos/nat-n/homebrew-poethepoet/actions/workflows/71211730/dispatches
-          -d '{"ref":"main", "inputs":{}}'
+          curl -L -X POST \
+            -H "Accept: application/vnd.github+json" \
+            -H "Authorization: Bearer ${{ secrets.homebrew_pat }}" \
+            -H "X-GitHub-Api-Version: 2022-11-28" \
+            https://api.github.com/repos/nat-n/homebrew-poethepoet/actions/workflows/71211730/dispatches \
+            -d '{"ref":"main", "inputs":{}}'
 
   github-release:
     name: >-

docs/tasks/task_types/cmd.rst

@@ -26,6 +26,7 @@ It is important to understand that ``cmd`` tasks are executed without a shell (t
 
 .. _ref_env_vars:
 
+
 Referencing environment variables
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -50,6 +51,29 @@ Parameter expansion can also can be disabled by escaping the $ with a backslash
   greet = "echo Hello \\$USER"  # the backslash itself needs escaping for the toml parser
 
 
+Parameter expansion operators
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When referencing an environment variable in a cmd task you can use the ``:-`` operator from bash to specify a *default value*, to be used in case the variable is unset. Similarly the ``:+`` operator can be used to specify an *alternate value* to use in place of the environment variable if it *is* set.
+
+In the following example, if ``AWS_REGION`` has a value then it will be used, otherwise ``us-east-1`` will be used as a fallback.
+
+.. code-block:: toml
+
+  [tool.poe.tasks]
+  tables = "aws dynamodb list-tables --region ${AWS_REGION:-us-east-1}"
+
+The ``:+`` or *alternate value* operator is especially useful in cases such as the following where you might want to control whether some CLI options are passed to the command.
+
+.. code-block:: toml
+
+  [tool.poe.tasks.aws-identity]
+  cmd = "aws sts get-caller-identity ${ARN_ONLY:+ --no-cli-pager --output text --query 'Arn'}"
+  args = [{ name = "ARN_ONLY", options = ["--arn-only"], type = "boolean" }]
+
+In this example we declare a boolean argument with no default, so if the ``--arn-only`` flag is provided to the task then three additional CLI options will be included in the task content.
+
+
 Glob expansion
 ~~~~~~~~~~~~~~
 
@@ -78,7 +102,7 @@ Here's an example of task using a recursive glob pattern:
 
 .. seealso::
 
-  Much like in bash, the glob pattern can be escaped by wrapping it in quotes, or preceding it with a backslash.
+  Just like in bash, the glob pattern can be escaped by wrapping it in quotes, or preceding it with a backslash.
 
 
 .. |glob_link| raw:: html

poethepoet/app.py

@@ -20,28 +20,38 @@ class PoeThePoet:
         this determines where to look for a pyproject.toml file, defaults to
         ``Path().resolve()``
     :type cwd: Path, optional
+
     :param config:
         Either a dictionary with the same schema as a pyproject.toml file, or a
         `PoeConfig <https://github.com/nat-n/poethepoet/blob/main/poethepoet/config/config.py>`_
         object to use as an alternative to loading config from a file.
     :type config: dict | PoeConfig, optional
+
     :param output:
         A stream for the application to write its own output to, defaults to sys.stdout
     :type output: IO, optional
+
     :param poetry_env_path:
         The path to the poetry virtualenv. If provided then it is used by the
         `PoetryExecutor <https://github.com/nat-n/poethepoet/blob/main/poethepoet/executor/poetry.py>`_,
         instead of having to execute poetry in a subprocess to determine this.
     :type poetry_env_path: str, optional
+
     :param config_name:
         The name of the file to load tasks and configuration from. If not set then poe
         will search for config by the following file names: pyproject.toml
         poe_tasks.toml poe_tasks.yaml poe_tasks.json
     :type config_name: str, optional
+
     :param program_name:
         The name of the program that is being run. This is used primarily when
         outputting help messages, defaults to "poe"
     :type program_name: str, optional
+
+    :param env:
+        Optionally provide an alternative base environment for tasks to run with.
+        If no mapping is provided then ``os.environ`` is used.
+    :type env: dict, optional
     """
 
     cwd: Path
@@ -58,6 +68,7 @@ def __init__(
         poetry_env_path: Optional[str] = None,
         config_name: Optional[str] = None,
         program_name: str = "poe",
+        env: Optional[Mapping[str, str]] = None,
     ):
         from .config import PoeConfig
         from .ui import PoeUi
@@ -75,6 +86,7 @@ def __init__(
         )
         self.ui = PoeUi(output=output, program_name=program_name)
         self._poetry_env_path = poetry_env_path
+        self._env = env if env is not None else os.environ
 
     def __call__(self, cli_args: Sequence[str], internal: bool = False) -> int:
         """
@@ -212,9 +224,9 @@ def get_run_context(self, multistage: bool = False) -> "RunContext":
         result = RunContext(
             config=self.config,
             ui=self.ui,
-            env=os.environ,
+            env=self._env,
             dry=self.ui["dry_run"],
-            poe_active=os.environ.get("POE_ACTIVE"),
+            poe_active=self._env.get("POE_ACTIVE"),
             multistage=multistage,
             cwd=self.cwd,
         )

poethepoet/env/template.py

@@ -56,7 +56,7 @@ def apply_envvars_to_template(
     content: str, env: Mapping[str, str], require_braces=False
 ) -> str:
     """
-    Template in ${environmental} $variables from env as if we were in a shell
+    Template in ${environment} $variables from env as if we were in a shell
 
     Supports escaping of the $ if preceded by an odd number of backslashes, in which
     case the backslash immediately preceding the $ is removed. This is an

poethepoet/exceptions.py

@@ -6,9 +6,19 @@ class PoeException(RuntimeError):
     cause: Optional[str]
 
     def __init__(self, msg, *args):
+        super().__init__(msg, *args)
         self.msg = msg
-        self.cause = args[0].args[0] if args else None
-        self.args = (msg, *args)
+
+        if args:
+            cause = args[0]
+            position_clause = (
+                f", near line {cause.line}, position {cause.position}."
+                if getattr(cause, "has_position", False)
+                else "."
+            )
+            self.cause = cause.args[0] + position_clause
+        else:
+            self.cause = None
 
 
 class CyclicDependencyError(PoeException):
@@ -34,7 +44,7 @@ def __init__(
         task_name: Optional[str] = None,
         index: Optional[int] = None,
         global_option: Optional[str] = None,
-        filename: Optional[str] = None
+        filename: Optional[str] = None,
     ):
         super().__init__(msg, *args)
         self.context = context

poethepoet/helpers/command/__init__.py

@@ -31,7 +31,7 @@ def resolve_command_tokens(
     patterns that are not escaped or quoted. In case there are glob patterns in the
     token, any escaped glob characters will have been escaped with [].
     """
-    from .ast import Glob, ParamExpansion, ParseConfig, PythonGlob
+    from .ast import Glob, ParamArgument, ParamExpansion, ParseConfig, PythonGlob
 
     if not config:
         config = ParseConfig(substitute_nodes={Glob: PythonGlob})
@@ -56,23 +56,54 @@ def finalize_token(token_parts):
         token_parts.clear()
         return (token, includes_glob)
 
+    def resolve_param_argument(argument: ParamArgument, env: Mapping[str, str]):
+        token_parts = []
+        for segment in argument.segments:
+            for element in segment:
+                if isinstance(element, ParamExpansion):
+                    token_parts.append(resolve_param_value(element, env))
+                else:
+                    token_parts.append(element.content)
+
+        return "".join(token_parts)
+
+    def resolve_param_value(element: ParamExpansion, env: Mapping[str, str]):
+        param_value = env.get(element.param_name, "")
+
+        if element.operation:
+            if param_value:
+                if element.operation.operator == ":+":
+                    # apply 'alternate value' operation
+                    param_value = resolve_param_argument(
+                        element.operation.argument, env
+                    )
+
+            elif element.operation.operator == ":-":
+                # apply 'default value' operation
+                param_value = resolve_param_argument(element.operation.argument, env)
+
+        return param_value
+
     for line in lines:
         # Ignore line breaks, assuming they're only due to comments
         for word in line:
             if isinstance(word, Comment):
                 # strip out comments
                 continue
 
-            # For each token part indicate whether it is a glob
             token_parts: list[tuple[str, bool]] = []
             for segment in word:
                 for element in segment:
                     if isinstance(element, ParamExpansion):
-                        param_value = env.get(element.param_name, "")
+                        param_value = resolve_param_value(element, env)
                         if not param_value:
+                            # Empty param value has no effect
                             continue
                         if segment.is_quoted:
-                            token_parts.append((env.get(element.param_name, ""), False))
+                            token_parts.append((param_value, False))
+                        elif param_value.isspace():
+                            # collapse whitespace value
+                            token_parts.append((" ", False))
                         else:
                             # If the the param expansion it not quoted then:
                             # - Whitespace inside a substituted param value results in