parametrize: expand grouped parametrizations.

[kaos]

This adds support for having parametrized fields within a parametrize group.

Example:

 __defaults__({
    (python_sources, python_tests): dict(
        **parametrize(
            "py310-compat",
            resolve="service-a",
            interpreter_constraints=[
                "CPython == 3.9.*",
                "CPython == 3.10.*",
            ]
        ),
        **parametrize(
            "py39-compat",
            resolve=parametrize(
                "service-b",
                "service-c",
                "service-d",
            ),
            interpreter_constraints=[
                "CPython == 3.9.*",
            ]
        )
    )
})

Notice

Parametrize groups does not work well with defaults for all, as the parametrized fields will be added to all targets and will result in errors unless it's a field that all targets support.

[cyraxjoe]

Adding my response from the original slack thread from where the suggesting to implemented this functionality came around.

Awesome! I’ve just tested and it works great! With only one small catch which I’m not sure if is expected or not.

If I setup my defaults like:

__defaults__(
    all={
        **parametrize(
            "py310-compat",
            resolve="service-a",
            interpreter_constraints=[
                "CPython == 3.10.*",
            ]
        ),
        **parametrize(
            "py39-compat",
            resolve=parametrize(
                "service-b",
                "service-c",
                "service-d",
                "service-e",
            ),
            interpreter_constraints=[
                "CPython == 3.9.*",
            ]
        )
    },
)

I get the following error:

15:23:08.92 [ERROR] 1 Exception encountered:

Engine traceback:
  in `list` goal
  in Find targets from input specs

InvalidTargetException: libraries/clients/foo-client-x: Unrecognized field `interpreter_constraints=('CPython == 3.9.*',)` in target libraries/clients/foo-client-x:reqs#numpy@parametrize=py39-compat,resolve=service-d. Valid fields for the target type `python_requirement`: ['_find_links', 'dependencies', 'description', 'entry_point', 'modules', 'requirements', 'resolve', 'tags', 'type_stub_modules'].

I can make the error go aways if I do this more verbose/specific alternative (and this works!):

__defaults__(
    { (python_sources, python_tests): dict(
        **parametrize(
            "py310-compat",
            resolve="service-a",
            interpreter_constraints=[
                "CPython == 3.10.*",
            ]
        ),
        **parametrize(
            "py39-compat",
            resolve=parametrize(
                "service-b",
                "service-c",
                "service-d",
                "service-e",
            ),
            interpreter_constraints=[
                "CPython == 3.9.*",
            ]
        )
    ),
      (python_requirement, ): dict(
            resolve=parametrize(
                "service-a",
                "service-b",
                "service-c",
                "service-d",
                "service-e",
            ),
      )
     }
)

I was under the impression that the all keyword should take care of this type of situations, is it failing because of the new parametrize functionality? Or am I missing something?

[cyraxjoe]

Follow up on my previous comments about the behavior of all in combination of this feature of nested parametrization, I implemented a macro that gets the job done for my particular use case, maybe a little too restrictive, but I'll use it to continue exploring pants.

def default_resolvers_per_python_version(**kwargs):
    """Expand a mapping of python version and resolvers into a  BUILD __defaults__
    specialized for python targets.

    Expands:

       default_resolvers_per_python_version(
           py310=[
              "service-a",
              "service-b",
           ],
           py39=[
              "service-c",
              "service-d",
              "service-e,
           ],
       )

   Into:

    __defaults__(
        { (python_sources, python_tests): dict(
            **parametrize(
                "py310-compat",
                resolve=parametrize("service-a", "service-b")
                interpreter_constraints=[
                    "CPython == 3.10.*",
                ]
            ),
            **parametrize(
                "py39-compat",
                resolve=parametrize(
                    "service-c",
                    "service-d",
                    "service-e",
                ),
                interpreter_constraints=[
                    "CPython == 3.9.*",
                ]
            )
        ),
          (python_requirement, ): dict(
              resolve=parametrize(
                    "service-a",
                    "service-b",
                    "service-c",
                    "service-d",
                    "service-e",
              ),
          )
         }
    )
    """
    supported_versions = {
        "py39": [
            "CPython == 3.9.*",
        ],
        "py310": [
            "CPython == 3.10.*",
        ]
    }
    defaults_for_python_sources = {}
    all_resolvers = set()
    for version, resolvers in kwargs.items():
        if version not in supported_versions:
            raise ValueError(
                "Invalid python version {}, we only support {}".format(
                    version, list(supported_versions.keys())
                )
            )
        # parametrize resolvers and interpreter_constraints
        defaults_for_python_sources.update(
            parametrize(
                f"{version}-compat",
                resolve=parametrize(*resolvers),
                interpreter_constraints=supported_versions[version],
            )
        )
        all_resolvers.update(resolvers)

    default_mapping = {
        (python_sources, python_tests): defaults_for_python_sources,
        (python_requirement): dict(
            resolve=parametrize(*all_resolvers)
        )
    }
    __defaults__(default_mapping)

[huonw]

Nice! A few questions/minor tweaks.

[huonw]

Nice!