Wrong template field types for AWS operators

[pavelhlushchanka]

Apache Airflow version: 2.1.1

Environment:

• Cloud provider or hardware configuration: AWS

What happened:

After migrating from Airflow 2.0.1 to 2.1.1, the Rendered Template tab stopped working for dictionaries. I checked it for Sagemaker and ECS operators, the code is not shown.

However for some operators where i have customized list of template_fields, it partly works (only for extra fields).

So then i checked a release notes and it looks like it's related to #15130 and for some AWS operators the type of the fields should be not py, but json since all those fields mostly simply propagated to boto3 as json.

[pavelhlushchanka]

It might be not just for AWS operators, but these are the ones i use and i could test

[uranusjr]

Hi! Would you be interested in submitting a PR to fix this?

[pavelhlushchanka]

@uranusjr i can make pr but only for the operators i use

[uranusjr]

That’s perfectly fine! If some other operators also have this issue, people who use them can submit fixes for them.

[jordanjeremy]

As mentioned previously, this is not specific to AWS operators. This issue also happens with @task decorated (TaskFlow API) operators, where the op_kwargs is a dict and is setup to be rendered as py.

Possible solution is to update the "get_python_source" in utils/code_utils.py to handle list and dict. Possibly something like:

    if isinstance(x, list):
        return [str(v) for v in x]

    if isinstance(x, dict):
        return {k: str(v) for k, v in x.items()}

The converting values to strings seems necessary in order to avoid errors when this is passed to the pygments lexer.

[kaxil]

Closed by #16820