Enable lstrip_blocks Jinja2 Environment Option for Intended Configs

[cablesquirrel]

🌎 Environment

• Python version: 3.9
• Nautobot version: 1.5.18
• nautobot-golden-config version: branch: feat-update_config_contexts

Enable use of the lstrip_blocks jinja2 environment option to enable golden config templates to use indented code blocks without the added whitespace showing in the rendered config.

👨‍⚖️ Proposed Functionality

As a golden config template maintainer, I'd like to be able to indent my code blocks in my jinja2 templates so that I can enhance readability in situations such as nested loops. This would make the templates easier to read and maintain across teams of developers, and lower the barrier to entry for template creation.

👩‍💻 Use Case

This NTC Blog on Whitespace Control does a great job explaining the need and use cases for lstrip_blocks.

However, the listed directive #jinja2: lstrip_blocks: True which the article mentions can be added to a Jinja template when used with Ansible, does not seem to have any effect when used with Nautobot templates.

In a large scale deployment of Nautobot with dozens or even hundreds of templates, ensuring that directive is properly added to all templates and their includes may be cumbersome.

😡 Solutions That Don't Work

The first solution I tried was to create a custom jinja2 environment and install it as a module during the build of Nautobot.

# my_module/jinja2_environment.py
"""Custom Jinja2 environment."""

from jinja2.sandbox import SandboxedEnvironment


class CustomSandboxedEnvironment(SandboxedEnvironment):
    """Custom Jinja2 environment."""
    def __init__(self, *args, **kwargs):
        SandboxedEnvironment.__init__(self, *args, **kwargs)
        self.lstrip_blocks = True

# nautobot_config.py
###############################
# Template Backend Settings   #
# django==TEMPLATES[0]        #
# Jinja ==TEMPLATES[1]        #
###############################
TEMPLATES[1]["OPTIONS"]["environment"] = "my_module.jinja2_environment.CustomSandboxedEnvironment"
TEMPLATES[1]["BACKEND"] = "django_jinja.backend.Jinja2"

At first glance, the code in nornir_plays/config_intended.py references the jinja templating engine created in django. However, this engine is only imported so that the associated filters can be passed to the generate_config task. The environment itself is not sent with the task.

Modifying both the nautobot-golden-config and nornir-nautobot modules to pass the environment as a parameter, does cause nornir to use the custom environment, but also causes any quoted values in your source YAML files to be escaped as HTML entities in the rendered intended config. This seems to be a side effect of django's intended use of their jinja engine.

✅ Working Solution

I was able to get the desired functionality working by creating a jinja2 environment inside the nautobot-golden-config module with the intention of this environment being specifically used for controlling the rendering of jinja inside the plugin.

I added 2 new settings to our nautobot_config.py for external control of the options.

    "nautobot_golden_config": {
        "per_feature_bar_width": 0.15,
        "per_feature_width": 13,
        "per_feature_height": 4,
        "enable_backup": False,
        "enable_compliance": True,
        "enable_intended": True,
        "enable_sotagg": True,
        "enable_config_context_sync": True,
        "sot_agg_transposer": None,
        "platform_slug_map": None,
        "get_custom_compliance": "nautobot_cox_communications_plugin.compliance.custom_compliance_check",
        "jinja_env_trim_blocks": True,  # <--- ADD THIS LINE (newer configs use is_truthy() method)
        "jinja_env_lstrip_blocks": True,  # <--- ADD THIS LINE (newer configs use is_truthy() method)
    },

In nautobot_golden_config/nornir_plays/config_intended.py, I replaced

jinja_env = engines["jinja"].env

with

# Use a custom Jinja2 environment instead of Django's to avoid HTML escaping
# Trim_blocks option defaulted to True to match nornir's default environment
OPTION_LSTRIP_BLOCKS = False
OPTION_TRIM_BLOCKS = True

if PLUGIN_CFG.get("jinja_env_trim_blocks"):
    OPTION_TRIM_BLOCKS = PLUGIN_CFG.get("jinja_env_trim_blocks")
if PLUGIN_CFG.get("jinja_env_lstrip_blocks"):
    OPTION_LSTRIP_BLOCKS = PLUGIN_CFG.get("jinja_env_lstrip_blocks")

jinja_env = SandboxedEnvironment(
    undefined=StrictUndefined,
    trim_blocks=OPTION_TRIM_BLOCKS,
    lstrip_blocks=OPTION_LSTRIP_BLOCKS,
)
# Pull in Django's built-in filters
jinja_env.filters = engines["jinja"].env.filters

In the same file's run_template method, I added an additional parameter to task.run.

def run_template(  # pylint: disable=too-many-arguments
    task: Task, logger, device_to_settings_map, nautobot_job
) -> Result:
{
    # ...
    generated_config = task.run(
        task=dispatcher,
        name="GENERATE CONFIG",
        method="generate_config",
        obj=obj,
        logger=logger,
        jinja_template=jinja_template,
        jinja_root_path=settings.jinja_repository.filesystem_path,
        output_file_location=output_file_location,
        default_drivers_mapping=get_dispatcher(),
        jinja_filters=jinja_env.filters,
        jinja_env=jinja_env,  # <---  ADD THIS LINE
    )
    # ...
}

Finally, in the nornir-nautobot module, in the file nornir-nautobot/plugins/tasks/dispatcher/default.py, I continued to ensure the parameter for the environment was passed through to the template_file task.

class NautobotNornirDriver:
"""Default collection of Nornir Tasks based on Napalm."""
    # ...
    @staticmethod
    def generate_config(
        task: Task,
        logger,
        obj,
        jinja_template: str,
        jinja_root_path: str,
        output_file_location: str,
        jinja_filters: Optional[dict] = None,
        jinja_env: Optional[jinja2.Environment] = None,  # <--- ADD THIS LINE
    ) -> Result:
    {
        # ...
        filled_template = task.run(
                        **task.host,
                        task=template_file,
                        template=jinja_template,
                        path=jinja_root_path,
                        jinja_filters=jinja_filters,
                        jinja_env=jinja_env,  # <--- ADD THIS LINE
                    )
        # ...
    }

It should be noted that the nornir-jinja2 module creates its own jinja2 environment if one is not sent. With this environment, it enables the option trim_blocks by default. This is the section of code controlling this behavior.

# nornir_jinja2/plugins/tasks/template_file.py

    if jinja_env:
        env = jinja_env
        env.loader = FileSystemLoader(path)
    else:
        env = Environment(
            loader=FileSystemLoader(path), undefined=StrictUndefined, trim_blocks=True,
        )
    env.filters.update(jinja_filters)

🏁 Conclusion

• It seems possible to enable the lstrip_blocks option globally, and would be helpful to template maintainers.
• Nornir-jinja2 is currently creating its own jinja2 environment with trim_blocks enabled by default and cannot be disabled at present.
• Filters are being pulled from the django jinja template engine and passed to nornir. I retained them in my example solution. What are these filters?
• Feedback welcome!

[cablesquirrel]

@jdrew82 @bile0026

This is a continuation of our discussion.

[itdependsnetworks]

You should be able to update the configurations now, not sure what the issues is right now, but I think there should be a solution. That being said, I think this is a common enough request to make better integrations as you mentioned.

@jdrew82 @bile0026 can you put in a PR for this? Not too opinioated on the solution, but I think it makes sense to have the config in nautobot_config.py.

[cablesquirrel]

Adding additional clarification that I have 2 PRs associated with this issue. One is for nautobot-golden-config and the other is for nornir-nautobot.

nornir-nautobot/pull/97
nautobot-plugin-golden-config/pull/527