pymultigen - Multi-file frontend for single-file code generators

This small library adds multi-file management on top of one or more existing single-file code generators.

Contents

• Why would I need pymultigen?
• Installation
• Usage
  • Using the Jinja2 integration
• Extending pymultigen
  • Formatters
  • Templating engine

Why would I need pymultigen?

Code generators like Mako or Jinja are great and can be used to generate just about any kind of textual output from templates with a nice template language. They are very mature and battle-proven. However, most of those generators have their origin in the web application domain. The typical usecase is to dynamically render a single HTTP response (most of the time an HTML page) from one or more templates. One HTML page.

If you want to use these generators in other scenarious, e.g. to generate code or reports, but not to one but to multiple files in different folders, pymultigen can help. It simply adds an easy to configure file and folder management layer on top of one or more existing code generators.

Installation

pymultigen comes in form or a regular Python distribution and can be installed from Github or PyPI with a simple:

$ pip install pymultigen

The library works with any version of Python >= 3.3.

Usage

The overall concept of pymultigen is simple:

• A Generator class controls the overall generation workflow. The most important method it implements is generate(model, folder). This is the single method called by users of the created multi-file generator.
• The Generator has a static list of Task objects. Each Task describes a step executed at generation time.
• One Task is responsible for translating a specific set of elements in the input model to one output file in the output folder. The input set can be chosen arbitrarily, often this is the list of a certain model element type (e.g. instance of a Table class in a relational model from which SQL statements should be generated).

Using pymultigen means therefore to create one Generator class for your new generator and one or more Task classes, one for each type of output artifact. If you are using a template-based code generator under the hood, you usually will have one Task per output template.

Before you start, you need to check, whether pymultigen already has an integration for your single-file code generator built-in. Currently, the following integrations are available:

• Jinja2

If you want to use another generation engine, you can easily add support yourself (the current Jinja2 integration consists of less than 20 lines of code). If you've done so, please consider giving back to the community. Your contribution is welcome! Please see below for instructions how to extend pymultigen with a new integration.

Using the Jinja2 integration

You may want to check out pyecoregen, a code generator from pyecore-based models to Python classes. It is a concrete Jinja2-based code generator built with pymultigen.

Jinja2 is a template-based text generator. Writing a file-generator with Jinja therefore involves writing a template for each type of output file. In pymultigen you will then implement a Task class per output file type, i.e. per Jinja template.

The general form of such a Task looks like this:

class MyTask(multigen.jinja.JinjaTask):

    # Name of template file used by this task.
    template_name = 'name-of-template.tpl'

    def filtered_elements(self, model):
        """Return iterator over elements in model that are passed to the above template."""

    def relative_path_for_element(self, element):
        """Return relative file path receiving the generator output for given element."""

The workflow engine will initially call filtered_elements. This method is expected to return an interator over model elements for which a single file needs to be generated. Model is meant here in an abstract way: It may be an instance of a formal metamodel, but it could be any Python object, like a dictionaries or lists. The contained elements being iterated over are accessible from within a template as element.

Once Jinja has produced a textual result it must be written to file. This is where relative_path_for_element comes into play. For a given element that was filtered from the model before, it returns the corresponding filepath. Note that this path is interpreted to be relative to the top-level output path of the overall generation (see below). If subfolders are mentioned here, they are created on demand.

One or more task classes like this must then be registered with a top-level generator. Just like before, a new Generator class is derived from the appropriate base class:

class MyGenerator(multigen.jinja.JinjaGenerator):

    # List of task objects to be processed by this generator.
    tasks = [
        MyTask(),
    ]

    # Root path where Jinja templates are found.
    templates_path = os.path.join(
        os.path.abspath(os.path.dirname(__file__)),
        'templates'
    )

    def create_environment(self, **kwargs):
        """Create Jinja2 environment."""
        environment = super().create_environment(**kwargs)
        # Do any customization of environment here, or delete this method.
        return environment

The base class implementation of {{create_environment}} passes {{templates_path}} to the created environment object to allow Jinja to find the template names specified in a Tasks's template_name. By overriding this method you can extend the environment, e.g. to add filters and tests. Of course you can also completely replace the implementation, e.g. to change the way how templates are looked up.

The example above simply instantiates the new Task class. Here you can optionally pass a formatter function, that is then applied to the output of Jinja. Formatters are simple string transformations, some of which are built into the formatters.py module. If you actually are writing a Python code generator you may want to clean up the generated code according to pep8, so simply pass the appropriate formatter during task instantiation:

class MyGeneratorWithPep8(multigen.jinja.JinjaGenerator):

    # List of task objects to be processed by this generator.
    tasks = [
        MyTask(formatter=multigen.formatter.format_autopep8),
    ]

    ...

Extending pymultigen

Contributions welcome!

Below the most typical extension scenarios are described. Note that in theory pymultigen can be used with any code that produces text, not just a templating engine. Take a look at the class hierarchy in generator.py to get more insights or drop me a note if this is something you plan to do.

Formatters

Writing a new formatter is trivial: Simply create a function that transforms an input string into the nicely formatted output string. If you want to get your formatter added to pymultigen, please make sure that:

• New dependencies (like autopep8 in the existing pep8 formatter) are only imported in the formatting function. This way user only pay for what they use.
• Please write unittests and add your possible dependencies to the tests_require argument in setup.py.

There is not much more to it.

Templating engine

For a live sample, look at the Jinja2 integration in jinja.py. For your templating engine X, you probably have to write small Generator and Task base classes like this:

class XGenerator(TemplateGenerator):

    def __init__(self, environment=None, **kwargs):
        super().__init__(**kwargs)
        # Add any attributes to the generator that are static with respect to a full generation
        # run (over all files), like a Jinja2 environment.
        ...


class XTask(TemplateFileTask):

    def generate_file(self, element, filepath):
        """Actual generation of element."""

Each element that is iterated over from the input model is eventually passed to the tasks's generate_file method. Here simply call you template engine to produce the output string. You also want to apply the optional formatter before writing the string to disk. This is how the Jinja task does it:

def generate_file(self, element, filepath):
    template = self.environment.get_template(self.template_name)
    context = self.create_template_context(element=element)

    with open(filepath, 'wt') as file:
        file.write(self.formatter(template.render(**context)))

The implementation shows two more things:

• The template to be used is retrieved from an environment that is specific to the template engine. Such an environment is usually passed down from the Generator class to the Task.
• create_template_context is a function implemented in base class TemplateTask. It implements the very common case of dictionaries being used as template context objects. Of course you can override this if it doesn't match your engine.