ScaffOps

Take a sip of coffee while your project is being generated ☕

Idea

Tools like cookiecutter or copier allow for creating repositories that define reusable structure of your projects. These repositories, further called upstream projects, are projects themselves—they often use Jinja templates to generate files, which is an error-prone process that implies the necessity of creating tests for them. Downstream projects, those that are generated by rendering templates from upstream projects, should ideally never rely on pre/post "hooks" to complete project scaffolding—instead, the downstream project should feed templates with parameters written the way it's never needed to run arbitrary code other than the one in Jinja expressions.

What ScaffOps does differently than other scaffolding tools?

1. Provide a preview framework.

   Developing a project scaffolding is tiring if it's impossible to review your changes easily. Using a convenient facade built on top of watchfiles and "downstream samples", it is possible to preview how your changes will affect the downstream projects before things get out of control.

2. Provide a testing framework.

   Similar to CTT, released in the ScaffOps package.

3. Integrate with Git out of the box.

   Maintaining a scaffolded project costs some pollution of the project history. To make the process minimally painful, ScaffOps will make sure to maintain a list of ScaffOps-originated commits to provide the possibility of periodically squashing template history and adding commits that change template-generated parts of source files to .git-blame-ignore-revs.

4. Provide an auto-update service.

   When maintaining more than 3 scaffolded projects at a time, it can become tedious to manually update all the projects with the newest template copies from the upstream. A working solution is to write a simple script to walk through all projects and update them.

   An auto-update service would extend this script into a separate tool that would allow to apply, revert upstream updates on dynamic or static lists of projects.

5. Copy unrendered upstream templates into the target (downstream) project.

   With this approach, every downstream project pulls the templates from upstream and stores them inside a dedicated directory of configurable name, by default scaffops/. The project can then tweak the templates locally in commits that would be respected in next template pulls. So for example, I would pull a template scaffops/README.md.jinja and override some text inside. Whenever I pull it again, my change stays, but the change from the template is applied.

   This approach gives various benefits:

   • Reduction of conditional blocks.

     The less logical branches, the longer a piece of software will live (in other words, the easier it will be to maintain).

     It's often unnecessary to make a project scaffolding support every case. During working with cookiecutter or copier, I often find myself categorizing my projects in hundreds of ways: does it use A or B? Well if A, scaffold X. If B, scaffold Y. In practice, 99% of the projects used A 1% used B. However, I strongly felt that it is still worth adding this to a template for fully reproducible project generation. That is now possible—ScaffOps will expect the downstream projects to override templates, not files generated (which is still possible for source files, though). Thus, whenever a template change feels like it could land as a logical branch at upstream, it is easy to do, since the change was applied to a downstream version of the same template.

   • Easy attribution.

     Anytime an update from an upstream repo is pulled, separate update commits are applied per template in the repo with proper co-authorships out of the box. This way, people who contributed to developing upstream templates are also credited in every downstream project, without necessarily having to copy the whole history.

   • Easy guidance and contribution transfers.

     It is often a problem that new contributors change generated files instead of proper templates to solve a problem at the infrastructure level. If the templates live inside the downstream project, the user can be guided via file comments to easily apply their idea there. If a commit deserves a transfer to upstream, it can be easily cherry-picked.

   • Downstream projects become upstreams themselves.

     Storing a copy of templates with own, locally adapted history, can be particularly useful for creating templates that are themselves created from templates. This might happen when developing highly extensible systems or plugin templates.

     So, in short, it is now possible to generate downstream projects from already downstream projects.

Note

More to come in this section later.