Can Wyam be used as an explicit, componentized pipeline, built on SOLID principles?

[jhgoodwin]

Hi David,

I looked at Wyam and it looks like an interesting project.

As a C# developer, it has a certain appeal, even over typescript. That said, I have some questions about Wyam that I thought might come across as criticism, but I wanted to first say it's not meant as a slight, just me trying to understand whether your project fits into my mental picture of software. I would rather not make you create Wyam into something it is not, if the fit is not what works for me.

Questions:

1. I noticed Wyam (like webpack) seems to have a decent amount of implicit behavior. Is there a way to get it to emit all the resource files, pipelines and such into static versions so they can be inspected, then edited?
2. Is there a component pattern for Wyam vs the template pattern that seems to be shown? For me, I've always found the amount of risk of "global" state that can easily float between templates to be somewhat unsettling.
3. In my current compiled projects, I tend to rely on constructor injection for all my dependency injection with the container kept separated from my dependencies. It seems like the pattern in Wyam docs promotes a state bag policy which passes around data. Is it possible instead to create a pattern which more resembles setting up an ioc container ( somewhat dense read, but a page like this should let you know what I'm talking about: https://simpleinjector.readthedocs.io/en/latest/aspnetintegration.html ). I feel these principles are related to my question:
   https://en.wikipedia.org/wiki/SOLID

In short, I like the idea of gatsby, but quite honestly, I don't like writing in Javascript as I find most implicit logic frustrating - (Not meant to start a flame war, just my personal preference - also, yes, I'm aware of typescript). Also, I find webpack a bit frustrating to setup as it relies on so many seemingly magical behaviors that you must read the documentation, experiment, repeat until finished. I feel like a well done strongly typed language should largely direct you in the right path just by the nature of how the parts fit together.

Any thoughts you can provide would be appreciated,

John

[daveaglick]

I noticed Wyam (like webpack) seems to have a decent amount of implicit behavior.

That's sort of true. Right now Wyam is almost like two different projects.

By itself, Wyam has no implicit behavior at all. You have to configure everything by creating pipelines that string together modules which operate on documents. If you've ever used Metalsmith it's kind of like that. This proves very powerful for complex sites. For example, Discover .NET is built using a totally custom set of pipelines. In this mode, Wyam is more like a toolkit that you can use to make you own bespoke static site generator that does exactly what you need it to.

Most people find that using a recipe is easier though and is a good way to get up and running quickly. These are essentially pre-canned sets of pipelines that build a particular kind of site such as a blog or API documentation. The recipes are rather opinionated. Each one has a decent set of options for customization and while they can be extended, it's not trivial.

Note that this relationship is changing in the near future. The next major release of Wyam will eliminate the concept of recipes, introduce an entirely new application/project to fill that role, and shift using the core libraries to a more .NET-centric within a standard csproj. If you want to know where it's heading, I suggest reading #668.

Is there a way to get it to emit all the resource files, pipelines and such into static versions so they can be inspected, then edited?

No. You can view the pipelines that the recipes configure in the source code (for example, the blog recipe is here), but it's not easy code to wade through given the way the blog and docs recipes share some pipelines and those have been abstracted.

You could theoretically copy the entire code from one of the recipes, paste it into a config file with minor changes for the scripting environment of the config files, and edit it that way...but I wouldn't recommend it. If you really need that much control, creating a config file from scratch and setting up the pipelines you need is the way to go. Most of the complexity in the recipes comes from having to support so many different scenarios.

Is there a component pattern for Wyam vs the template pattern that seems to be shown?

I assume you're talking more about the front-end and themes here (if not, please clarify)? Wyam is totally agnostic to the style of front-end in use. Razor tends to lend itself to more template-driven layouts, but there's no reason you couldn't use something like Vue components for the front-end. In fact, Discover .NET does exactly that.

There's probably some work that could be done in this area around integrating with common front-end build chains like webpack. I have a couple of ideas for that in #568.

It seems like the pattern in Wyam docs promotes a state bag policy which passes around data.

That's correct, Wyam IDocument objects are essentially an immutable state bag.

Is it possible instead to create a pattern which more resembles setting up an ioc container

I'd need to know a little more about what you're thinking here to give a good answer. One of the primary motivators for introducing IoC in a library is so that the calling application can invert control and supply it's own implementations. I'm not entirely sure where that would be appropriate here. Are you thinking of swapping out specific portions of functionality such as the virtual file system implementation? Or are you thinking it would just be nice to have some more control over global state (which wouldn't necessarily mean introducing IoC)?

All that said, I have been considering providing more interfaces and allowing callers to swap out implementations if they want to for key functionality (see #336). That'll become more important as I work on #668 since callers will be an actual application and it'll make more sense for them to replace certain components.

---

Hope that at least answered some of your questions. Happy to continue discussing any follow-up you might have.

[jhgoodwin]

Hi daveaglick, thanks for the reply.

To frame my understanding of wyam, I already read probably 60-80% of the source code in an attempt to follow the flows.

In regards to how I tend to think about software, I don't generally like to rewrite software, once the software is written. I prefer to fix software by writing more modules/classes/code, but favoring leaving old code alone.

To above, it seems to me that a pipeline consistent with these ideals would contain:

• Repositories
  • These provide access to a particular kind of data. By favoring interfaces over concretions, how you discretely get access to that data is open to reinterpretation, later.
• Transformers / Adapters
  • These take one kind of shape and create another shape.
• Actions / Emitters
  • These are either eager repositories, or otherwise do some perceived useful work in a logical workflow. This could be writing files, sending emails, performing git commits, so on.

So, using terminology like above, if I was wiring up an IoC style system, it would probably look something like this:

private readonly IConfiguration _config;
...
private void SetupContainer(Container container)
{
  var blogSettings =
    config.GetSection("Root:Blog").Get<MarkDownRepositorySettings>();

  container.Register<INoSqlStore>(() => new MarkDownRepository(blogSettings));
  container.Register<IBlogRepository, NoSqlBlogRepository>();
  ...
  container.Collection.Register<IFileWriter, BlogWriter>();
  ...
}

So, the idea here is that you wire up all your constructor dependencies, then when setting up your pipeline, you might focus primarily on your sequence of operations for your actions, since they're not just a datasource:

private void AppendToPipeline(IPipeline pipeline, Container container)
{
  var fileWriters = container.GetAllInstances<IFileWriter>();
  pipeline.Add(fileWriters);
}

Maybe my suggestion is only half baked, considering I free typed it, but the point I'm making is that each stage in your pipeline is a small modular chunk which has a type-safe opinion on how to inform consumers of its value how to know they're using it properly.

I saw your mention of MetalSmith, but right in their tutorial, they do things I don't agree with. For example, taking the input blog data, then erasing entries which are draft of true. To me, this is a serious no no. Instead, I encourage my teammates to make data streams. One with all the entries, one with published entries, and published entries are simply an IEnumerable filter over the full list, which only yields for ones in which draft is not true.

In the end, my goal is that I should be able to make understandable software which is primarily composed of small bits of software. Most of it creates data streams. By arranging the data streams, giving it some data source, you can get some useful work from it (in this case, pre-rendering a blog site or whatnot).

So, regarding the state bag thing I mentioned, my preference would be that most things would NOT consume this unstructured data. Instead, you'd send it through filters which makes it super clear to the other side exactly what can be expected. In this way, no pre-known keys would be needed for downline consumers and everything would be strictly typed quite early in the processing.

Thoughts?

[jhgoodwin]

@daveaglick I spent a little more time thinking about the issues.

I dug into the experience a bit more and thought about what you said about Recipes. I agree that's where 99% of my concern comes from. I also agree that they simplify getting started. But, I think they get you started in a way that's like using C#'s DataAnnotations. Simple to do exactly what they're designed to do, but way too hard to see the path to extend them. By contrast, validators, like the https://github.com/JeremySkinner/FluentValidation are more verbose, but simple to see how you would extend them to do "just a little more".

I think moving towards a model closer to dotnet new xxx will end up helping your users build out their needs in your platform, while reducing the barriers to entry.

If you can think of a way for me to help that effort ( #668 ), let me know.

In the mean time, consider this item closed, unless someone else wants/needs to discuss it further.