Wyam vNext strategy

[daveaglick]

After Wyam 2.x ships, work will commence on a large shift to the Wyam architecture and the way it works to better align with lessons-learned from the last several years. I'll continue to update this issue content as my thinking evolves.

Splashdown (static site generator application)

Wyam has traditionally straddled the fence between a "static site generator" when used with one of the recipes or a "static generator toolkit" when used with custom configuration files. That distinction will be further emphasized by splitting Wyam into two projects: a static site generator application similar in nature to Hugo, Jekyll, etc. (named Splashdown) and a framework for building your own custom static generators (named Wyam).

The application will ship in a variety of ways:

• A .NET Core global/local tool (that will use the tool name splash)
• A .NET Core zip file
• A .NET Framework zip file
• Chocolaty (.NET Core version)
• Homebrew (?) (.NET Core version)

It will contain a single "recipe" that covers both of the existing blog and docs use cases. It will use Wyam libraries for most functionality (including the CLI commands) and mostly consists of the pipelines needed for the "recipe". Otherwise, the concept of distinct recipes for specific use cases will be totally removed (it was too confusing). The term "recipe" won't be used in either Splashdown or Wyam going forward. Configuration will be provided via a JSON or YAML file (user's choice, maybe other formats too) which will allow for code-free configuration like specifying global settings.

Themes will be plain old files in the theme directory and can be installed by copying files, using git submodules, or using the CLI (which will ship with some theme management commands, see below).

Splashdown.Templates

A set of .NET Core templates for different kinds of sites, for example:

• dotnet new splash-blog: scaffolds a new sample blog site with example posts and pages, config file, etc.
• dotnet new splash-docs: scaffolds a new sample docs site with example posts and pages, config file pointing to source code, etc.
• dotnet new splash-site: scaffolds a new sample site with just individual pages

Splashdown.Core

The core library used for Splashdown that contains all the pipelines. This will be separated into a library and distributed on NuGet so that custom Wyam applications can use it as a starting point to extend the default behavior instead of starting totally from scratch if they want to.

Wyam (static generator toolkit)

Wyam will live on as the engine powering Splashdown as well as whatever custom generators a user wants to build. It will use a builder model for configuring from code, similar to the one used by ASP.NET. CLI command handling will also be provided as a library so that the user's custom application can do things like serve the site on a preview host, manage themes, etc. the same way Splashdown can. Other existing Wyam libraries will continue essentially the way they are (Wyam.Hosting, modules, etc.).

Cake.Wyam

Will support both the new static site generator application as well as custom Wyam applications.

Wyam.Cli

Shipped as a library for integration into Wyam applications (including Splashdown).

• Commands:
  • build: Builds the site
  • preview: Builds the site, previews it, and watches for changes (this is different than the current preview command which just runs the preview server)
  • theme: Manages the themes folder with several sub-commands
    • install [path or repo of theme]: Installs a new theme into the theme folder, removing all files that currently exist there (show a warning if there are files already in the theme folder) - also creates a _theme (actual name/format TBD) file in the project root with the path/repo for future reference and a hash of all installed theme files
    • update [path or repo of theme]: If no path is specified, will use the existing _theme file to locate the theme source and install it, otherwise will use the provided path - before installation, compares existing theme files to the stored hashes and warns if any theme files have been changed
    • changes: Reports on changes to the theme by comparing existing theme files to the stored hashes
  • publish: native support for publishing to locations like Netlify or GitHub Pages (should be extensible so more can be added using an extension project and libraries)
  • new: generate new files from templates (similar to Hugo's archetypes)

Wyam.Templates

A set of .NET Core application templates.

• dotnet new wyam: scaffolds a new Wyam app with example pipelines, config file, pages, etc.
• dotnet new wyam-splash: scaffolds a new Wyam app that uses Splashdown.Core with the intent to extend default Splashdown behavior.

Other templates TBD.

[k94ll13nn3]

It seems like it is a great idea for the future of Wyam, especially for the debugging/IntelliSense part. Having a compile time check for the configuration or the recipes would be a huge improvement. I like the idea of going on a ASP.NET Core like model.

However, as mainly a user of Wyam in a CI context, I have some difficulty to see how it would change the experience : at the present time, when I want to update my static site, I update the input files, and I use the executable (through Cake) to generate a new version of my site. How would this change with the new version ? If I understood the proposition correclty, i would have to build an executable each time I want to do some modifications ?

Aside from this, theses changes looks good.

[MisinformedDNA]

I completely agree with you that we need to make use of preexisting and familiar tools. The more Wyam resembles a standard ASP.NET project and process, the easier and better it will be.

I would like to see a "normal" VS/dotnet build, output static files instead of DLLs. Any custom build components should be built into the project type/configuration. (i.e. dotnet build mywyamsite)

I'm not sure why we would need an exe for each project. Are you suggesting that the config helps generate a custom exe and then we use that exe to build the site (i.e. mywyamsite.exe build)?

I would love to see us use an ASP.NET project with UseStaticFiles, so we can store our themes in wwwroot while keeping our MD files and partials in the Posts and Pages directory.

I think the setup should resemble an ASP.NET Razor Pages project as much as possible. This way, if someone wants to add dynamic functionality, they still can, while those who just want the static content can just grab the output directory.

So Wyam would be added using something like

app.UseWyam(options => options.WithRecipe(new BlogRecipe().WithXXXX(YYYY))

Also, instead of a pipeline, I'm thinking that we would just use Middleware, which already forms a pipeline.

[LokiMidgard]

I like the idea of better tooling. And this way nobody needs to create it from scratch.

Instead of build we now need to run our project. I think I can get used to it when I get debugging :)

You could still use some pre-generation stuff to instantiate modules using a code generator.

[tbolon]

It seems this system will make Wyam less easy to use for non-technical users (or even technical users without a c# background).

As a .net developer, I can feel all the benefits of the classic csproj format, but I can't refrain myself from thinking that a lot of users will not be able to use the tool or fix any compilation issue they could face (not closing all parenthesis, difference between ',' and ';', ...).

How will you be able to document how to add a pipeline manually, customize a theme and so on ? I can easily think of a developer-oriented documentation, but something on the level of jekyll seems to be hard to achieve (or I can't perceive it).

(Also note I have just discovered wyam with the launch of discoverdot.net, I have already used jekyll and some other tools, and I am evaluating wyam to convert some of our websites, so I don't have an expert eye on the tool).

[MisinformedDNA]

We would only need "compilation" for advanced scenarios, like Pipelines (which is taking the text and compiling it anyway, right?) And even that would only be necessary during initial development. Once all that was done, adding a new page would be as simple as creating a new page and running a single command from the command line. But your point is well received, I would love to see Wyam match Jekyll in its simplicity and power, but still making it even more awesome for those who want tooling, especially for initial development.

[daveaglick]

Thanks for the comments everyone - this is great because the dialog is helping clarify some things for me. Here's some initial thoughts...

---

@k94ll13nn3

However, as mainly a user of Wyam in a CI context, I have some difficulty to see how it would change the experience

I don't think it'll change much at all, especially if you're already using Cake. The Cake addin will change under the hood to call dotnet run instead of wyam.exe but otherwise the experience will be the same in this scenario. The Cake addin will still produce the CLI arguments Wyam expects and the dotnet run command will build your generator app (if it needs to be built) and run it.

---

@MisinformedDNA

I would like to see a "normal" VS/dotnet build, output static files instead of DLLs.

A VS/dotnet run would result in static files being generated. In that sense, this isn't too much different than what we're doing today. It's just that instead of your code going in a script and then wyam.exe "compiling" the code, your code will go in a project and MSBuild will compile it.

I'm not sure why we would need an exe for each project. Are you suggesting that the config helps generate a custom exe and then we use that exe to build the site

Yes, except the config part. There won't be a config.wyam file anymore. All the code for setting up your pipelines, loading a recipe, etc. will go in the Main() of your app (or go in other classes that get referenced from the main entry point). Under .NET Core, building and running your generator app would happen when you call dotnet run - so in a sense, the execution phase is very similar. Instead of wyam.exe you call dotnet run and out comes static files.

I think the setup should resemble an ASP.NET Razor Pages project as much as possible.

This probably won't happen directly - one of the guiding principles of Wyam is that it's an anything generator that most folks just happen to use for web sites. I don't want to get away from that and modeling the layout and structure (in addition to the tooling) after ASP.NET is probably too web-centric.

That said, I could see developing a package that contains Wyam middleware for ASP.NET (or maybe MSBuild tasks) that does exactly what you're talking about and lets you use Wyam to process static files in your dynamic ASP.NET application.

Once all that was done, adding a new page would be as simple as creating a new page and running a single command from the command line. But your point is well received

Yes, exactly. The app is essentially your custom, bespoke static site generator. As with other generators, you don't need to recompile the generator to add new content or pages, only to change the features of the generator itself.

---

@LokiMidgard

You could still use some pre-generation stuff to instantiate modules using a code generator.

As usual, you're already on my wavelength 😄. The trick would be making it seamless to users - perhaps as an MSBuild pre-build task. Another thought was to have modules expose an extension method on a common dummy object so at least you'd get Intellisense for all the available modules. Even better would be extension properties, but looks like the wait will be a bit longer for those.

---

@tbolon

It seems this system will make Wyam less easy to use for non-technical users

You're absolutely correct. That's my biggest concern with this change. The whole scripting DSL was created specifically to make onboarding for non-.NET folks easier (which is why it does things like hide common lambda expressions behind the @doc and @ctx syntax). This will also require a little knowledge of the .NTE project system, or at least how to open a .cs file whereas the current way just requires you to edit a script file.

That said, I suspect the idea that Wyam would ever catch on beyond the .NET community was naive. There are already great static site generators for the masses with huge contributor pools and giant usage numbers. I mean, Gatsby just scored over $3 million in funding last week - that boggles the mind. This change doubles-down on .NET developers while admitting that's who the project is mainly for. I think I'm okay with that.

[phillipsj]

I like the sound of the changes, my biggest concern is it is moving away from how most static blog generators already work. Most execute similarly to how Wyam currently works. This change does open up the ability to create a CLI on top that adds a little additional functionality that doesn't currently exist like generators for posts, drafts, pages, etc.

The change also seems to make it easily integrate with asp .net core sites which would be good for several workflows.

As a user, I feel like it opense up to evaluating if I want to stay on Wyam or explore other options that provide cross platform support, why I was looking forward to 2.0, and an additional want was not having to need the .net runtime installed.

I look forward to getting to try it as I think this is the best .NET generator out there.

[MisinformedDNA]

I do think we still need to support config files, but keep them optional. I'm sure there will be lots of different reasons not to hardcode values, reusability being one of them.

[RLittlesII]

Also, instead of a pipeline, I'm thinking that we would just use Middleware, which already forms a pipeline.

@MisinformedDNA is this suggesting a refactor of all the pipelines to aspnetcore middleware? That's an interesting concept

[MisinformedDNA]

@RLittlesII I originally thought we could do an ASP.NET Core project, but we need an app that runs, transforms and returns, so a Console project is more applicable. Whether we can use ASP.NET Core Middleware inside a console app, would be an interesting study. ASP.NET Core was made modular, so the possibility is there, but we'd have to see. Reusing pre-existing, familiar frameworks/modules is generally a win/win, if it makes sense.

[daveaglick]

@RLittlesII @MisinformedDNA

It's definitely an interesting thought experiment. What I can say with pretty high confidence is that Wyam's pipelines (or probably modules is more accurate) won't change to ASP.NET middleware. IMO, ASP.NET middleware too tightly coupled to the idea of a web server serving a single request, and I'm not a huge fan of the amount of magic that is involved in passing data between them.

That said, I would be curious to see what it would look like to write a generic ASP.NET middleware wrapper around Wyam's IModule. It would need to convert the input middleware data to an IDocument and then convert it back once the module is done. I wonder what the use cases for such a wrapper would be...

Also...

I do think we still need to support config files, but keep them optional

My thinking here is that tools like dotnet-script would fill this need nicely. You'd reference the new Wyam.App package (or whatever we call it) from the script and then write the code in the script just like you would if it were an app. Voila, scripted Wyam with a config file. And I don't need to maintain a script host anymore 😄

[MisinformedDNA]

To make sure I understand you, what do you mean by "maintain a script host"? Are you referring to wyam.exe?

And then how does dotnet-script simplify things?

Finally, I thought we wanted to create a custom generator, in the form of an exe, for each project. C# scripts will not create that for us.

NOTE: Azure Functions for VS was once built on C# scripting, but everyone kept asking for features that only existed in normal C#, so they changed their course and focused on C# "Classic".

[daveaglick]

No problem...

To make sure I understand you, what do you mean by "maintain a script host"?

Wyam uses a custom scripting host that compiles and evaluates your config.wyam file using Roslyn. Roslyn actually has a out-of-the-box script host, but it's quite limited and wasn't enough for Wyam. Note that I'm using "script host" in the general sense that it's an engine that can run C# code without using MSBuild or a project file - not to be confused with the more specific C# Scripting capability which is a part of Roslyn and is itself a "script host".

The Wyam script host does a bunch of stuff:

• It's responsible for interpreting your preprocessor directives like #n before doing anything else
• It installs NuGet packages, resolves dependencies, and figures out which libraries to reference
• Sets up the Roslyn context for script compilation
• Generates a syntax tree and then walks it to convert some of the custom Wyam DSL stuff to standard C# (like the special @doc syntax)
• Creates a special wrapper class and puts your code in a method inside of it
• Lifts statics and classes outside of the wrapper method and into the parent
• Scans all referenced assemblies for IModule implementations and then creates global properties for them
• Compiles the wrapper class and any lifted static and class definitions into an in-memory assembly
• Uses reflection to execute the wrapper method (that contains your script code) from the in-memory assembly

That's...a lot. And it's all Wyam code. I'd be happy to make most of that someone else's problem.

dotnet-script already does all those things (except for the custom Wyam DSL bits, but that was probably a bad idea to begin with). In fact, the script host in that project and the one in Wyam are very similar feature-wise.

It's also worth point out that Wyam is already usable as a library without going through the Wyam script host today. I know for a fact that some folks even prefer to use it from LINQPad. The main difference with this proposal is that the scripting capability will go away and embedded use will become the preferred method along with some additional APIs to better support it.

Finally, I thought we wanted to create a custom generator, in the form of an exe, for each project. C# scripts will not create that for us.

That'll be the end result for the most common scenario once Wyam is only shipped as a library (as described above). However, the user just needs a way to write, compile, and run code to use that library. If they'd prefer do that via a scripting tool instead of MSBuild/project file, that's just fine. In that mode it won't look much different than how it does today.

[tbolon]

I was thinking suggesting using scriptcs as a script host as a simpler alternative of csproj + cs. Initially, I supposed they were the same (using the same .csx extension) but it seems it is not the case (despite having some contributors in common) dotnet-script seems to be the new engine.

I can totally understand the need to remove all code which is better maintained by other projects to focus on core features. The hosting part is clearly a good candidate.

I would suggest to try using dotnet-script with a wyam.csx script file as a new config/engine handler, and write some test scripts to see how different scenarios could be written this way.

In the most simplest way, you will need to run commands such as dotnet script wyam.csx to build content. There will be no way to scaffold a new website without creating and installing you own custom tool (which could also rely on dotnet-script itself ?). Perhaps this new global tool should only allow to make contributor life easier, not being mandatory to build the content.

Scripts could looks like this:

#! "netcoreapp2.1"
#r "nuget: Wyam.Scripting, 1.0.0"
using Wyam.Scripting;

// static "Settings" class with static indexer
Settings["Host"] = "example.com";
Settings["Title"] = "Example Site";

// static "App" class which builds a single website
// App.Use<BlogRecipe>();
App.UseBlogRecipe();

App.Use("Markdown", p => p
  .Read("*.md")
  .FrontMatter(Yaml())
  .Markdown()
  .Write(".html")
);

// mandatory to run the app
App.Run(Args);

[MisinformedDNA]

I like where this is going, but I'm still unsure about the theme command. Wyam is good for templating (preparing recipes) and generating static content and I think theme takes away from that. There are already lots of tools for getting CSS and JS into a folder: npm, yarn, LibMan (upcoming VS tool) or just zip files.

Note: I verified that Jekyll also does not make use of a special theme command. They just instruct their users to use gem.

[LokiMidgard]

I'll guess that the watching feature that automatical builds when input changes will no longer work when transformed to an msbuild project?

[daveaglick]

I'll guess that the watching feature that automatical builds when input changes will no longer work when transformed to an msbuild project?

That'll still be a feature, but it'll be delivered via the local per-project CLI tool that gets pulled down via NuGet (using a DotNetCliToolReference). You'll be able to write wyam preview and the CLI tool will watch for changes and re-run your generator. Still need to work out how caching from one build to the next should work so that generation is faster after changes, but that's solvable.

[MisinformedDNA]

I'll guess that the watching feature that automatical builds when input changes will no longer work when transformed to an msbuild project?

@LokiMidgard, see https://docs.microsoft.com/en-us/aspnet/core/tutorials/dotnet-watch.

[daveaglick]

FYI to the watchers of this thread - I've edited the main issue a couple times and will probably do so a bit more. I've been going back and forth for months on a few ideas with regard to how the CLI is shipped and custom code is integrated:

• Ship a full-featured CLI that can build a site without anything else and allow the user to create a library project that extends the build process
• Ship a lightweight CLI that mainly acts as a proxy to a required user application which references a main Wyam library
• Don't ship a CLI at all, instead the required user application takes care of everything by linking to a Wyam library with all the housekeeping functionality (dotnet run ... and/or dotnet myapp ...)

You'll notice all these options make use of a custom user application - the big question is when is it required (just for extending vs. all the time) and what kicks off generation (a CLI app vs. the user app). Hopefully getting close to settling on an approach. Bear with me :)

[ghost]

How far away are we from the don't ship a CLI at all sub-goal?

It'd be awesome to have at least that feature until someone finds
the time to write a (potentially extensible) CLI someday.

[RLittlesII]

@daveaglick I noticed earlier in the thread we made reference to json or yaml configuration. I was wondering, is your thought that Wyam configuration would go away?

I was also thinking that if it does not, having the ability to pull the wyam.config into visual studio and make it a first class citizen with intellisense might go a long way towards support. I know a lot of my question around Wyam come from what are the correct API's to use to customize my build. If I had intellisense, I could browse the public API like any .NET library and maybe find my way a bit better.

[daveaglick]

I was wondering, is your thought that Wyam configuration would go away?

Yes. At least in it's current scripted form. The idea is that the new SDK-style .csproj files are so light, and the dotnet tooling like dotnet run so good, that we might as well make custom configurations a full-blown project and toss all the custom scripting, NuGet API usage, etc. That'll be a huge maintenance burden off my back, allow me to focus more on features, and still get Intellisense and other editing capabilities folks have been asking for since day 1.

The alternative would be to build a language server or Visual Studio addin for Wyam config files - that's doable, but watching the Cake folks figure this out has given me an appreciation for how hard it is. I'd rather focus my effort elsewhere.

[RLittlesII]

I down vote language server and visual studio addin

[daveaglick]

I've continued to refine my ideas about what comes after version 2.0 and have updated the initial issue to reflect my current thinking.

One thing I want to point out separately though (and get some feedback on) is the idea of splitting Wyam the static site generator from Wyam the static generator toolkit.

The former would become it's own application (possibly called Celilo) and would encompass what is now the blog and docs recipes in a single, pre-configured experience (think traditional site generators like Jekyll, Hugo, etc.). It will be configurable via JSON, YAML, etc. but will not support custom config files or C# code (outside of Razor files). Essentially, it would be similar to other static site generators.

Wyam the static generator toolkit would live on as a series of libraries with a builder pattern for creating your own application, similar to the way ASP.NET works. It would still include all the modules, pipeline support, etc. we have now. To create your own generator (which I used to call a custom configuration, though I think that name is confusing) you would write a .NET application that uses the Wyam libraries.

I think this split has a few advantages:

• It would allow for separate web sites and documentation for the static site generator and the libraries making the distinction clearer.
• It would reduce confusion over what you get with Celilo vs. what you get with Wyam and generally make the distinction between "recipe" vs. "custom config" much clearer than it is today.
• Celilo would serve as a good example of how you might build your own static generator using Wyam.
• It'll be easier to write the new tool from scratch and use it as a proving ground for the shift in Wyam to use in your own application.

[tbolon]

I agree that a solution for both paradigms (expert tooling & config vs ease-to-use) was difficult to find. I think your approach of splitting the projects in two is nice (even changing the name).

So you expect Celilo to be the "brand" (as Jekyll, hugo, etc.), and Wyam the name of the toolkit/technology behind ? Do you expect to have one single website for both ?

What will certainly be hard will be to find an easy path for users who started a website with Celilo then want to migrate to a full custom app. Celilo will both require to be easy to use (consumer) and easy to understand when browsing source code. Perhaps a "dotnet new celilo" project template could be a nice feature.

[daveaglick]

@tbolon Was updating the parent issue and you beat me to it :).

Yes, the intent is to totally separate the two into distinct projects (both under the Wyamio organization), each with their own website, brand, etc. Something like "Celilo, powered by Wyam".

What will certainly be hard will be to find an easy path for users who started a website with Celilo then want to migrate to a full custom app.

We'll be shipping a Celilo.Core library that lets you hook the Celilo logic into your custom app as a starting point. That way you can choose to extend the behavior from Celilo or start fresh from scratch.

Perhaps a "dotnet new celilo" project template could be a nice feature

Totally agree - planning on two sets of project templates, one for Celilo sites and another for getting started with Wyam applications.

[daveaglick]

Finally settled on a name for the new static site generator application. In the end, I think Celilo was too difficult to remember (though maybe I'll use it for something eventually). Instead I'll call the new generator Splashdown. It follows the whole water/waterfall/waterdrop theme of Wyam and has a linguistic similarity to Markdown which is nice. Not to mention it seems to be totally unused in the OSS space and a domain (splashdown.io) was available.

[jhgoodwin]

@daveaglick Regarding my comments in #754 , I created this:
https://github.com/jhgoodwin/wyam-blog-template

IMO, this would have been an easier starting point for me than the original blog recipe.

If the config.wyam was converted into a normal console C# project, would this be your vision for the changes discussed in this issue ( #668 ) ?

[daveaglick]

@jhgoodwin That’s really cool! I’ll refer people to it when they want to edit the existing recipe. I agree it’s gotten too complex - I even have trouble working on the recipes right now.

The vision is to split Wyam into a separate application (Splashdown) that’s a combination of the current blog and docs recipes without the ability to customize via a config file. In other words, Splashdown will be analogous to other generators like Jekyll and Hugo with settings and file-based customization but no coding.

Then Wyam will refer to the collection of libraries that you can use to build your own generators in a standard C# project and code. In that sense it’ll be more similar to what you put together (only in a normal project and source files instead of a config script).

Hopefully during the process of combining the blog and docs recipes for Splashdown, I can simplify those pipelines by a lot since we won’t need as much abstraction as with multiple recipes. Then if someone wants to use that as a starting point it’ll essentially mean just forking Splashdown and taking it from there.

[daveaglick]

Probably fair to say this is done :)