Shipping the new spec

[wbamberg]

I'd like us to think about how to ship the new spec.

At the moment the "new spec" is built from https://github.com/wbamberg/matrix-hugo. I think there are a couple of barriers in getting from there to a shipped spec building from matrix-doc:

• the spec is a moving target. My version is already out of date.
• more importantly, the spec at https://github.com/wbamberg/matrix-hugo contains a bunch of changes I've made that haven't been reviewed, and trying to compare it with the original is going to be super-difficult at this point.

So I'd like to suggest a process like this:

1. we finish testing out the new spec, and decide it's what we want (or not, in which case none of the rest applies...)

2. we finish applying the design updates from Dean

3. we suspend changes to the spec (to be clear, this doesn't stop us writing and approving MSCs: just that we won't be able to update the spec itself to reflect approved MSCs).

4. we make a series of PRs to matrix-doc, and they get reviewed. Changes include:
   a) the underlying infrastructure (mostly adding /themes/docsy as a submodule)
   b) a /content directory containing the spec prose. As far as possible we can make PRs that are reviewable, but some changes, like RST->MD conversion will be pretty hard to review.
   c) a /data directory, containing spec data (OpenAPI and event schemas). Again I can do this as multiple reviewer-friendly PRs.
   d) a /layouts directory containing template code

5. eventually we're able to build the new spec from matrix-doc, and all the changes have been reviewed. Now we can publish the new spec to, say, spec.matrix.org/unstable.

6. we un-suspend spec changes, but make them now apply to the new spec organization.

7. we remove vestiges of the old spec platform

8. when we are ready, make version 1.1 of the new spec and publish it at spec.matrix.org

I think before this I ought to provide a summary of the changes I've made. Because it would be a shame to get to, say, 4b and realise that I've made, and relied on, some change that's fundamentally not acceptable.

I think one drawback of this process is that we have to reapply all the changes. This is OK for the big changes like Markdown conversion, but for small ad-hoc changes (like odd bits of RST that needed manual fixing up) it will be easy to miss stuff. But I think that's not a huge problem.

[wbamberg]

I've been thinking a bit about how to organize the work into coherent PRs:

• Hugo, Docsy, basic MD content and styling: add the basic Hugo/Docsy machinery, and layouts, and most of the spec content, migrated to MD. At this point it should be possible to view the spec pages. Client-server modules and all template calls are not supported.
• Modules: add the client-server module sources as MD files, and template to add them to the client-server API
• HTTP API templates: add the HTTP API data, and all the templates to render it
• Table of contents: add the JS-generated table of contents.
• Changelog: add the changelog sources and templates to render them.
• Event schema templates: add the event schema data, and all the templates to render it
• other templates: add all the other templates

[wbamberg]

I've been thinking a bit about how to organize the work into coherent PRs:

In the end I've gone with a different way to organize things:

• Basic machinery for the new spec platform: add the Hugo/Docsy machinery, layouts, table of contents, and one spec page, migrated to Markdown.

• Migrate all spec pages: convert all pages to Markdown and add them to the Hugo site. Replace simple templates (specifically, ones that are not data-driven).

• Support rendered data: support rendering HTTP APIs and event definitions from structured data.

• Support changelogs: add content and templates to represent the changelog.

[anoadragon453]

Here's my battleplan for getting this all into CI and deployed.

Initial goal: Build and validate the new spec for PRs, build on changes and publish to https://spec.matrix.org.

The current state of our CI is:

• When a PR is made against matrix-doc
  • CircleCI runs
    • Validates doc and swagger sources.
    • Generates the old spec
    • Hosts the docs at a public CircleCI asset link, where the PR author can view them
    • Hosts swagger in the same manner
  • Buildkite runs
    • Builds the spec
    • Generates the proposals page
    • Builds swagger
    • If the build was on master, rebuild matrix.org@master
• When a PR is made against matrix-org
  • Buildkite runs
    • The spec is built (by fetching the latest master build of the spec from buildkite)
    • The rest of the site is built
    • The site content is published at a buildkite artifact (content.tar.gz)
• Every night
  • A buildkite build on matrix.org master is run.
• When a build completes on matrix.org master
  • An instance of buildkite-webhook-listener that's running will pull and publish the current state of matrix.org/the spec.

The changes to this that I propose are:

• Ditch the nightly buildkite build, which is currently only necessary to update the spec proposals table.
  • The new spec platform has a new script for fetching proposals, which dumps proposal information to /data/msc/proposals.json.
  • A hugo template just reads this data and generates HTML based off its contents.
  • We should just run this nightly on our infra and re-run hugo to rebuild the site.
• Add a new step to matrix-doc's CircleCI to build the spec and host it at a publicly available endpoint.
• Add a new step to matrix.org's Buildkite to run hugo and output the spec to a directory.
• Switch spec.matrix.org from a redirect to instead point to the directory where hugo outputs its static files.
  ...
• Eventually retire the old spec and machinery/repo files around it.

Once we do a release of a Matrix version, we'll get some versions to play around with. As detailed in #2992 this is implemented via git branches. We'll probably need to modify the webhook listener to pick up on changes to specific branches and unpack the resulting buildkite builds to a version-specific directory. Will likely need some nginx rules to route things to the right place.

[wbamberg]

Thanks for pushing on with this @anoadragon453 ! I will have some docs in the next few days (possibly the weekend if I don't get to it first).

[anoadragon453]

Bit of a shakeup in the plan as I've become more familiar with how the whole CI pipeline fits together and how it should eventually be untangled:

New changes:

• Keep the nightly buildkite build. We'd like to avoid building the spec on the server, even if it is very cheap to do so. Instead we'll eventually stop triggering a matrix.org build on every matrix-doc build, so the nightly builds will become much leaner.
• Modify the CI of matrix-doc:
  • CircleCI should additionally host the new spec docs for each PR
    • Change guilesbot to link the new spec as well as the old for now.
  • Buildkite should additionally build the new spec and store it in a spec.tar.gz artifact.
• Currently when matrix.org finishes building, a webhook hits our infrastructure and pushes us to download a built copy of matrix.org.
  • We should add a new webhook for matrix-doc which downloads an artifacts containing the new spec (with proposals bundled).
  • We should then remove the redirect of spec.matrix.org and just host the contents of that downloaded and extracted artifact.
• Eventually start phasing out of the old spec
  • Remove build step of legacy spec from CircleCI and Buildkite on matrix-doc, and stop triggering a rebuild of matrix.org every time a build of matrix-doc finishes.
  • Remove all of the legacy spec infra files from the matrix-doc repo.
  • Start using netlify to host the temp versions of the spec, as it has better support for serving index.html and other things that require workarounds in CircleCI. I've been told matrix-react-sdk will be/is using netlify with buildkite and can crib off of there.

[richvdh]

Change guilesbot

giles, not guiles. named after a certain librarian :)

[anoadragon453]

As of now the new spec docs are available in their unstable form at https://spec.matrix.org/unstable 🎉

We now exist in a spooky world between two spec platforms, and need to eventually get back to one. I'm going to leave that to a separate issue though, as per this one, the new spec is now officially shipped.