Add a github action

[Veetaha]

Added a github action template that will allow anyone to include marker into their CI using the following code:

- uses: rust-marker/marker@v0.3

This PR encompasses several important changes.

New action.yml template

This is a dead simple composite github action template file that just runs a couple of bash lines. It is cross-platform, meaning the same syntax may be used for both windows and unix. The action handles the selection of the bash or powershell script internally.

It runs cargo marker command by default. That can be disabled with install-only option if there is some more elaborate use case than this. The action is pretty conservative in its API, as it exposes just this single input parameter for now. I believe in the future its API may grow. For example, it we may have lints input variable that would use toml-syntax to pass that as --lints input parameter. But I'm not sure that is actually needed. I believe that will be a really rare use case. 99% of the time simple cargo marker will suffice. We'll hear from anyone who uses the action if they want more builtin configurablity for the run command, otherwise if they are stuck they may use this syntax:

- uses: rust-marker@v0.3
  with: { install-only: true }
- run: cargo marker --any-command-you-want

The action will install the version of marker specified in the git tag it was referenced with. To make the syntax higher work I had to amend the releases flow to support the sliding v{major} and v{major}.{minor} tags.

New sliding v{major} and v{major}.{minor} tags.

I updated the release.md documentation about this.

A bunch of improvements and most importantly compatibility fixes in the installation scripts

I knew the installation scripts will break on Github Actions, because I didn't test them there. I only tested them on my home Ubuntu and Windows environments. So no wonder they broke when I ran them as part of the Github Action on CI on various OSes that Github Actions support today.

I also improved logging output of both scripts. They now have a consistent logging style for both bash and powershell. You can see example output on CI in my fork where I tested the Github Action.

New CI tests for the Github Action

I extended the current CI with tests for the Github Action. With the new github-action-test CI job we test both the Github Action and the installation scripts.

Right now these jobs fail on rust-marker CI and they will be red until 0.3 is officially released and we have the pre-compiled binaries that these CI jobs actually depend on.
I would like to leave this job like this even if it is red for now. It won't block the merge of any PRs unless you add the new CI jobs as required in the repo settings, and this is what you should do after 0.3 is released.

The CI jobs run on every OS that Github Actions support today, and we'll need to maintain the list of runs-on platforms used in the job matrix manually. Unfortunately there isn't a syntax to expand the list of all OSes in GitHub, but anyway, this way it makes CI more stable, and it makes more explicitly that we need to switch our OSes support when a new one is added and we have to update the CI code.

Documentation

No documentation was updated yet, mainly because the action isn't usable yet because we don't have pre-compiled binaries released.

However, I'll update the docs in #255 a bit later this week

Call to action

I'd like to ask you to announce the 0.3 release on Reddit only after you sanity check that the installation scripts and the Github Action work for this release manually. This is because I tested all of the things in my custom fork, and I had to do some manually replacements of the URLs and tags in my fork to make it work there temporarily and returned my changes back to rust-marker and 0.2.1 manually. Hopefully, I didn't make a mistake there, but a double-check won't hurt.

You may test that by running the following 4 versions of the installation scripts with the following URLs to also check that the sliding tags and master work:

https://raw.githubusercontent.com/rust-marker/marker/v0.3.0/scripts/release/install.sh
https://raw.githubusercontent.com/rust-marker/marker/v0.3/scripts/release/install.sh
https://raw.githubusercontent.com/rust-marker/marker/v0/scripts/release/install.sh
https://raw.githubusercontent.com/rust-marker/marker/master/scripts/release/install.sh

You may also install powershell if your don't have Windows, or if you have run powershell versions of these as well.

For the Github Action you may just create a draft PR in marker repo itself you may replace the uses: ./ in the new CI job with the following combinations:

uses: rust-marker/marker@v0.3.1
uses: rust-marker/marker@v0.3
uses: rust-marker/marker@v0

The most important is @v0.3 variant, because this ensures the updates of that tag will be compatible, and users will be recommended to use this syntax.

Github Marketplace

I also figured out that you may publish the action to Github Marketplace. It's the website that Github uses to promote Github Actions. It often appears in search results in google when you search for some github action. It should be a good source incoming interest. To publish the action to marketplace you actually need to have a Github Release with that action. I'm not yet sure how exactly it works. Hopefully you'll have to do this once. So once you create a Github Release, you should go to edit that Github Release, and there you will see a checkmark at the top that suggests publishing the Github Action to the Marketplace like this:

(actually if you do this for the first time it will ask you to accept some terms-and-conditions, but I suppose you'll figure this out).

I didn't try to publish the Github Action from my fork for testing, because I'm not sure if I can remove it from there and I wouldn't like to occupy the name of the action.

[Veetaha]

I mistakenly created a PR in this repo yesterday, when it was not ready for review.
Now it's ready for review

[xFrednet]

For example, it we may have lints input variable that would use toml-syntax to pass that as --lints input parameter. But I'm not sure that is actually needed. I believe that will be a really rare use case. 99% of the time simple cargo marker will suffice.

I like the idea of exposing the lint parameter, but agree that this will probably not be used anytime soon. Right now, it's the technical idea and elegance of the idea, that I like. The current setup sounds excellent to me :)

Unfortunately there isn't a syntax to expand the list of all OSes in GitHub, but anyway, this way it makes CI more stable, and it makes more explicitly that we need to switch our OSes support when a new one is added and we have to update the CI code.

Do you know if there is a repo or something we can subscribe to, to get notified about relevant changes like a new OS being added? (Without our inbox being flooded with unrelated notifications.)

No documentation was updated yet

The screenshot shows that it discovered a readme file, related to the action template. I'm assuming that it find the main readme of the repo. Do you know if it's easily possible to use a different file, which focuses on the GH action instead?

A related question: Numerous other projects have the action template in a separate repo instead of their main one. Have you considered the tradeoffs between having it in this repo or creating a separate one? Updating the version and tag is obviously a lot easier, since this repo already has the infrastructure for it, but I wonder if there are other things 🤔

I'd like to ask you to announce the 0.3 release on Reddit only after you sanity check that the installation scripts and the Github Action work for this release manually. This is because I tested all of the things in my custom fork, and I had to do some manually replacements of the URLs and tags in my fork to make it work there temporarily and returned my changes back to rust-marker and 0.2.1 manually. Hopefully, I didn't make a mistake there, but a double-check won't hurt.

Yes! Something like this has been on my todo list. I plan to migrate our lint crate template as well, which will act like a second sanity check. That crate has already help to catch so many bugs. 🐛

For the announcement: I still plan to do the announcement for v0.3.0 on Reddit to reach a larger audience. But I intended to hold off until a few days after the release. At first, this wasn't my plan, but I believe this to be the better approach than to rush the release, template migration and testing. People will also focus on the new Rust release first. My current dream schedule is as follows:

1. Push the last breaking changes and CI updates into the finish line
2. Publish v0.3.0 on 2023-10-05
3. Test everything and migrate the template
4. Add more documentation if there is time
5. Announce Marker on Tuesday 2023-10-10

I think having those five days will be a good thing, while not putting it too far into the future.

I also figured out that you may publish the action to Github Marketplace.

That sounds awesome. As usual, thank you for the excellent documentation!

---

I'll try to review this soon, but it might need to wait until the weekend. RL is currently taking up sometime, and I'm also trying to open the last big breaking changes PRs. (I'm debating of moving marker_utils into marker_api after the discussion we had in the other PR, but that's probably a discussion for a different PR :))

A sincere thank you for this PR, you have basically single-handedly reworked Marker's CI and important infrastructure in the last couple of weeks. ❤️

[Veetaha]

Do you know if there is a repo or something we can subscribe to, to get notified about relevant changes like a new OS being added? (Without our inbox being flooded with unrelated notifications.)

Yeah, there is this repo, but seems to have quite often releases with just pure tools version bumps, which is going to be quite chatty. GitHub also has twitter and own blog, but they also post ton of assorted posts there almost every day. So I would just be unnotified about this. I don't think it's very important for the marker Github Action tests to stay very up-to-date with the new Github Actions OS images. New images of major OS versions are released very rarely. Also, for example, when ubuntu 24.04 comes out I'm sure you'll know about that, and you'll know to expect GitHub to pick that up soon for the runners. So this comes down to just lazily checking the availability of now OS images every half a year or less.

Do you know if it's easily possible to use a different file, which focuses on the GH action instead?

Unfortunately, not, the README for the marketplace is not configurable.

Have you considered the tradeoffs between having it in this repo or creating a separate one?

Yes, I did. I decided to keep the action in the same repo. I know GitHub recommends using a separate repo for the ability of versioning and releasing the action separately. But having the action in the repo is easier at least at this stage. Also, given how simple the action actually is, I think it's fine to couple its release with the release of other marker code.

I know of at least of crate-ci/typos hosting the Github Action right within the CLI tool repo, same for cargo-machete.

On the other hand cargo-deny-action is separate from the main cargo-deny repo, and I find it confusing for me, because the action has versioning different from cargo-deny itself. Also having a separate repo means you can't do code changes to the action and the installation scripts at the same time, so it brings more complexity in synchronizing changes. I generally like the synchronous versioning where cargo-marker, rust crates, the github action all have the same version. It makes it easier to reason about for users, although couples the release of them (so you can't make a patch to the action without releasing cargo-marker and crates). Let's see how it works and if there will be pain in this regard it's possible to just create a new repo for the action and ask everyone to update the action references.

I may not see the whole picture because I didn't develop a GitHub Action before, but for this simple action this should suffice.

---

Thank you for your great effort on the API as well ❤️

[xFrednet]

This version looks good to me, I have a few notes and still want to test the PowerShell script, before merging this. I should be able to test it sometime this weekend :)

---

I have one comment on the install.sh script, which is unrelated to the changes made in this PR. I wanted to test the modified script and ran sh ./install.sh and got an error. This StackOverflow explains that sh != bash. Now, I wonder if the file should have a .bash ending instead of .sh. I know that unix doesn't care too much about file endings, but as a user I would first run sh instead of bash. 🤔

For reference, this is the error I got:

$ sh ./install.sh 
./install.sh: 15: set: Illegal option -o pipefail

While $ bash ./install.sh works perfectly fine :)

---

Either way, thank you for writing this action!

[Veetaha]

I've checked the file tree under the action path in a test CI run. There is no .git repository there, but there are all files for the repo there, so this isn't a spare checkout. I suppose github could use some internal API to download the repo without the git cruft.

The resulting repo size is 2.6 MB (see output from du below). The download happens in the job setup stage, but there are no details of how that is implemented, but since it takes under a second that's not a problem today.

[xFrednet]

This version looks good to me! I tried it out, and everything works like a charm 🎉

I would like to hold off on merging this, since it will make the CI and master red. I'll probably merge it on Tuesday or Wednesday if that's alright with you?

You can already update the documentation in #255. I plan to merge that PR shortly before the release.

[Veetaha]

Okay, I guess it's fine. Hopefully there won't be merge conflicts, but if there are any I'll try to fix them. If I don't respond quickly enough you have write access to my fork according to this setting on the PR (which is by default though):

I already updated the docs in #255, you can take a look there.

[xFrednet]

Now that I kicked off the merge of #262 I think we can merge this as well. It won't make a difference, and that's one less thing to keep track of. Sorry for the back and forth. Would you mind rebasing one last time?

I have one last big PR planned before v0.3.0 that will move all semantic structs from the marker_api::ast to a new marker_api::semantic module. The CI won't matter for that PR :)

Btw. Do I remember correctly, that the release CI pushes the commits directly to master? In that case, I have to check the settings that the branch protection won't prevent that :)

[Veetaha]

Yes, you are right, you need to make sure that maker CI account can bypass the branch protection rules for master. There is definitely an option to make exceptions for particular users.

[Veetaha]

I rebased an squashed everything into a single commit.

[xFrednet]

Perfect, thank you very much ❤️

[xFrednet]

Yes, you are right, you need to make sure that maker CI account can bypass the branch protection rules for master. There is definitely an option to make exceptions for particular users.

I found this. The required status badge thing confuses me a bit. Maybe it buffers the commit and runs the CI on it? IDK, I'll read up on it until Thursday and maybe do an early check by adding @xFrednet and pushing something with that setting. :)

[Veetaha]

I think this is the right setting according to the settings in other repos that I have. It should work