LLVM release workflows

[mutantcornholio]

I've been testing these on a fork with public runners.
Windows, unfortunately, still needs work, and linux fails on a 6h timeout, when executed on ubuntu-latest 😅.
But as at least some of it works, let's kick off the review, and hopefully, I'll be able to finish testing workflows and windows LLVM release in parallel (given the duration of the feedback loop on these tasks)

[xermicus]

In #201 it was discussed to not do LLVM releases, now I'm a bit confused 😅 Do you plan to have separate workflows, but the binary release workflow is depending on this first? That would make sense to me (i.e. a release tag is first causing the LLVM workflows to run, and afterwards the binary is build using those artifacts - but the LLVM artifacts and the binary artifacts end up in the same release).

linux fails on a 6h timeout, when executed on ubuntu-latest

Yeah those are absolute potato VMs, I suggest using our parity runners.

[mutantcornholio]

In #201 it was discussed to not do LLVM releases, now I'm a bit confused 😅

I read the discussion there and I see no contradiction. #201 ended up a bit ahead of me (while I was, or rather still am, stuck with mac / windows LLVM builds), so Evgeny did a temporary solution in a single workflow.
I don't see why we should release LLVM on each release of revive.
Even with cache, it seems a kinda wasteful.

[xermicus]

I don't see why we should release LLVM on each release of revive.

What you are suggesting is that we use two tags to trigger either the LLVM build or the resolc binaries release build, right? And the resolc release workflow always downloads the latest (or specified) LLVM build for each platform, instead of compiling it on it's own.

LLVM is just a build artifact. In that sense we don't release LLVM. We depend on it and attach it to the release as artifacts for convenience reasons. Having a dedicated tag and release for LLVM vs. releasing the artifacts together with the binary release seems mostly bikeshedding to me, with dedicated release increasing complexity.

Even with cache, it seems a kinda wasteful.

I'm unsure about the added complexity is worth it (still unclear to me what exactly we gain from dedicated LLVM releases), but if you think of it as the better solution go for it. Keep in mind that every polkadot-sdk PR likely uses more compute than a full release build for all platforms.

[athei]

LLVM is just a build artifact. In that sense we don't release LLVM. We depend on it and attach it to the release as artifacts for convenience reasons. Having a dedicated tag and release for LLVM vs. releasing the artifacts together with the binary release seems mostly bikeshedding to me, with dedicated release increasing complexity.

If we don't create a release where would we store the LLVM artifact so that it can be downloaded by the revive release?

[xermicus]

We just cache it as we do currently.

[athei]

We just cache it as we do currently.

Ahh okay you mean basically no new workflow and we just have #201 ? To be honest I like that we have those rare LLVM tags and don't have to deal with it in the other workflow.

[xermicus]

Yeah this is what I meant. But having it separate is also fine for me.

[athei]

So tl;dr we are fine with this PR and just waiting for macOS and windows to get fixed?

[xermicus]

Yes I'm fine, waiting for everything fixed and main merged and then I'll give it a proper review.

[mutantcornholio]

MacOS does work already, but Windows builds are inconsistent. Trying to figure them out.

[mutantcornholio]

Should we maybe go forward and do the first iteration without windows? The issue with windows is not trivial, and the feedback loop on windows runners is two hours.
If we can have llvm release pipeline working for linux and mac, we can go forward with integrating it with testing pipeline and releasing pipeline for linux and mac, which would bring the end result closer.

[athei]

Use msvc since you are building on windows. It is the most compatible and easiest to build for natively. No reason to pull mingw.

[mutantcornholio]

As a matter of fact, I've started with msvc and failed. The suggestion from @xermicus was to look for inspiration in how matter-labs do their windows releases, and they use mingw

[mutantcornholio]

I think this was the issue that I've been hitting with MSVC

[athei]

Have you read this comment: llvm/llvm-project#47189 (comment)

I would be really surprised if nobody could build LLVM with the Microsoft toolchain.

[mutantcornholio]

I would be really surprised if nobody could build LLVM with the Microsoft toolchain.

I wouldn't be surprised if nobody has built matterlabs fork of LLVM with MS toolchain, as they're using mingw themselves.
Given the sort of the issue I'm getting right now, however, I'm inclined to give it another go.

[athei]

The comment I posted sounds like the issue with MSVC just happens under some rare circumstances you need to rule out. Like just setting some CMAKE var.

[athei]

• Windows should use MSVC
• macOS needs to be a universal binary: Build twice: Once on a Intel runner and once on a AppleSilicon runner. Then use lipo to merge the two binaries. You can also build both on any macOS runner since cross compiling in macOS is easy.

[athei]

Should we maybe go forward and do the first iteration without windows?

To me it does not make sense to ship this incomplete. We don't gain anything from that because we don't even have the revive workflow done. So nobody could do anything with those artifacts anyways.

The issue with windows is not trivial, and the feedback loop on windows runners is two hours.

Switching to the default toolchain on windows (msvc) hopefully makes it easier.

[mutantcornholio]

Should we maybe go forward and do the first iteration without windows?

To me it does not make sense to ship this incomplete. We don't gain anything from that because we don't even have the revive workflow done. So nobody could do anything with those artifacts anyways.

I don't get it. These artifacts will be used in the revive workflow. Revive workflow depends on it, not the other way around. Why would revive workflow be a blocker here?
What I'm suggesting is to finalize LLVM workflow, use the artifacts in revive release and test workflows at least for one platform, which should make the other platforms easier, as we'll hit some of the same issues (like, artifact size limit).

The issue with windows is not trivial, and the feedback loop on windows runners is two hours.

Switching to the default toolchain on windows (msvc) hopefully makes it easier.

Looking a the previous roadblock that I had with MSVC, I can give it another shot, especially as my current issue with mingw happens when compiling builtins, which matterlabs don't seem to be doing.

• macOS needs to be a universal binary: Build twice: Once on a Intel runner and once on a AppleSilicon runner. Then use lipo to merge the two binaries. You can also build both on any macOS runner since cross compiling in macOS is easy

Ah, right. I'll get to it in parallel to working on windows 👍.

[athei]

I don't get it. These artifacts will be used in the revive workflow. Revive workflow depends on it, not the other way around. Why would revive workflow be a blocker here?

So you would change the in-progress other PR then?

[mutantcornholio]

I don't get it. These artifacts will be used in the revive workflow. Revive workflow depends on it, not the other way around. Why would revive workflow be a blocker here?

So you would change the in-progress other PR then?

Not sure if I understand what you mean, but workflows in other PRs would be changed to use artifacts produced by this one. Is that what you're asking?

[athei]

Yes. I am just saying me mindful of that potential interaction with that other open PR.

[mutantcornholio]

That interaction is intended and planned from the start

[athei]

That interaction is intended and planned from the start

Okay that is good to know. Because of right now the other PR just uses caching so there is no indication of that. Can't read minds.

[mutantcornholio]

• macOS needs to be a universal binary: Build twice: Once on a Intel runner and once on a AppleSilicon runner. Then use lipo to merge the two binaries. You can also build both on any macOS runner since cross compiling in macOS is easy.

You mean resolc, not llvm though, right? If cross-compiling is easy, we wouldn't need to create a universal binary of every llvm / clang tool, right?

[xermicus]

LLVM is a dependency (LLD is a runtime dependency). If you cross-compile you need to cross-compile everything.

[athei]

• macOS needs to be a universal binary: Build twice: Once on a Intel runner and once on a AppleSilicon runner. Then use lipo to merge the two binaries. You can also build both on any macOS runner since cross compiling in macOS is easy.

You mean resolc, not llvm though, right? If cross-compiling is easy, we wouldn't need to create a universal binary of every llvm / clang tool, right?

You are right. LLVM is an intermediate artifact statically linked by the resolc binary. So it is not really important for them to be universal as they are not used directly by users. If that is easier (it probably is) you can just can do two separate artifacts for LLVM in this workflow: One for x64 and one for aarch64. But the point is that have to build it twice for macOS in this workflow. Either on different runners or via cross compilation on the same runner.

You could unify those two LLVM artifacts then by making all the binaries and libs universal. But you are right: I wouldn't do that. It's not worth it. We'll only do that for resolc.

[athei]

Let's skip the windows build for now and do that as a follow up. macOS and Linux is enough for now since those seem to be not problematic. I will look into the Windows build when the other things are working.

[mutantcornholio]

I'll take this to draft for the time being, after #201 is merged, I'll alter that release pipeline to use llvm from this one, and hopefully add macos to it as well.