Moving to Ubuntu 20.04 for the official Linux binaries

[lnicola]

GitHub is upgrading to Node 20, but we can't use that in our Ubuntu 18.04 container. There is an opt-out, but I think we should put it out of its misery and run the release job on Ubuntu 20.04.

Anyone affected should upgrade to a newer distro (Ubuntu 18.04 LTS support ended in May 2023), use the rustup component, or build from source.

[lnicola]

@Timmmm

RHEL 8 full support ends in May 2024. Using the rustup component seems like a reasonable solution if you're not planning to upgrade.

This article has a simple thing to try that might make it quite easy to do.

The problem is that we used the container for the whole workflow, not just for the build step, but actions now require a newer version of Node. We could probably build on Rocky Linux 8, but that's also EOL in May and at this point it feels like playing whack-a-mole with outdated distros.

[Timmmm]

RHEL 8 has 5 more years of maintenance support unfortunately, so I don't think that's going to convince my company to move. On the other hand I do agree it's a pain to support such ancient distros, especially for an open source project. The blame lies with GNU, but I can't see them ever fixing it because it goes against their philosophy.

Anyway perhaps as a workaround the extension could detect when glibc is too old and tell you how to use rustup instead, because I have no idea how to do that and until you mentioned it I didn't even know it was possible.

Probably just hackily grepping stdout for 'GLIBC_2.27' not found is good enough, and that should be easy enough to implement. What do you think?

[danilaml]

Can't you just switch to RHEL8 based container instead? Should fix nodejs issue and I'd wager it'd be a baseline supported distro for many tools going forward, similarly to how RHEL 7 is/was.

[lnicola]

I don't know which RHEL8 container you mean, but I did look at rockylinux:8. It has GLIBC 2.28, which is good enough, however it's only actively supported until May 31, 2024. Same for CentOS Stream 8.

Affected distributions include Debian 10, Ubuntu 18.04, Enterprise Linux 7 and 8, and Amazon Linux 2.

Debian 10 and Ubuntu 18.04 are already EOL, and Amazon Linux 2 has GLIBC 2.26, so a binary built on Rocky 8 won't work there. Of the newer distros, EL9 and AL2023 are already on GLIBC 2.34. Debian 11 and Ubuntu 20.04 are on GLIBC 2.31.

So the only extra major distro we could support is EL8, at the potential cost of using an EOLed Rocky container.

[danilaml]

@lnicola "active" support is until May 31, but EOL is in 31 May 2029. There is no need for "active" support of the build machine. Node.js (as far as I understand - the main limiting build dependency) Has been building on CentOS7 (rhel 7 based) until NodeJS 18 in 2022, which was on 2+ years of "maintenance updates" by that point. There is a high chance that for the next 2-4 years Rocky Linux 8 or equivalent will be such "lowest common denominator".
Anyway, it depends on the "potential cost" you mention. If for some reason maintaining the build is more than one line in config to switch the container and maybe a few lines in the script to install the dependencies, then it might make sense to bump the support, however, there is a chance that your choice of Ubuntu 20.04 container will become deprecated quicker, than rockylinux8.

[Timmmm]

I totally understand not wanting to have to build on ancient distros. Have you considered linking with Musl? I always distribute my Rust binaries linked with Musl precisely to avoid this Glibc compatibility fiasco. It helps that Rust makes it super easy to do!

You get slightly larger binaries but it's easily worth it IMO.

[lnicola]

@danilaml on paper, the idea is that we don't go through too many hoops to support old distros. Ubuntu 20.04 is already 4 years old at this moment. But on every GLIBC bump we've had to argue for days that no, we don't want to support Debian oldoldstable or whatever people were running. The rustup component is perfectly usable (and runs on EL 7), and so are source builds. Just because you (well, not you in particular) think oldoldstable is the best distro to run because you want a stable system, it doesn't mean you're entitled to request that we accommodate you. If you want that stable, maybe you can run a 4 year old rust-analyzer and Rust toolchain, you don't need the latest and greatest. Still, every time, people show up saying "it's not EOL, it's extended support" or "my outdated distro is still very important, unlike the other outdated distros".

That's actually similar to our policy regarding the supported toolchains and versions of VS Code.

there is a chance that your choice of Ubuntu 20.04 container will become deprecated quicker, than rockylinux8

You're assuming that we find it desirable to use that Rocky 8 container for 13 years, or until its package servers drop off the Internet. I personally think that's the job of the rustup builds and I'd rather use something close to what most of the users will be running.

But I'm personally pretty tired of all these arguments. I switched to the Rocky 8 container in #16928.

@Timmmm

Have you considered linking with Musl?

Yes, and benchmarked it years ago. "Recently" (in 2020) it got an improved memory allocator, but it's still significantly slower (75 to 89 seconds on analysis-stats self). That's not a penalty we're willing to impose on everyone just to make life easier for people running EL8. We can get the performance back by using jemalloc or mimalloc, but they have their own share of issues. And if we were to enable them, I'd rather do that on GLIBC and get even better performance (75 to 71 seconds on the same benchmark) than today. Doing it on MUSL would be a step backwards.

Of course, we can (should?) build an extra binary, but VS Code users would have to update it manually, and that's not easier than using rustup.

[Timmmm]

Ok that all sounds reasonable to me. So perhaps the best solution is to detect the glibc error and tell the user how to fix it then?

I imagine you would like to not have to keep answering these questions every time the glibc developers say "la la la la can't hear you" to backwards compatibility....

I certainly wouldn't have come here if it had just said "Your system is too old for the bundled language server. Please run this command instead: ..."

Incidentally what's the reason for bundling the binary if it could just be installed via rustup without any of these issues?

[lnicola]

@Timmmm next release will be compatible with EL8, should be good for 10 years or so 😩. If anyone runs into this, it's easy enough to search or explain.

Incidentally what's the reason for bundling the binary if it could just be installed via rustup without any of these issues?

The rustup component is generally up to a week old in nightly and 6 (?) weeks old in stable, but sometimes it might lag a lot more. That shouldn't be an issue for Ubuntu 18.04 users, they clearly can't expect the latest and greatest.

[lnicola]

By the way, here's what's triggering the "compatibility fiasco" as you call it:

                 w statx@@GLIBC_2.28
                 w posix_spawn_file_actions_addchdir_np@@GLIBC_2.29
                 U exp2@@GLIBC_2.29
                 U exp@@GLIBC_2.29
                 U log2@@GLIBC_2.29
                 U log@@GLIBC_2.29
                 U pow@@GLIBC_2.29

Regarding the math functions:

These new routines also follow the new C standard regarding error handling, which results in less overhead. The performance improvements range from two to three times on both latency and throughput.

Not sure if that only applies to ARM, but a 2-3x improvement sure sounds good. If you're wondering why we need log2, it's used for const eval.

As for statx and posix_spawn_file_actions_addchdir_np, they're used by the Rust standard library. It used to have a different mechanism that produced backward-compatible binaries, but it changed a couple of years ago.

[danilaml]

@lnicola Thanks for the change. I'm totally in agreement with "not going through many hoops" idea, but your rl8 change seems to be 9 lines, most of which is a simple boilerplate. To me it doesn't look like even a single "hoop". If you had to do some tricky compilation hacks/compiler workarounds or build your own dependencies, then it approaches the category of "not worth it", but as long as it doesn't, I see no cons in doing so.

[Timmmm]

next release will be compatible with EL8, should be good for 10 years or so 😩

Thanks! I feel your pain and really appreciate this! Maybe one day Rust won't depend on libc at all on Linux and we can forget about this whole problem...

[danilaml]

@Timmmm won't help with nodejs, afaiu.

[JoshMcguigan]

I can build from source, but is there a way to tell the standard rust-analyzer VS code plugin where to look for the path to the language server binary?

VS code seems to ignore this configuration option for me.

[lnicola]

rust-analyzer.server.path should work, maybe you don't have the extension enabled?

[JoshMcguigan]

Further testing shows rust-analyzer.serverPath works, even though it also shows greyed out in my configuration.

rust-analyzer/editors/code/src/bootstrap.ts

Line 39 in 1e61ed7

const explicitPath = process.env["__RA_LSP_SERVER_DEBUG"] ?? config.serverPath;

Does this line explain rust-analyzer.serverPath working while rust-analyzer.server.path does not?

[lnicola]

No, it's defined here: https://github.com/rust-lang/rust-analyzer/blob/master/editors/code/package.json#L390-L398.

I'm not sure what's up, are you using a container or SSH?

[JoshMcguigan]

are you using a container or SSH?

I'm using Remote - SSH.

[lnicola]

You might be aware of this, but you need to install the extension on the remote side. Maybe the settings editor doesn't recognize it because it's not installed locally?

That still doesn't explain why the old key name works. Unless you have an ancient RA on the remote.

[JoshMcguigan]

I confirmed the extension is installed on the remote. It is not installed locally, but I'm not sure if that explains why the editor doesn't recognize the settings, because other rust-analyzer settings are recognized in that config file.

As an aside, I just updated to version 0.3.1940 of the extension, and it seems that the default rust-analyzer that it pulls down now only needs GLIBC 2.28 again. Was something changed to support this older GLIBC again? If so, that is great!

[lnicola]

Yeah, it's now compatible with EL8 since #16928.

[JoshMcguigan]

Wow, thats great - thank you!