realizing added Rust dependency as pre-build wheel

[cmeesters]

Hi,

As half-way discussed in the slack channel, I would like to bring to attention the following thoughts:

As of 2021b when installing Python a build-dependency for Rust in introduced to satisfy the cryptography module. This is a considerable overhead as building the Rust module (a one-time need for those only installing Python with it) takes about 1.5 h (on our machine). Also, it is only possible on head nodes and not in a job context for those sites where internet access on compute nodes is disabled.

Perhaps this is a non-issue, when we envision more and more software written in Rust to be rolled out in the future. Bio-Rust for instance might gain impact on the field.

IF the build overhead of Rust to support Python is a considerable overhead, one could consider deviating from eb policies and replacing the extension with a pre-build wheel.

Cheers,
Christian

[boegel]

@easybuilders/easybuild-easyconfigs-maintainers Thoughts on this?

I think using a pre-built wheel for cryptography to avoid Rust as a build dependency for Python is worth consdering...

Additional context: pyca/cryptography#5771

[branfosj]

I think using a pre-built wheel for cryptography to avoid Rust as a build dependency for Python is worth consdering...

In out 2021b generation Python we have cryptography 3.4.7. Looking at https://pypi.org/project/cryptography/3.4.7/#files I am only seeing wheels for Python 3.6 (cp36). Are they more generally usable?

Otherwise we hit the lack of POWER support, but there are issues with >= 2021a anyway that may make that an irrelevance.

[Micket]

One possible alternative can also be to break out some extensions from Python, in particular to reduce the number of dependencies from Python. This could also save us from having to make a -bare python variant.
This would probably also mean that building a toolchain wouldn't pull in Rust, but many things would still depend indirectly on Rust since cryptography is such a common dep.

There is probably also some stuff we can do to improve the Rust build, perhaps an external LLVM, or make it build it's LLVM in parallel, or patch it in some way so that it doesn't do downloads during the build (based on comments from other maintainers, I haven't actually looked into this myself).
So, I'm inclined to say; lets keep it? But we could always prepare the python easyconfigs to that it is easy to opt-out of Rust for those who can and want to use the pre built cryptography.

[migueldiascosta]

Regarding the time it takes to build Rust's LLVM, one issue is that we disabled Ninja because it would introduce a circular dependence on Python, but that's not actually the case if we change the Ninja easyconfigs to use Python -bare as a builddep, same as we already do in Rust. Then we can add Ninja as a builddep in Rust and remove

# avoid build dependency on Ninja, which requires Python,
# since Rust is a build dependency for cryptography that is included with Python
configopts += "--set=llvm.ninja=false "

That does speed up the Rust build a bit (and we should also be able to use Ninja when building our own LLVM), but maybe not enough.

I do think that we should at least try to build LLVM only once per toolchain :) In the case of Rust, configopts += "--llvm-root=$EBROOTLLVM --enable-llvm-link-shared" using the latest LLVM as dependency worked for me, but I don't know if this ends up missing some important LLVM patch that Rust is adding. I suppose it will depend on the versions and timings of the releases of Rust and LLVM (?)

(for reference, the build step of Rust just took me ~40 min as it is, ~30 min when using Ninja to build its LLVM and ~20 min when using the external LLVM instead of building its own, so at least in this system it's clearly not only about building LLVM, but of course, ymmv)

[smoors]

alternatively, as discussed during the last conf call, we could create our own pre-built wheels for cryptography for the missing Python versions and architectures we care about, and upload them to https://sources.easybuild.io/

[ocaisa]

It is possible to install Rust without network access, a workflow is in https://github.com/dtolnay/bootstrap

Big note of caution there though:

Be aware that the complete bootstrap uses a lot of disk space. Bootstrapping up through rustc 1.40 requires 120 GB.

[casparvl]

Just as a note: I've described two approaches I tried to (manually) install Rust on an offline system here. While the https://github.com/dtolnay/bootstrap proved difficult, I could in the end use the 'normal' installation procedure when passing the appropriate flags, predownloading some tarballs and putting them in the right cachedir.

The thing left todo is to automate it in an EasyBlock...

[Micket]

cryptography is broken out from the base Python package going forward, though, one will still eventually need the Rust compiler for a bunch of other extensions, and the list of Rust based software and Python packages will only grow, so avoiding it using pre-built wheels seems like an uphill battle. In fact, you probably end up needing to installing multiple Rust compiler versions due to how rapidly it updates)

I made Cargo, CargoPythonPackage, CargoPythonBundle easyblocks which at least allows this to be fully offline.
The issue with the Rust compiler going online is a seperate issue #13548 but as far as Rust and python is concerned, I think we can say we have decided to continue building these.