use rustup llvm/clang + install llvm-tools-preview & std for target

[dvc94ch]

depends on rust-lang/rust#56371

so thinking about this some more, this would be very nice

[MarijnS95]

@dvc94ch Any chance that this is going to land at all? I want to start handing over our Android application through x to my colleagues on Windows, but that (along adb which you seem to prefer choco as installer for) requires llvm executables such as llvm-readelf available in PATH.

Since these are shipped with the ndk, should we bundle them (for at least Android on Windows) for the time being?

EDIT: Alternatively I can see an option to work with an unaltered NDK. It'll be a bigger download and disk-space when unzipped (going to be improved with https://r.android.com/2039604 / https://r.android.com/2043003) and we'll need to change certain paths a bit, but it seems more convenient to me while also calming those who rather download an official distribution.

[dvc94ch]

well, adb and llvm tools are two different issues. I'm currently focused on making xbuild useful and will be a while until I can focus on install issues. the stuff I'm working on is tracked in the milestone good platform suport while this issue is in the category painless setup.

If someone where to land clang builds in rustup, it would certainly be awesome and would help me prioritize this issue (as that's the main blocker and will take a while to make it to stable)

[MarijnS95]

Fair enough, but you're not alone and I'm currently on the virtual make `xbuild` work for our projects on my setup, and subsequently my colleagues' Windows machines milestone. Willing to pick up some of the work, but as written in my comment it doesn't seem like clang/llvm in rustup is going to land soon if at all given how little traction there is in that issue (as such there's nothing to prioritize).

As such I'm looking for alternatives to get the necessary llvm tools going on Windows (same for adb whcih is why I brought that up) while sticking to the otherwise straightforward cargo install xbuild && x run ... approach. Should that perhaps be discussed in a separate issue?

[dvc94ch]

probably yes. although I'm wondering if we can just provide a powershell script that sets this stuff up instead. the problem with including clang/adb in the sysroot is that we need to have a different sysroot for each host platform, which is quite a maintainance burden we currently don't have. Maybe adding something like this to the repo would be sufficient:

windows_setup.ps1:

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))
choco install llvm
choco install adb

[MarijnS95]

In my environment people seem generally opposed to setup scripts and third-party package managers, they much rather prefer a pure cargo install xbuild && x run ... approach even if it means downloading a slightly bigger package up-front.

In that sense I'm not fully sure what to do as I prefer to use system binaries where easily available - on ie. Linux - but feel like we can probably ship the used LLVM tools (and adb) in the Windows repackaged "sdk/ndk" and expand PATH. Thoughts?

[dvc94ch]

in any case it should be a separate thing, like "ndk-tools-for-windows" or something that is downloaded additionally on windows. I'll have a look this week how much work it would be to replace adb.

In my environment people seem generally opposed to setup scripts and third-party package managers

did they give any rational reason? security could be one potentially (increases attack surface). however it's a bit weird to use windows and claim to be security concious

[MarijnS95]

General annoyance of "yet another tool" to use/manage/update. For a more specific example, we've had some trouble with shims being provided for applications that were not installed yet. They would do nothing but supposed to prompt the user to install said application, but the prompt never showed and the program just exited with code 0 breaking our build.rs scripts in very mysterious ways that took quite a while of dissecting to figure out what exactly was going on. We don't want that to repeat.

And again, just a lean setup. rustup and everything "just works™" in our project, and would like to mimic that for Android also.

[dvc94ch]

currently building rust's llvm and trying to figure out how to enable building clang.

[dvc94ch]

ok, I think we can forget clang being shipped by rustup. instead of taking clang from the ndk (since one clang should be enough for all platforms) our only other hope is the llvm project shipping their stuff. can we download clang/lldb releases from the llvm project?

[dvc94ch]

they release some kind of exe installer for windows: https://github.com/llvm/llvm-project/releases

@MarijnS95 can we make that work for us?

[MarijnS95]

@dvc94ch Yes I was eyeing a plain LLVM/clang release since that's not Android-specific (neither ndk-version specific) and it seems to work thus far on Linux with clang from the host (assuming cross-compile support for the desired target triple is included).

Will have to test this out on a Windows machine and/or with a colleague; will get back to you when I've done so.

[MarijnS95]

@dvc94ch It looks like those releases only contain a Windows installer, not an archive we can unpack and use a couple binaries from.

Perhaps we'd need to repack this in the CI as well?

[dvc94ch]

I guess that could work. Still not clear if we just need clang/lldb and can use the rest from rustup. Maybe that would be worth investigating first? But I'd really like to focus on shipping mdb first

[MarijnS95]

What "rest" ships with rustup? We need the llvm-readelf utility and maybe some more?

[dvc94ch]

yes, we need llvm-readobj (llvm-readelf is a symlink), lld (lld-link is a symlink), llvm-ar (llvm-lib is a symlink). In addition to that we use clang/lldb which are not shipped by rustup.

[dvc94ch]

in adition we might need llvm-lipo in the future to create multi architecture macho binaries for macos/ios #56

[MarijnS95]

yes, we need llvm-readobj (llvm-readelf is a symlink), lld (lld-link is a symlink), llvm-ar (llvm-lib is a symlink).

Where are these shipped in rustup? Or is that the upstream tracking issue?

[dvc94ch]

rust-lld I believe should always be installed? for llvm-readobj and llvm-ar you need to rustup component add llvm-tools-preview-x86_64-unknown-linux-gnu. everything should be installed in $HOME/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin.

[dvc94ch]

would still need to figure out how to set arg0. on unix there is this [0], but it has not been implemented for windows yet [1]. since we pass lld/llvm-ar to cargo/cc-rs via env variables we probably wont get around using symlinks anyway. probably the best would be to symlink them into the target dir and pass the symlinks to cargo.

• [0] https://doc.rust-lang.org/std/os/unix/process/trait.CommandExt.html#tymethod.arg0
• [1] Feature request: add method to unix::process::CommandExt to set argv[0] rust-lang/rust#66510

[dvc94ch]

out of scope