rust: Initial commit - rust lang #13916

Grommish · 2020-11-14T00:53:21Z

RFC - Rust Language integration into OpenWrt build environment.

This is a first try and will need to be gone over, corrected, fixed, etc.. I'm opening it up to the public for assistance.

Signed-off-by: Donald Hoskins grommish@gmail.com

lang/rust/Makefile

lang/rust/rustc-triple.mk

lang/rust/rust_compile.mk

BKPepe

Just quick a look at this PR. It requires to have commit messages for patches, which you are adding,

lang/rust/Makefile

Grommish · 2021-01-11T19:37:59Z

The artifacts? The dist tarball is 500mb+ per triple target, and is dependant on the host and target combo. There isn't anything to download. You run the package in your source to create and handle the rust tool chain for your specific build. Perhaps I'm not unstanding what you're asking. The toolchain isn't used unless something is looking for it (like suricata), at which point it'll call rust/host and will (create and) install the rust toolchain. If the toolchain isn't in dl/, it creates the dist before installing. If it is there, it just installs it.

…

On Mon, Jan 11, 2021, 2:29 PM Josef Schlehofer ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In lang/rust/Makefile <#13916 (comment)>: > +# +include $(TOPDIR)/rules.mk + +PKG_NAME:=rust +PKG_VERSION:=1.49.0 +PKG_RELEASE:=1 +PKG_LICENSE:=Apache-2.0 MIT +PKG_LICENSE_FILES:=LICENSE-APACHE LICENSE-MIT + +PKG_SOURCE_PROTO:=git +PKG_SOURCE_URL:=https://github.com/rust-lang/rust.git +PKG_SOURCE_DATE:=2020-11-11 +#PKG_SOURCE_VERSION:=b2d115f6db5172c961dfeb50de15f35784dbc7c9 +#PKG_SOURCE_VERSION:=5a6a41e7847ff5f85a31b87431ce2af29c567f1d +PKG_SOURCE_VERSION:=fa55f668e5ea5388ec98b9340969527252239151 +PKG_HASH:=skip <https://github.com/rust-lang/rust.git+PKG_SOURCE_DATE:=2020-11-11+#PKG_SOURCE_VERSION:=b2d115f6db5172c961dfeb50de15f35784dbc7c9+%23PKG_SOURCE_VERSION:=5a6a41e7847ff5f85a31b87431ce2af29c567f1d+PKG_SOURCE_VERSION:=fa55f668e5ea5388ec98b9340969527252239151+PKG_HASH:=skip> Giving me the ability to quickly jump between milestones was/is something I need to hold on to for the moment, but will be removed and SHA256 hashed correctly before committing to the upstream. Why you don't want to have CI passed in the meantime, so anyone can download artifacts from this PR? Right now, nobody knows that it even compiles. See: https://github.com/openwrt/packages/runs/1548152198?check_suite_focus=true#step:4:3730 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#13916 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBY6JRFHJUPYWFDHIAQOUTSZNGSPANCNFSM4TVGXOBQ> .

lang/rust/Config.in

jyn514 · 2021-02-15T14:46:16Z

lang/rust/Config.in

+	config RUST_OPTIMIZE
+		bool "Build optimized rust code (--enable-optimize)"
+		default n


Does this correspond to optimize = false in config.toml? Setting that to false isn't supported, turning it off by default seems like a bad idea.

https://github.com/rust-lang/rust/blob/9503ea19edbf01b9435e80e17d60ce1b88390116/config.toml.example#L332-L336

There are only a few things I set in the TOML via --set having to do with the build system. (https://github.com/openwrt/packages/pull/13916/files#diff-000828b821201f172a6a3c1b97c5a89fd38753d46e7b0c11e475b7e61a99a24cR49)

These simply add (or not) the -- flag indicated (--enable-optimize in this case) to the arguments passed to ./configure

Well right, but it seems strange to me to expose this flag at all when there are a bunch of others you've left out. A compiler with optimize=false is basically useless, it will take a half hour just to compile the standard library.

I copied the output from ./configure --help to generate the list. Which did I miss? As we've discussed, I'm totally open to suggestions. :)

Oh, I don't think you need to expose every option by hand, there's a whole ton of them:

> grep -E '^(#[^ ]|\[)' config.toml.example #profile = <none> [llvm] #download-ci-llvm = false #skip-rebuild = false #optimize = true #thin-lto = false #release-debuginfo = false #assertions = false #ccache = false #ccache = "/path/to/ccache" #version-check = true #static-libstdcpp = false #ninja = true #targets = "AArch64;ARM;Hexagon;MSP430;Mips;NVPTX;PowerPC;RISCV;Sparc;SystemZ;WebAssembly;X86" #experimental-targets = "AVR" #link-jobs = 0 #link-shared = false #version-suffix = "-rust-dev" #clang-cl = '/path/to/clang-cl.exe' #cflags = "-fextra-flag" #cxxflags = "-fextra-flag" #ldflags = "-Wl,extra-flag" #use-libcxx = true #use-linker = "lld" #allow-old-toolchain = false #polly = false [build] #doc-stage = 0 #build-stage = 1 #test-stage = 1 #dist-stage = 2 #install-stage = 2 #bench-stage = 2 #build = "x86_64-unknown-linux-gnu" #host = ["x86_64-unknown-linux-gnu"] #target = ["x86_64-unknown-linux-gnu"] #build-dir = "build" #cargo = "/path/to/bin/cargo" #rustc = "/path/to/bin/rustc" #rustfmt = "/path/to/bin/rustfmt" #docs = true #compiler-docs = false #submodules = true #fast-submodules = true #gdb = "gdb" #nodejs = "node" #python = "python" #locked-deps = false #vendor = false #full-bootstrap = false #extended = false #tools = ["cargo", "rls", "clippy", "rustfmt", "analysis", "src"] #verbose = 0 #sanitizers = false #profiler = false #cargo-native-static = false #low-priority = false #configure-args = [] #local-rebuild = false #print-step-timings = false [install] #prefix = "/usr/local" #sysconfdir = "/etc" #docdir = "share/doc/rust" #bindir = "bin" #libdir = "lib" #mandir = "share/man" #datadir = "share" #infodir = "share/info" #localstatedir = "/var/lib" [rust] #optimize = true #debug = false #download-rustc = false #codegen-units = if incremental { 256 } else { 16 } #codegen-units-std = 1 #debug-assertions = rust.debug (boolean) #debug-assertions-std = rust.debug-assertions (boolean) #debug-logging = rust.debug-assertions (boolean) #debuginfo-level = 0 #debuginfo-level-rustc = 0 #debuginfo-level-std = 0 #debuginfo-level-tools = 0 #debuginfo-level-tests = 0 #run-dsymutil = false #backtrace = true #incremental = false #parallel-compiler = false #default-linker = "cc" #channel = "dev" #description = "" #musl-root = "..." #rpath = true #verbose-tests = false #optimize-tests = true #codegen-tests = true #ignore-git = true #dist-src = false #save-toolstates = "/path/to/toolstates.json" #codegen-backends = ["llvm"] #lld = false #use-lld = false #llvm-tools = false #deny-warnings = true #backtrace-on-ice = false #verify-llvm-ir = false #thin-lto-import-instr-limit = 100 #remap-debuginfo = false #jemalloc = false #test-compare-mode = false #llvm-libunwind = 'no' #control-flow-guard = false #new-symbol-mangling = false [target.x86_64-unknown-linux-gnu] #cc = "cc" #cxx = "c++" #ar = "ar" #ranlib = "ranlib" #linker = "cc" #llvm-config = "../path/to/llvm/root/bin/llvm-config" #llvm-filecheck = "/path/to/FileCheck" #android-ndk = "/path/to/ndk" #sanitizers = false #profiler = false #crt-static = false #musl-root = "..." #musl-libdir = musl-root/lib #wasi-root = "..." #qemu-rootfs = "..." [dist] #sign-folder = "path/to/folder/to/sign" #gpg-password-file = "path/to/gpg/password" #upload-addr = "https://example.com/folder" #src-tarball = true #missing-tools = false #compression-formats = ["gz", "xz"]

My point was that this option doesn't seem particularly useful to expose, and especially shouldn't be on by default.

jyn514 · 2021-02-15T14:47:23Z

lang/rust/Config.in

+	config RUST_OPTIMIZE_LLVM
+		bool "Build optimized LLVM (--enable-optimize-llvm)"
+		default n


I don't know what OpenWRT policies are for release vs debug builds, but the compiler is significantly slower without ThinLTO, you may want to consider turning it on by default.

Then let's turn that on :D I was not going to set defaults for things I didn't know about unless I needed something to work and it didn't. This worked fine off by default, and I was waiting for those with more knowledge (ahem.. poke) to step in and tell me different :) I'll try it

jyn514 · 2021-02-15T14:48:21Z

lang/rust/Config.in

+	config RUST_FULL_BOOTSTRAP
+		bool "Full Bootstrap - Build three compilers instead of two (--enable-full-bootstrap)"
+		default y


Why enable this by default? It shouldn't be needed and it will make the build take half again as long. https://github.com/rust-lang/rust/blob/9503ea19edbf01b9435e80e17d60ce1b88390116/config.toml.example#L243-L248

It was my understanding that in order to get stage2 artifacts, the full bootstrap needed to happen (so it didn't stop at just stage0/1)? Remember, almost everyone using Rust-lang in an OpenWrt context will be cross-compiling and will need the stage2 artifacts that would be skipped without the full bootstrap. Did I misunderstand?

I think x.py should automatically rebuild stage2 if it needs to for cross compiling. I would try that first to see if it works.

Also, I'm slightly confused why you would need 3 compilers in the first place, I would expect a stage 2 compiler to work fine. That's what's distributed by rustup.

x.py only gets call ONCE, during the initial compile. After that, the installation binaries I create are stored in an archive that is used instead of touching the rust source at all (unless the rust package version changes in the OpenWrt upstream). This means I only run x.py a single time for an install..

-rw-r--r-- 1 grommish grommish 520960752 Feb 11 00:57 rust-1.49.0-x86_64-unknown-linux-gnu_mips64-openwrt-linux-musl-install.tar.xz

That is an example of my toolchain installation files (basically, a one-off version of rustup I guess?) that are built during the first compile. Subsequent runs, even on a make clean, sees that file, extracts it, runs the installs, and calls it a day.

Seems reasonable, but I'm not sure how it's relevant. This is specifically about the rustc-dev component, which gets compiled with x.py at the same time as everything else. If you remove rustc-dev I think this option won't even be used either way.

I'm testing removing the -dev options as we speak to see if it breaks anything.. then, if not, I'll test without the full bootstrap

lang/rust/patches/00-default_dynamic_linking.patch

jyn514 · 2021-02-15T14:51:42Z

lang/rust/patches/01-add_mips64_musl_support.patch

+pub fn target() -> Target {
+    let mut base = super::linux_musl_base::opts();
+    base.cpu = "mips64r2".to_string();
+    base.features = "+mips64r2,+soft-float".to_string();
+    base.max_atomic_width = Some(64);
+    Target {
+        // LLVM doesn't recognize "muslabi64" yet.
+        llvm_target: "mips64-unknown-linux-musl".to_string(),
+        pointer_width: 64,
+        data_layout: "E-m:e-i8:8:32-i16:16:32-i64:64-n32:64-S128".to_string(),
+        arch: "mips64".to_string(),
+        options: TargetOptions { endian: "big".to_string(), mcount: "_mcount".to_string(), ..base },
+    }


This looks useful to other distros too :) have you considered upstreaming it to rust-lang/rust?

I don't see why not, although it's just the built-in triple renamed and defined with a soft-float since the dynlinking is handles in the MUSL define.

Is there any progress with upstreaming patches which you have for mips64, aarch64, etc?

mips64-openwrt-linux-musl was upstream'ed and merged into rust-lang master back in February, 2022. It's the only actual hardware I have to test on, architect-wise.

As I continue with testing, or getting confirmation that the various tuples don't cause issues (armv5 still doesn't work, for example) and figuring out whether errors I see are because of OpenWrt build system, the tuple, the rust install, or the package itself (or some other interaction and/or combined situations). For example, I've had a request for the vaultwarden package. I was seeing errors, I reported errors, and in the end, it was because the ring crate (which is a common crypto crate from what I can tell) uses BoringSSL, and BoringSSL doesn't work for BigEndian archs (like mips64) sigh. So chasing errors to ensure stability is a long, drawn out process.

And, even when everything seems to work, the end product may or may not work as intended. I had issues with suricata6 in which it would build with soft-float flagged, enabled in the mips64-openwrt-linux-musl tuple, but was SIGSEV erroring when I attempted to run it. Turns out, OpenWrt for mips64 didn't enable MIPS soft-float emulation in the kernel.

So, if anyone wants to help me test an arch, verification can happen and I can upstream them. They are responsive enough.

jyn514 · 2021-02-15T14:52:53Z

lang/rust/rust_compile.mk

+define Host/Compile
+	cd $(HOST_BUILD_DIR) && \
+	RUST_BACKTRACE=full $(PYTHON) x.py --config ./config.toml dist cargo extended \
+	   library/std llvm-tools miri rust-dev rustc-dev


Why rust-dev and rustc-dev? That's the LLVM dynamic library and libraries for using rustc_private; they shouldn't be needed unless you're using a custom rustc driver.

Grommish · 2021-02-15T15:09:37Z

Hi @jyn514 ;)

Honestly, I did the rust thing because I needed it, not because I know how to use it. Most of the comments I received while doing this went from "why?" to "can't be done". The suggestions/comments you've made I am certainly willing to entertain, as I KNOW there are better ways to optimize this. I do know that it WORKS, as is, so further testing would be needed..

I can't even beg people to identify their triples so I can patch them in :D and I've been too lazy to build out targets I don't have just to build a rust toolchain. As you mentioned, the first time takes a while..

I'll work through these suggestions and see what/if anything breaks. Thanks for the nudge!

Grommish · 2021-02-15T18:07:34Z

Would there be best practices for giving those options? Or any other you can suggest to remove (or add?)

…

On Mon, Feb 15, 2021 at 12:30 PM Joshua Nelson ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In lang/rust/Config.in <#13916 (comment)>: > + config RUST_OPTIMIZE + bool "Build optimized rust code (--enable-optimize)" + default n Oh, I don't think you need to expose every option by hand, there's a whole ton of them: > grep -E '^(#[^ ]|\[)' config.toml.example #profile = <none> [llvm] #download-ci-llvm = false #skip-rebuild = false #optimize = true #thin-lto = false #release-debuginfo = false #assertions = false #ccache = false #ccache = "/path/to/ccache" #version-check = true #static-libstdcpp = false #ninja = true #targets = "AArch64;ARM;Hexagon;MSP430;Mips;NVPTX;PowerPC;RISCV;Sparc;SystemZ;WebAssembly;X86" #experimental-targets = "AVR" #link-jobs = 0 #link-shared = false #version-suffix = "-rust-dev" #clang-cl = '/path/to/clang-cl.exe' #cflags = "-fextra-flag" #cxxflags = "-fextra-flag" #ldflags = "-Wl,extra-flag" #use-libcxx = true #use-linker = "lld" #allow-old-toolchain = false #polly = false [build] #doc-stage = 0 #build-stage = 1 #test-stage = 1 #dist-stage = 2 #install-stage = 2 #bench-stage = 2 #build = "x86_64-unknown-linux-gnu" #host = ["x86_64-unknown-linux-gnu"] #target = ["x86_64-unknown-linux-gnu"] #build-dir = "build" #cargo = "/path/to/bin/cargo" #rustc = "/path/to/bin/rustc" #rustfmt = "/path/to/bin/rustfmt" #docs = true #compiler-docs = false #submodules = true #fast-submodules = true #gdb = "gdb" #nodejs = "node" #python = "python" #locked-deps = false #vendor = false #full-bootstrap = false #extended = false #tools = ["cargo", "rls", "clippy", "rustfmt", "analysis", "src"] #verbose = 0 #sanitizers = false #profiler = false #cargo-native-static = false #low-priority = false #configure-args = [] #local-rebuild = false #print-step-timings = false [install] #prefix = "/usr/local" #sysconfdir = "/etc" #docdir = "share/doc/rust" #bindir = "bin" #libdir = "lib" #mandir = "share/man" #datadir = "share" #infodir = "share/info" #localstatedir = "/var/lib" [rust] #optimize = true #debug = false #download-rustc = false #codegen-units = if incremental { 256 } else { 16 } #codegen-units-std = 1 #debug-assertions = rust.debug (boolean) #debug-assertions-std = rust.debug-assertions (boolean) #debug-logging = rust.debug-assertions (boolean) #debuginfo-level = 0 #debuginfo-level-rustc = 0 #debuginfo-level-std = 0 #debuginfo-level-tools = 0 #debuginfo-level-tests = 0 #run-dsymutil = false #backtrace = true #incremental = false #parallel-compiler = false #default-linker = "cc" #channel = "dev" #description = "" #musl-root = "..." #rpath = true #verbose-tests = false #optimize-tests = true #codegen-tests = true #ignore-git = true #dist-src = false #save-toolstates = "/path/to/toolstates.json" #codegen-backends = ["llvm"] #lld = false #use-lld = false #llvm-tools = false #deny-warnings = true #backtrace-on-ice = false #verify-llvm-ir = false #thin-lto-import-instr-limit = 100 #remap-debuginfo = false #jemalloc = false #test-compare-mode = false #llvm-libunwind = 'no' #control-flow-guard = false #new-symbol-mangling = false [target.x86_64-unknown-linux-gnu] #cc = "cc" #cxx = "c++" #ar = "ar" #ranlib = "ranlib" #linker = "cc" #llvm-config = "../path/to/llvm/root/bin/llvm-config" #llvm-filecheck = "/path/to/FileCheck" #android-ndk = "/path/to/ndk" #sanitizers = false #profiler = false #crt-static = false #musl-root = "..." #musl-libdir = musl-root/lib #wasi-root = "..." #qemu-rootfs = "..." [dist] #sign-folder = "path/to/folder/to/sign" #gpg-password-file = "path/to/gpg/password" #upload-addr = "https://example.com/folder" #src-tarball = true #missing-tools = false #compression-formats = ["gz", "xz"] My point was that this option doesn't seem particularly useful to expose, and especially shouldn't be on by default. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#13916 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBY6JRBRCWFXXBO7JXSPYLS7FK3NANCNFSM4TVGXOBQ> .

jyn514 · 2021-02-15T18:43:09Z

I don't know what the purpose of the options are, I guess. What are people looking for when they customize builds? If it were me I would probably pick one set of options and use them for everyone, but I also have never maintained packages before 😆

Grommish · 2021-02-15T19:06:17Z

Well, the biggest issues I have with this is that no one touches configure, config.toml, or x.py directly. So options that might needed I want to have available should the need arise for an individual. I can put common ones, less common ones, and default ones. I just need to figure out which would go where

…

On Mon, Feb 15, 2021, 1:43 PM Joshua Nelson ***@***.***> wrote: I don't know what the purpose of the options are, I guess. What are people looking for when they customize builds? If it were me I would probably pick one set of options and use them for everyone, but I also have never maintained packages before 😆 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#13916 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBY6JR3NMNAG7FC3J66AJLS7FTM5ANCNFSM4TVGXOBQ> .

Grommish · 2021-03-21T08:43:02Z

Updated PR to include mipsel-24kc

lang/rust/rustc-config.mk

wulfy23 · 2021-03-23T16:28:20Z

compile tested for aarch64/rpi4 @ 3:23mins ( 2 core pentium )

-needed ccache ( host/apt )
-would be good to see some forking or similar optimisations
-runs un-needed download.pl call but not entirely a bad move in the longer term scheme of things ( potential precompiled bins down the track? ;) )

other than that pretty smooth... thanks for the hard work

Grommish · 2021-03-23T21:06:48Z

compile tested for aarch64/rpi4 @ 3:23mins ( 2 core pentium )

-needed ccache ( host/apt )
-would be good to see some forking or similar optimisations
-runs un-needed download.pl call but not entirely a bad move in the longer term scheme of things ( potential precompiled bins down the track? ;) )

other than that pretty smooth... thanks for the hard work

I actually wanted to use the download.mk and it's hash functions to create/verify the binaries hashes prior to using them. I have, however, redone how the binaries are stored, so it means MUCH shorter compile times in the future, and a smaller footprint in dl/. In theory, the buildbot could generate the toolchains and dole them out, but I don't know if that should/would ever happen.

cccache is selectable in the settings, although I believe I also just set it for the builder if the build env ccache is set (https://github.com/openwrt/packages/pull/13916/files#diff-f2f5c684ef57dc9ef001c539fb9a97c8a66e2efe84b68985fa23b45f8f4a1dc4R30).

Grommish · 2021-03-26T00:54:19Z

Update to the build logic. The package now breaks the install binaries into the Host and Target components. This reduces on-disk footprint by roughly 360Mb for every extra target toolchain. It also allows for faster toolchain building when the Host toolchain is already in place.

predators46 · 2022-09-22T00:23:06Z

finally llvm-rust rust rust_host can be created in openwrt 22.03.0.

robotronnn · 2022-09-22T23:24:27Z

Will there be a tutorial how to put this on my device?

predators46 · 2022-09-27T21:20:08Z

Will there be a tutorial how to put this on my device?

you must add the packages llvm-rustc rust and rustc_host when you build openwrt. and keep in mind that some packages are made for the host, not packages that can be installed on the device

lucize · 2022-09-29T08:15:54Z

don't understand how you guys can compile this, despite that there are errors in Makefile (fixable) like

Makefile:120: *** Package/rust_host is missing the TITLE field.  Stop.
Makefile:96: *** Package/llvm-project-rustc is missing the TITLE field.  Stop.

I also get this

configure: processing command line
configure:
configure: error: Option '--program-prefix=' is not recognized

can't seems to understand from where is this coming
--build=x86_64-pc-linux-gnu --program-prefix="" --program-suffix="" --prefix=/usr

predators46 · 2022-09-29T08:23:28Z

don't understand how you guys can compile this, despite that there are errors in Makefile (fixable) like
Makefile:120: *** Package/rust_host is missing the TITLE field.  Stop.
Makefile:96: *** Package/llvm-project-rustc is missing the TITLE field.  Stop.
I also get this
configure: processing command line
configure:
configure: error: Option '--program-prefix=' is not recognized
can't seems to understand from where is this coming --build=x86_64-pc-linux-gnu --program-prefix="" --program-suffix="" --prefix=/usr

I hope you use the makefile here
https://github.com/Itus-Shield/packages/tree/rust_host

I didn't get the error you found

lucize · 2022-09-29T19:40:16Z

I'm on Ubuntu 22.04.1
downloaded your files


ERROR: please fix package/feeds/packages/llvm-project-rustc/Makefile - see logs/package/feeds/packages/llvm-project-rustc/dump.txt for details
ERROR: please fix package/feeds/packages/rust_host/Makefile - see logs/package/feeds/packages/rust_host/dump.txt for details


[...]

touch /home/build/staging/build_dir/target-x86_64_musl/rust-1.64.0/.prepared_7f4a872e4632bc670b50d7f04d738180_6664517399ebbbc92a37c5bb081b5c53
rm -f /home/build/staging/build_dir/target-x86_64_musl/rust-1.64.0/.configured_*
rm -f /home/build/staging/staging_dir/target-x86_64_musl/stamp/.rust_installed
(cd /home/build/staging/build_dir/target-x86_64_musl/rust-1.64.0/./; if [ -x ./configure ]; then find /home/build/staging/build_dir/target-x86_64_musl/rust-1.64.0/ -name config.guess | xargs -r chmod u+w; find /home/build/staging/build_dir/target-x86_64_musl/rust-1.64.0/ -name config.guess | xargs -r -n1 cp --remove-destination /home/build/staging/scripts/config.guess; find /home/build/staging/build_dir/target-x86_64_musl/rust-1.64.0/ -name config.sub | xargs -r chmod u+w; find /home/build/staging/build_dir/target-x86_64_musl/rust-1.64.0/ -name config.sub | xargs -r -n1 cp --remove-destination /home/build/staging/scripts/config.sub; AR="x86_64-openwrt-linux-musl-gcc-ar" AS="x86_64-openwrt-linux-musl-gcc -c -Os -pipe -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -fmacro-prefix-map=/home/build/staging/build_dir/target-x86_64_musl/rust-1.64.0=rust-1.64.0 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro" LD=x86_64-openwrt-linux-musl-ld NM="x86_64-openwrt-linux-musl-gcc-nm" CC="x86_64-openwrt-linux-musl-gcc" GCC="x86_64-openwrt-linux-musl-gcc" CXX="x86_64-openwrt-linux-musl-g++" RANLIB="x86_64-openwrt-linux-musl-gcc-ranlib" STRIP=x86_64-openwrt-linux-musl-strip OBJCOPY=x86_64-openwrt-linux-musl-objcopy OBJDUMP=x86_64-openwrt-linux-musl-objdump SIZE=x86_64-openwrt-linux-musl-size CFLAGS="-Os -pipe -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -fmacro-prefix-map=/home/build/staging/build_dir/target-x86_64_musl/rust-1.64.0=rust-1.64.0 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro " CXXFLAGS="-Os -pipe -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -fmacro-prefix-map=/home/build/staging/build_dir/target-x86_64_musl/rust-1.64.0=rust-1.64.0 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro " CPPFLAGS="-I/home/build/staging/staging_dir/toolchain-x86_64_gcc-11.3.0_musl/usr/include -I/home/build/staging/staging_dir/toolchain-x86_64_gcc-11.3.0_musl/include/fortify -I/home/build/staging/staging_dir/toolchain-x86_64_gcc-11.3.0_musl/include " LDFLAGS="-L/home/build/staging/staging_dir/toolchain-x86_64_gcc-11.3.0_musl/usr/lib -L/home/build/staging/staging_dir/toolchain-x86_64_gcc-11.3.0_musl/lib -znow -zrelro "   ./configure --target=x86_64-openwrt-linux --host=x86_64-openwrt-linux --build=x86_64-pc-linux-gnu --program-prefix="" --program-suffix="" --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/lib --sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --mandir=/usr/man --infodir=/usr/info --disable-nls  ; fi; )
configure: processing command line
configure:
configure: error: Option '--program-prefix=' is not recognized
make[2]: *** [Makefile:128: /home/build/staging/build_dir/target-x86_64_musl/rust-1.64.0/.configured_68b329da9893e34099c7d8ad5cb9c940] Error 1
make[2]: Leaving directory '/home/build/staging/feeds/packages/lang/rust'
time: package/feeds/packages/rust/compile#1.31#1.49#2.84
    ERROR: package/feeds/packages/rust failed to build.
make[1]: *** [package/Makefile:116: package/feeds/packages/rust/compile] Error 1
make[1]: Leaving directory '/home/build/staging'
make: *** [/home/build/staging/include/toplevel.mk:231: package/rust/compile] Error 2

sha256sum

91b33f45ac46790cae91e350e21881020915766d78675fe8d83c8a9d01d0fde6 devel/llvm-project-rustc/Makefile
4237dd900a40d8f7e99f48656b1ad69816f4925cf1f0e4cfa3104f074c44da07 lang/rust/Makefile
9a6682b40582a547db844fee03dad1d11feb31ad2e4c7f420e988ee3c012e8d6 lang/rust_host/Makefile

predators46 · 2022-09-29T21:52:25Z

I'm on Ubuntu 22.04.1 downloaded your files


ERROR: please fix package/feeds/packages/llvm-project-rustc/Makefile - see logs/package/feeds/packages/llvm-project-rustc/dump.txt for details
ERROR: please fix package/feeds/packages/rust_host/Makefile - see logs/package/feeds/packages/rust_host/dump.txt for details


[...]

touch /home/build/staging/build_dir/target-x86_64_musl/rust-1.64.0/.prepared_7f4a872e4632bc670b50d7f04d738180_6664517399ebbbc92a37c5bb081b5c53
rm -f /home/build/staging/build_dir/target-x86_64_musl/rust-1.64.0/.configured_*
rm -f /home/build/staging/staging_dir/target-x86_64_musl/stamp/.rust_installed
(cd /home/build/staging/build_dir/target-x86_64_musl/rust-1.64.0/./; if [ -x ./configure ]; then find /home/build/staging/build_dir/target-x86_64_musl/rust-1.64.0/ -name config.guess | xargs -r chmod u+w; find /home/build/staging/build_dir/target-x86_64_musl/rust-1.64.0/ -name config.guess | xargs -r -n1 cp --remove-destination /home/build/staging/scripts/config.guess; find /home/build/staging/build_dir/target-x86_64_musl/rust-1.64.0/ -name config.sub | xargs -r chmod u+w; find /home/build/staging/build_dir/target-x86_64_musl/rust-1.64.0/ -name config.sub | xargs -r -n1 cp --remove-destination /home/build/staging/scripts/config.sub; AR="x86_64-openwrt-linux-musl-gcc-ar" AS="x86_64-openwrt-linux-musl-gcc -c -Os -pipe -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -fmacro-prefix-map=/home/build/staging/build_dir/target-x86_64_musl/rust-1.64.0=rust-1.64.0 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro" LD=x86_64-openwrt-linux-musl-ld NM="x86_64-openwrt-linux-musl-gcc-nm" CC="x86_64-openwrt-linux-musl-gcc" GCC="x86_64-openwrt-linux-musl-gcc" CXX="x86_64-openwrt-linux-musl-g++" RANLIB="x86_64-openwrt-linux-musl-gcc-ranlib" STRIP=x86_64-openwrt-linux-musl-strip OBJCOPY=x86_64-openwrt-linux-musl-objcopy OBJDUMP=x86_64-openwrt-linux-musl-objdump SIZE=x86_64-openwrt-linux-musl-size CFLAGS="-Os -pipe -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -fmacro-prefix-map=/home/build/staging/build_dir/target-x86_64_musl/rust-1.64.0=rust-1.64.0 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro " CXXFLAGS="-Os -pipe -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -fmacro-prefix-map=/home/build/staging/build_dir/target-x86_64_musl/rust-1.64.0=rust-1.64.0 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro " CPPFLAGS="-I/home/build/staging/staging_dir/toolchain-x86_64_gcc-11.3.0_musl/usr/include -I/home/build/staging/staging_dir/toolchain-x86_64_gcc-11.3.0_musl/include/fortify -I/home/build/staging/staging_dir/toolchain-x86_64_gcc-11.3.0_musl/include " LDFLAGS="-L/home/build/staging/staging_dir/toolchain-x86_64_gcc-11.3.0_musl/usr/lib -L/home/build/staging/staging_dir/toolchain-x86_64_gcc-11.3.0_musl/lib -znow -zrelro "   ./configure --target=x86_64-openwrt-linux --host=x86_64-openwrt-linux --build=x86_64-pc-linux-gnu --program-prefix="" --program-suffix="" --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/lib --sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --mandir=/usr/man --infodir=/usr/info --disable-nls  ; fi; )
configure: processing command line
configure:
configure: error: Option '--program-prefix=' is not recognized
make[2]: *** [Makefile:128: /home/build/staging/build_dir/target-x86_64_musl/rust-1.64.0/.configured_68b329da9893e34099c7d8ad5cb9c940] Error 1
make[2]: Leaving directory '/home/build/staging/feeds/packages/lang/rust'
time: package/feeds/packages/rust/compile#1.31#1.49#2.84
    ERROR: package/feeds/packages/rust failed to build.
make[1]: *** [package/Makefile:116: package/feeds/packages/rust/compile] Error 1
make[1]: Leaving directory '/home/build/staging'
make: *** [/home/build/staging/include/toplevel.mk:231: package/rust/compile] Error 2

sha256sum

91b33f45ac46790cae91e350e21881020915766d78675fe8d83c8a9d01d0fde6 devel/llvm-project-rustc/Makefile
4237dd900a40d8f7e99f48656b1ad69816f4925cf1f0e4cfa3104f074c44da07 lang/rust/Makefile
9a6682b40582a547db844fee03dad1d11feb31ad2e4c7f420e988ee3c012e8d6 lang/rust_host/Makefile

don't use my changes. why don't you use here

https://github.com/Itus-Shield/packages/

lucize · 2022-09-30T07:17:16Z

tried the rust-1.59 build, that is only host build (what I need) and failed after a long period of time with

  = note: /usr/bin/ld: cannot find -lz: No such file or directory
          collect2: error: ld returned 1 exit status


error: could not compile `cargo` due to previous error

I doubt that llvm makefile is failing for me, on the new makefile aproach

edit: fixed -lz with apt install zlib1g-dev

Akira25 · 2022-09-30T09:33:52Z

@Grommish This PR is almost two years around now. I'd really love to get this into main-branch soon, like most people here. Is there any chance in supporting you in the final steps to take?

lucize · 2022-10-01T09:05:20Z

for me, https://github.com/Itus-Shield/packages/commits/working is what works and I need, even if it's building everything and takes forever, guess that just updating to latest version will also work
now I only have to see how tu use the actual binaries, I guess some modifications to python is also needed

Ripgrep recursively searches directories for a regex pattern while respecting your gitignore Requires rust-lang (openwrt#13916) Signed-off-by: Donald Hoskins <grommish@gmail.com>

lu-zero · 2022-10-02T10:59:08Z

I updated to rust 1.64 here https://github.com/domo-iot/packages/tree/rust-update

After a couple of fixes everything seems working as intended.

Ripgrep recursively searches directories for a regex pattern while respecting your gitignore Requires rust-lang (openwrt#13916) Signed-off-by: Donald Hoskins <grommish@gmail.com>

lu-zero · 2022-10-20T13:37:10Z

There is still a bug in the rustc build since it seems to use the wrong dynamic linker when you link external libraries.

neggles · 2022-10-22T10:34:16Z

Hi all,

I don't know that we can expect Grommish to come back to this one anytime soon. He was using an Azure VM I loaned him for building and testing, and I've just had a look and the last time it was powered on was back in July... I've not seen or heard from him on IRC, either. Hope he's okay.

Regardless, IMO, this PR is not a great way to get Rust in OpenWrt. The overall implementation here is... messy. That's not a reflection on Grommish - debugging toolchain build issues on Rust is not a lot of fun and the documentation is not great, it's impressive that he got this far - but long-term I don't think this is maintainable, and it's horribly inefficient at build-time.

One major thing is, at last check, this PR builds an entire copy of LLVM and two copies of the rust toolchain, rather than starting with a prebuilt binary rustup toolchain and using that to bootstrap the target one; this seemed like it might be necessary at one point, but after digging into how this all works for quite some time, it's not necessary at all. This implementation does some unpleasant things with shared library paths and the like which shouldn't be required either.

I did get partway through building out Rust support in a similar fashion to how golang support is laid out and functions - the rust and golang toolchain build systems follow fairly similar approaches, logically speaking, and the steps are broadly similar - but I haven't had the time or motivation to come back and finish it, not that I got super far.

If you'd like to push this forward, I'd suggest looking at how the golang support has been implemented, and working to build something similar for Rust that involves slightly fewer hackarounds, is a bit more resilient to Rust version changes, and works with a rustup toolchain at the first stage rather than having to compile everything from stage-zero LLVM on down.

That said I certainly don't speak for everyone and I can't speak for Grommish, whether he's here or not, so don't take my word as gospel 😅

lu-zero · 2022-10-22T16:13:27Z

We can try to upstream the other arch targets and have them in the tier-3 as the mips target is already.

neggles · 2022-10-25T06:41:36Z

Heh... those targets aren't actually necessary either, FWIW. OpenWrt's various arch targets are already covered by existing upstream tuples, and we're building the whole toolchain ourselves anyway so we can use whatever config we like...

lu-zero · 2022-10-25T07:38:03Z

Heh... those targets aren't actually necessary either, FWIW. OpenWrt's various arch targets are already covered by existing upstream tuples, and we're building the whole toolchain ourselves anyway so we can use whatever config we like...

Having them makes our lives a little simpler. once the json targets enter stable we could leverage upstream to install the host compiler and just compile std.

The wrong dynamic linker for the musl targets is yet to be investigated though. Anybody using alpine can check what they are doing?

lu-zero · 2022-11-07T13:31:03Z

I fixed the problem with the dynamic linker, @Grommish do you still have interest in completing it or may I take over?

Ripgrep recursively searches directories for a regex pattern while respecting your gitignore Requires rust-lang (openwrt#13916) Signed-off-by: Donald Hoskins <grommish@gmail.com>

Grommish commented Nov 14, 2020

View reviewed changes

lang/rust/Makefile Outdated Show resolved Hide resolved

Grommish commented Nov 14, 2020

View reviewed changes

lang/rust/Makefile Outdated Show resolved Hide resolved

Grommish mentioned this pull request Nov 14, 2020

rustup: Bring up rustup rustc/cargo language installer #13040

Closed

Grommish commented Nov 14, 2020

View reviewed changes

lang/rust/rustc-triple.mk Outdated Show resolved Hide resolved

Grommish changed the title ~~[rust] Initial commit - rust lang~~ rust: Initial commit - rust lang Nov 14, 2020

Grommish mentioned this pull request Nov 16, 2020

Suricata6: initial commit #13924

Closed

Grommish force-pushed the rust branch from 87b3b6c to 868705e Compare December 14, 2020 05:29

Grommish commented Dec 22, 2020

View reviewed changes

lang/rust/rust_compile.mk Outdated Show resolved Hide resolved

BKPepe requested changes Jan 11, 2021

View reviewed changes

lang/rust/Makefile Outdated Show resolved Hide resolved

lang/rust/Makefile Outdated Show resolved Hide resolved

lang/rust/Makefile Show resolved Hide resolved

Grommish force-pushed the rust branch 2 times, most recently from 06518c0 to 079d58c Compare January 25, 2021 05:18

This was referenced Feb 14, 2021

python-cryptography: Update to 3.3.2 #14744

Merged

Cryptography 3.3 LTS release without Rust bindings pyca/cryptography#5799

Closed

jyn514 reviewed Feb 15, 2021

View reviewed changes

Grommish force-pushed the rust branch from 2b30f6b to f079d93 Compare February 15, 2021 19:39

Grommish force-pushed the rust branch 2 times, most recently from 8d4b80f to 2b67d36 Compare March 21, 2021 08:42

Grommish force-pushed the rust branch from 2b67d36 to 0924afc Compare March 22, 2021 00:53

jyn514 reviewed Mar 22, 2021

View reviewed changes

lang/rust/rustc-config.mk Outdated Show resolved Hide resolved

Grommish force-pushed the rust branch from 0924afc to 57d7c9b Compare March 22, 2021 23:30

Grommish force-pushed the rust branch from 57d7c9b to cee25cd Compare March 26, 2021 00:50

lu-zero mentioned this pull request Nov 8, 2022

Another attempt to support the rust language #19863

Merged

BKPepe closed this Nov 8, 2022

rust: Initial commit - rust lang #13916

rust: Initial commit - rust lang #13916

Conversation

Grommish commented Nov 14, 2020

BKPepe left a comment

Choose a reason for hiding this comment

Grommish commented Jan 11, 2021 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jyn514 Feb 15, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Grommish commented Feb 15, 2021

Grommish commented Feb 15, 2021 via email

jyn514 commented Feb 15, 2021 • edited Loading

Grommish commented Feb 15, 2021 via email

Grommish commented Mar 21, 2021

wulfy23 commented Mar 23, 2021 • edited Loading

Grommish commented Mar 23, 2021

Grommish commented Mar 26, 2021

predators46 commented Sep 22, 2022

robotronnn commented Sep 22, 2022

predators46 commented Sep 27, 2022

lucize commented Sep 29, 2022

predators46 commented Sep 29, 2022

lucize commented Sep 29, 2022

predators46 commented Sep 29, 2022

lucize commented Sep 30, 2022 • edited Loading

Akira25 commented Sep 30, 2022 • edited Loading

lucize commented Oct 1, 2022

lu-zero commented Oct 2, 2022 • edited Loading

lu-zero commented Oct 20, 2022

neggles commented Oct 22, 2022

lu-zero commented Oct 22, 2022

neggles commented Oct 25, 2022

lu-zero commented Oct 25, 2022

lu-zero commented Nov 7, 2022

jyn514 Feb 15, 2021 •

edited

Loading

jyn514 commented Feb 15, 2021 •

edited

Loading

wulfy23 commented Mar 23, 2021 •

edited

Loading

lucize commented Sep 30, 2022 •

edited

Loading

Akira25 commented Sep 30, 2022 •

edited

Loading

lu-zero commented Oct 2, 2022 •

edited

Loading