Add stdlib & add bit module

[vrindisbacher]

No description provided.

[vrindisbacher]

@nilehmann, I'm trying to figure out what the best way to do this would be. I cannot add a module to flux-rs because you can't use proc macros in the same crate as they are defined. So I figured I would isolate the macros to a different crate and then have flux-rs re-export them, along with standard library modules.

This works fine for the macros, but unfortunately, it doesn't seem to work well for the stdlib. If I try to compile, I get errors like:

error[E0433]: failed to resolve: use of undeclared crate or module `flux_tool`
  --> lib/flux-rs/src/std/bit.rs:89:13
   |
89 | /             #[sig(fn ($t[@b1], $t[@b2]) -> bv_shl(b1, b2))]
90 | |             fn shr(self, rhs: $t) -> $t { $name(self.0 >> rhs.0) }
   | |__________________________________________________________________^ use of undeclared crate or module `flux_tool`
...
95 |   shr_impl! { B8, B8 B16, B16 B32, B32 B64, B64 B128, B128 Bi8, Bi8 Bi16, Bi16 Bi32, Bi32 Bi64, Bi64 Bi128, Bi128 }
   |   ----------------------------------------------------------------------------------------------------------------- in this macro invocation
   |
   = note: this error originates in the macro `shr_impl` (in Nightly builds, run with -Z macro-backtrace for more info)

I think this has to do with conditional compilation of proc macros using the flux_sysroot environment variable (seems like the proc macros are dummies during normal compilation). So I'm not quite sure what to do if I want to use those proc macros in the stdlib.

I'm thinking that somehow, I need to prevent flux from actually running these macros during compilation (as is done by the proc macro crate). Do you have any ideas on how to do that?

[nilehmann]

Hmm, I don't see why this isn't working 🤔 . I would have done the same. I'll take a look

[nilehmann]

Oh, I see. It's tricky. I'm not sure what is the best way to fix it.

As you mentioned, the crates in lib use the flux_sysroot cfg to be compiled in either "normal mode" or in "flux mode". The build scripts set the cfg if the FLUX_BUILD_SYSROOT env variable is present. If you add flux-rs as a dependency there are two cases:

• When running cargo build, the env var won't be set and the crates will be compiled in normal mode. This means all the proc macros are a no-op.

• When running cargo flux, the cargo-flux binary sets the env var (https://github.com/flux-rs/flux/blob/main/crates/flux-bin/src/bin/cargo-flux.rs#L45) and the crates will be compiled in flux mode.

In flux mode, all the proc macros get replaced by a #[flux_tool::*] attribute. flux_tool is not a real crate. We instead use the unstable register_tool feature which is enabled by the driver (https://github.com/flux-rs/flux/blob/main/crates/flux-driver/src/bin/main.rs#L52-L54)

All this is actually working. If you ignore the error you are getting with cargo x install and add flux-rs as a dependency to another crate, both cargo build and cargo flux will work. The problem though is that cargo x install also prebuilds the flux-rs in flux mode so it can be "linked" to the rustc-flux binary. To build flux-rs we (more or less) run

FLUX_BUILD_SYSROOT=1 cargo build -p flux-rs

So that's the issue, setting the en var will cause the proc macros in flux-rs to expand to #[flux_tool::*] but since we are running cargo build (and not cargo flux) the crate is compiled with plain rustc which doesn't enable the register_tool feature.

The bottom line is that we need to enable the register_tool feature when prebuilding flux-rs. That code is here https://github.com/flux-rs/flux/blob/main/xtask/src/main.rs#L150. I think the easiest is to set the RUSTFLAGS env var. Alternatively, we could prebuild flux-rs with one of our rustc wrappers (rustc-flux or flux-driver) that already enable the feature. That'd require tweaking the binaries though.

[nilehmann]

The same applies for tests. Running cargo x test prebuilds flux-rs such that it can be used by tests.

[nilehmann]

On further thought, the only option is to prebuild it with one of our wrappers to generate the flux metadata file

[vrindisbacher]

So you're saying that we need to build this using flux-driver? Would that just involve adding a build.rs that runs cargo build on flux-rs? And then omitting it from the xtask install?

[nilehmann]

I'm not entirely sure this is the best way, but I'm saying that in cargo x install we should run cargo flux -p flux-rs instead of cargo build -p flux-rs or something to that effect (and similar for cargo x test). Running cargo flux doesn't currently work though for reasons. It's probably better to talk in person.

[vrindisbacher]

@nilehmann, Ok I think this works. One thing I noticed is that I am unable to use std attributes when using rustc flux but i can use the others no problem.

Also, I think the CI will need some tweaks. The tests etc work for me locally.