Allow writing metadata without llvm #44085

bjorn3 · 2017-08-25T17:34:10Z

Todo:

Rebase
Fix eventual errors
~~Find some crate to write elf files~~ (will do it later)

Cc #43842

rust-highfive · 2017-08-25T17:34:15Z

r? @arielb1

(rust_highfive has picked a reviewer for you, use r? to override)

Mark-Simulacrum

I'm happy to see this as a possibility! Left a few minor comments about bootstrap.

Mark-Simulacrum · 2017-08-25T23:41:07Z

src/bootstrap/compile.rs

+            builder.cargo(compiler, Mode::Libstd, target, "build")
+        }else{
+            builder.cargo(compiler, Mode::Libstd, target, "check")
+        };


This seems wrong/unrelated? If not, reasoning would be appreciated.

Should make this no llvm only. When stage 1 rustc doesnt have llvm support it cant build regular rlibs, but only metadata only rlibs.

Ah. I suppose that makes sense.

Mark-Simulacrum · 2017-08-25T23:41:28Z

src/bootstrap/compile.rs

@@ -161,6 +165,7 @@ pub fn std_cargo(build: &Build,
        // missing
        // We also only build the runtimes when --enable-sanitizers (or its
        // config.toml equivalent) is used
+        //cargo.env("RUST_FLAGS", "-Zno-trans");


This will need to go before we merge.

bjorn3 · 2017-08-26T08:16:09Z

src/librustc_driver/driver.rs

+                        ::rustc_trans_utils::find_exported_symbols(tcx, &analysis.reachable);
+                    let (metadata, _hashes) =
+                        cstore.encode_metadata(tcx, &link_meta, &exported_symbols);
+                    let mut builder = Builder::new(File::create("abc.rlib").unwrap());


Forgot to make this write to the file asked to, instead of abc.rlib.

bjorn3 · 2017-08-26T15:42:09Z

$ ./x.py build --stage 1
[...]
Building stage1 compiler artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
   Compiling rustc_back v0.0.0 (file:///Users/bjorn/Documents/rust_fork/src/librustc_back)
   Compiling syntax v0.0.0 (file:///Users/bjorn/Documents/rust_fork/src/libsyntax)
   Compiling log v0.3.8
[...]
   Compiling rustc_save_analysis v0.0.0 (file:///Users/bjorn/Documents/rust_fork/src/librustc_save_analysis)
warning: LLVM not supported, so non rlib output type dylib will be treated like rlib libraries

warning: LLVM not supported, so non rlib output type dylib will be treated like rlib libraries

   Compiling rustc-main v0.0.0 (file:///Users/bjorn/Documents/rust_fork/src/rustc)
error: LLVM is not supported by this rustc

error: Could not compile `rustc-main`.

Caused by:
  process didn't exit successfully: `/Users/bjorn/Documents/rust_fork/no_llvm_build/build/bootstrap/debug/rustc --crate-name rustc /Users/bjorn/Documents/rust_fork/src/rustc/rustc.rs --error-format json --crate-type bin --emit=dep-info,link -C opt-level=2 --cfg feature="jemalloc" --cfg feature="rustc_back" -C metadata=9c436ad388a4a0ca -C extra-filename=-9c436ad388a4a0ca --out-dir /Users/bjorn/Documents/rust_fork/no_llvm_build/build/x86_64-apple-darwin/stage1-rustc/x86_64-apple-darwin/release/deps --target x86_64-apple-darwin -L dependency=/Users/bjorn/Documents/rust_fork/no_llvm_build/build/x86_64-apple-darwin/stage1-rustc/x86_64-apple-darwin/release/deps -L dependency=/Users/bjorn/Documents/rust_fork/no_llvm_build/build/x86_64-apple-darwin/stage1-rustc/release/deps --extern rustc_back=/Users/bjorn/Documents/rust_fork/no_llvm_build/build/x86_64-apple-darwin/stage1-rustc/x86_64-apple-darwin/release/deps/librustc_back-853a8b36deced665.dylib --extern rustc_driver=/Users/bjorn/Documents/rust_fork/no_llvm_build/build/x86_64-apple-darwin/stage1-rustc/x86_64-apple-darwin/release/deps/librustc_driver-cc8fef39b95e564b.dylib -L native=/Users/bjorn/Documents/rust_fork/no_llvm_build/build/x86_64-apple-darwin/stage1-rustc/x86_64-apple-darwin/release/build/miniz-sys-e7bda6e0ddcea6f1/out` (exit code: 101)
thread 'main' panicked at 'command did not execute successfully: "/Users/bjorn/Documents/rust_fork/no_llvm_build/build/x86_64-apple-darwin/stage0/bin/cargo" "build" "-j" "8" "--target" "x86_64-apple-darwin" "--release" "--features" " jemalloc" "--manifest-path" "/Users/bjorn/Documents/rust_fork/src/rustc/Cargo.toml" "--message-format" "json"
expected success, got: exit code: 101', /Users/bjorn/Documents/rust_fork/src/bootstrap/compile.rs:883:8
[...]

bjorn3 · 2017-08-28T17:06:03Z

What is this error message of the "Fix tidy errors" commit:

[00:16:47] error: linking with `cc` failed: exit code: 1
[00:16:47]   |
[00:16:47]   = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/release/build/rustc_driver-78c26d64350eb204/build_script_build-78c26d64350eb204.0.o" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/release/build/rustc_driver-78c26d64350eb204/build_script_build-78c26d64350eb204" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/release/build/rustc_driver-78c26d64350eb204/build_script_build-78c26d64350eb204.crate.allocator.o" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-nodefaultlibs" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/release/deps" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-ea2e08566ed2ca9b.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/librand-d011a0d28d72d28b.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc_jemalloc-117698016bc6c938.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc_system-aa6c11e69908f4f9.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-d94cd37ff18e6e1c.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-388107ef329739bd.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-c45b3574292a7ea1.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-c791e7351d8a272a.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_unicode-54feb014ef663465.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-81b84a407143b5e1.rlib" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-ba9896bad78dc777.rlib" "-Wl,-Bdynamic" "-l" "dl" "-l" "rt" "-l" "pthread" "-l" "pthread" "-l" "gcc_s" "-l" "c" "-l" "m" "-l" "rt" "-l" "pthread" "-l" "util"
[00:16:47]   = note: /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/librand-d011a0d28d72d28b.rlib: error adding symbols: Archive has no index; run ranlib to add one
[00:16:47]           collect2: error: ld returned 1 exit status
[00:16:47]           
[00:16:47] 
[00:16:47] error: aborting due to previous error
[00:16:47] 
[00:16:47] error: Could not compile `syntax`.

arielb1 · 2017-08-29T18:51:45Z

[00:02:07] ======================================================================

[00:02:07] FAIL: program_config (bootstrap.RustBuild)

[00:02:07] Doctest: bootstrap.RustBuild.program_config

[00:02:07] ----------------------------------------------------------------------

[00:02:07] Traceback (most recent call last):

[00:02:07]   File "/usr/lib/python2.7/doctest.py", line 2226, in runTest

[00:02:07]     raise self.failureException(self.format_failure(new.getvalue()))

[00:02:07] AssertionError: Failed doctest test for bootstrap.RustBuild.program_config

[00:02:07]   File "/checkout/src/bootstrap/bootstrap.py", line 519, in program_config

[00:02:07] 

[00:02:07] ----------------------------------------------------------------------

[00:02:07] File "/checkout/src/bootstrap/bootstrap.py", line 527, in bootstrap.RustBuild.program_config

[00:02:07] Failed example:

[00:02:07]     cargo_path.rstrip(".exe") == os.path.join("/tmp/rust",

[00:02:07]     "bin", "cargo")

[00:02:07] Expected:

[00:02:07]     True

[00:02:07] Got:

[00:02:07]     False

arielb1 · 2017-08-29T18:52:30Z

src/bootstrap/bin/rustc.rs

@@ -159,6 +159,10 @@ fn main() {
            cmd.arg("-C").arg("panic=abort");
        }

+        if cfg!(not(feature="llvm")) && stage != "0" {


Didn't you mean !self.build.config.llvm_enabled here?

This doesnt have self.build.config but i dont think its compiled with the feature, so i think i will have to use an env var.

Yeah, an env var would be fine. I think there's already precedent for that (i.e. grep for a env var with LLVM in it) so there may not be a need to add a new one.

arielb1 · 2017-09-05T18:10:31Z

[00:35:42] error[E0308]: mismatched types
[00:35:42]    --> /checkout/src/librustdoc/test.rs:260:31
[00:35:42]     |
[00:35:42] 260 |         driver::compile_input(&sess, &cstore, &input, &out, &None, None, &control)
[00:35:42]     |                               ^^^^^ types differ in mutability
[00:35:42]     |
[00:35:42]     = note: expected type `&mut rustc::session::Session`
[00:35:42]                found type `&rustc::session::Session`

bors · 2017-09-08T14:43:21Z

☔ The latest upstream changes (presumably #44142) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2017-09-16T19:57:02Z

☔ The latest upstream changes (presumably #43017) made this pull request unmergeable. Please resolve the merge conflicts.

bjorn3 · 2017-09-17T08:33:09Z

I added a generic TransCrate trait, which is to be implemented by compiler backends.

arielb1 · 2017-09-17T11:06:52Z

src/bootstrap/builder.rs

@@ -524,14 +524,20 @@ impl<'a> Builder<'a> {
        // For other crates, however, we know that we've already got a standard
        // library up and running, so we can use the normal compiler to compile
        // build scripts in that situation.


Also modify the comment?

arielb1 · 2017-09-17T11:08:04Z

src/librustc_driver/driver.rs

@@ -243,7 +248,48 @@ pub fn compile_input(sess: &Session,
                tcx.print_debug_stats();
            }

-            let trans = phase_4_translate_to_llvm(tcx, analysis, incremental_hashes_map,
+            #[cfg(not(feature="llvm"))]


Consider moving this outside of the main driver function?

Should remove the #[cfg]

arielb1 · 2017-09-17T11:10:47Z

src/librustc_driver/driver.rs

@@ -243,7 +248,48 @@ pub fn compile_input(sess: &Session,
                tcx.print_debug_stats();
            }

-            let trans = phase_4_translate_to_llvm(tcx, analysis, incremental_hashes_map,
+            #[cfg(not(feature="llvm"))]
+            {


I'm also not sure about the idea of creating dylibs that are actually rlibs.

Without llvm we cant create real dylibs, but many rustc crates are compiled to dylibs, so by emitting rlibs instead we can compile all rustc libs.

arielb1 · 2017-09-17T11:12:10Z

src/librustc_driver/lib.rs


-#[cfg(not(feature="llvm"))]
 mod no_llvm_metadata_loader {


Move this to an out-of-line module?

arielb1 · 2017-09-17T11:12:31Z

src/librustc_driver/lib.rs

-#[cfg(not(feature="llvm"))]
-mod rustc_trans {
-    use syntax_pos::symbol::Symbol;
+mod trans_metadata_only {


Move this to an out-of-line module?

arielb1 · 2017-09-17T11:17:16Z

src/librustc_trans_traits/lib.rs

+    fn get_dylib_metadata(&self, _target: &Target, _filename: &Path) -> Result<ErasedBoxRef<[u8]>, String> {
+        bug!("DummyMetadataLoader::get_dylib_metadata");
+    }
+}


Maybe also move the "metadata-only" TransCrate implementation ~~here~~ to rustc_trans_utils?

arielb1 · 2017-09-17T11:21:15Z

src/librustc_trans_traits/Cargo.toml

@@ -0,0 +1,17 @@
+[package]


Why does this exist as a separate crate from rustc_trans_utils? They are both depended upon by exactly rustc_trans and rustc_driver, and they semantically perform a very similar task.

I was thinking about utils to be used by backends only, while traits contains the glue between backends and rustc_driver. I can merge them though.

arielb1 · 2017-09-17T11:23:43Z

src/librustc_driver/lib.rs


-    use self::back::write::OngoingCrateTranslation;
+#[cfg(not(feature="llvm"))]
+mod rustc_trans {


Can you move all of these items to the TransCrate trait?

No, because most of them are llvm specific, eg they would still use llvm even if trans is non llvm. (enable_llvm_debug, print_passes)

bjorn3 · 2017-09-23T13:48:28Z

@arielb1 rebased and fixed most nits

arielb1 · 2017-09-24T09:38:24Z

@bors r+

bors · 2017-09-24T09:38:25Z

📌 Commit 3c32c6a has been approved by arielb1

bors · 2017-09-25T00:19:55Z

⌛ Testing commit 3c32c6a with merge 0e7f2f843dce90acc9e8ec1f29665ba69ac1f844...

bors · 2017-09-25T00:38:07Z

💔 Test failed - status-travis

bjorn3 · 2017-09-25T06:19:34Z

Error should be fixed now.

arielb1 · 2017-09-25T09:03:12Z

@bors r+

bors · 2017-09-25T09:03:12Z

📌 Commit 843cd5b has been approved by arielb1

bors · 2017-09-25T18:05:28Z

⌛ Testing commit 843cd5b with merge 3df1f7b...

Allow writing metadata without llvm # Todo: * [x] Rebase * [x] Fix eventual errors * [x] <strike>Find some crate to write elf files</strike> (will do it later) Cc #43842

bors · 2017-09-25T20:36:57Z

☀️ Test successful - status-appveyor, status-travis
Approved by: arielb1
Pushing 3df1f7b to master...

rust-highfive assigned arielb1 Aug 25, 2017

bjorn3 force-pushed the no_llvm_write_metadata branch from 2a63467 to dd253c1 Compare August 25, 2017 17:51

shepmaster added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Aug 25, 2017

Mark-Simulacrum requested changes Aug 25, 2017

View reviewed changes

bjorn3 commented Aug 26, 2017

View reviewed changes

arielb1 reviewed Aug 29, 2017

View reviewed changes

bjorn3 force-pushed the no_llvm_write_metadata branch 4 times, most recently from bec1f52 to ea497a2 Compare September 16, 2017 16:51

bjorn3 force-pushed the no_llvm_write_metadata branch 3 times, most recently from b88fcf7 to 243d64f Compare September 17, 2017 08:26

bjorn3 changed the title ~~[WIP] Allow writing metadata without llvm~~ Allow writing metadata without llvm Sep 17, 2017

arielb1 reviewed Sep 17, 2017

View reviewed changes

Fix for upstream changes

e130ccc

bjorn3 force-pushed the no_llvm_write_metadata branch from 06155cf to e130ccc Compare September 23, 2017 12:48

bjorn3 added 5 commits September 23, 2017 15:00

Fix some nits

43bfd4c

Remove build_diagnostic_array hack

793c02d

Fix tidy errors

70e5bae

Remove leftover

95fb1d0

Remove now unused dependency on flate2 from rustc_driver

1800520

Fix error

56c4670

bjorn3 force-pushed the no_llvm_write_metadata branch from 42e12da to 56c4670 Compare September 23, 2017 16:08

Fix some tests with no llvm build

3c32c6a

Fix error

843cd5b

bors merged commit 843cd5b into rust-lang:master Sep 25, 2017

This was referenced Sep 25, 2017

rustc: Implement ThinLTO #44841

Merged

rustc: Enable LTO and multiple codegen units #44783

Closed

bjorn3 deleted the no_llvm_write_metadata branch September 26, 2017 06:17

eddyb mentioned this pull request Oct 14, 2017

Remove librustc_codegen_utils. #45276

Closed

bjorn3 mentioned this pull request Jan 8, 2019

MetadataOnlyCodegenBackend: run the collector only once #57418

Merged

bjorn3 mentioned this pull request Mar 14, 2020

Refactorings to get rid of rustc_codegen_utils #69965

Merged

Allow writing metadata without llvm #44085

Allow writing metadata without llvm #44085

Conversation

bjorn3 commented Aug 25, 2017 • edited Loading

Todo:

rust-highfive commented Aug 25, 2017

Mark-Simulacrum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bjorn3 commented Aug 26, 2017

bjorn3 commented Aug 28, 2017

arielb1 commented Aug 29, 2017

arielb1 Aug 29, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arielb1 commented Sep 5, 2017 • edited Loading

bors commented Sep 8, 2017

bors commented Sep 16, 2017

bjorn3 commented Sep 17, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arielb1 Sep 17, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bjorn3 commented Sep 23, 2017

arielb1 commented Sep 24, 2017

bors commented Sep 24, 2017

bors commented Sep 25, 2017

bors commented Sep 25, 2017

bjorn3 commented Sep 25, 2017

arielb1 commented Sep 25, 2017

bors commented Sep 25, 2017

bors commented Sep 25, 2017

bors commented Sep 25, 2017

bjorn3 commented Aug 25, 2017 •

edited

Loading

arielb1 Aug 29, 2017 •

edited

Loading

arielb1 commented Sep 5, 2017 •

edited

Loading

arielb1 Sep 17, 2017 •

edited

Loading