miri test is painful to use with CARGO_TARGET_DIR set

[jonhoo]

I have CARGO_TARGET_DIR set on my machine to shared the target directories of the many rust projects on my computer. This generally works very well, and reduces build times (often by a lot!) by re-using build artifacts across crates I'm working on. Unfortunately, miri test also shares that build directory, and when it re-compiles crates with a modified std, it causes non-miri builds to fail with errors like:

error[E0460]: found possibly newer version of crate `core` which `lazy_static` depends on

This is solved easily enough by remembering to always run miri test with:

$ env -u CARGO_TARGET_DIR cargo miri test

But, well, I often forget, and then have to wipe my whole shared target directory, which leads to very long compile times the next time I work on a large project.

I wonder if it might be possible to have miri use it's own "build profile". rls already does this (its artifacts ends up with target/rls), and if miri also did that (target/miri), all of these issues would simply go away! I don't know what mechanisms would be involved, but it would save me (and probably others) a lot of headache by eliminating the error above :)

[RalfJung]

@jonhoo thanks for bringing this up! I assume this would also solve the problem where I have to rm -rf target before doing cargo miri test in some other crate I am debugging.
Unfortunately, I know nothing about "build profiles" and how they relate to the build cache and so on. Help would be welcome. :)

(For another target-dir issue that for some reason you do not run into, it seems, see japaric/xargo#286.)

[jonhoo]

Yes, exactly! It looks like RLS' code for dealing with this lives somewhere around here: https://github.com/rust-lang/rls/blob/1cfb87845f45758442830506b7242947dfc989d9/rls/src/build/cargo.rs#L668-LL686

Huh, yes, I have had nothing that looks like that issue, and I've been using CARGO_TARGET_DIR for a long time. I only use xargo in the context of miri though, so maybe that's why?

[jonhoo]

One simple option might be to use cargo_metadata::MetadataCommand::other_options to set cargo's --target-dir. You'll probably (I think) want to read out the current target-dir with Metadata::target_dir first, and then just append to it the same way rls does?

[jonhoo]

I think maybe you'll need to add support to --target-dir for xargo (something like japaric/xargo#177 ?) for this to work too. Not entirely sure. I have little experience with xargo :) May be worth roping in someone with more cargo experience for this.

[TriplEight]

FWIW as I've mentioned in #1421 (comment)
Right after I've added miri with dependencies to CI docker image, CI started to fail in unrelated places: https://gitlab.parity.io/parity/ink/pipelines/93478/builds

This is because I have CARGO_TARGET_DIR set in CI by default. In order to fix CI had to return to a previous env image and invalidate the corrupted target cache, pretty much the same issue with @jonhoo.

some logs

thread 'rustc' panicked at 'failed to lookup `SourceFile` in new context', src/librustc_middle/ty/query/on_disk_cache.rs:456:17
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/libunwind.rs:86
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:78
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:59
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1069
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1537
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:62
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:49
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:198
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:218
  10: rustc_driver::report_ice
  11: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:481
  12: rust_begin_unwind
             at src/libstd/panicking.rs:385
  13: core::panicking::panic_fmt
             at src/libcore/panicking.rs:89
  14: core::option::expect_failed
             at src/libcore/option.rs:1264
  15: <rustc_middle::ty::query::on_disk_cache::CacheDecoder as serialize::serialize::SpecializedDecoder<rustc_span::span_encoding::Span>>::specialized_decode
  16: <(T10,T11) as serialize::serialize::Decodable>::decode
  17: <rustc_middle::ty::query::on_disk_cache::CacheDecoder as serialize::serialize::SpecializedDecoder<rustc_span::span_encoding::Span>>::specialized_decode
  18: serialize::serialize::Decoder::read_struct_field
  19: <rustc_middle::mir::Statement as serialize::serialize::Decodable>::decode
  20: serialize::serialize::Decoder::read_seq
  21: <rustc_middle::mir::BasicBlockData as serialize::serialize::Decodable>::decode
  22: serialize::serialize::Decoder::read_seq
  23: <rustc_middle::mir::Body as serialize::serialize::Decodable>::decode
  24: rustc_middle::ty::query::on_disk_cache::OnDiskCache::try_load_query_result
  25: rustc_middle::query::<impl rustc_query_system::query::config::QueryDescription<rustc_middle::ty::context::TyCtxt> for rustc_middle::ty::query::queries::optimized_mir>::try_load_from_disk
  26: rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory
  27: rustc_data_structures::stack::ensure_sufficient_stack
  28: rustc_query_system::query::plumbing::get_query_impl
  29: rustc_metadata::rmeta::encoder::EncodeContext::encode_optimized_mir
  30: <rustc_metadata::rmeta::encoder::EncodeContext as rustc_hir::intravisit::Visitor>::visit_expr
  31: rustc_hir::intravisit::walk_expr
  32: <rustc_metadata::rmeta::encoder::EncodeContext as rustc_hir::intravisit::Visitor>::visit_expr
  33: rustc_hir::intravisit::walk_block
  34: <rustc_metadata::rmeta::encoder::EncodeContext as rustc_hir::intravisit::Visitor>::visit_expr
  35: rustc_hir::intravisit::walk_fn
  36: rustc_hir::intravisit::walk_item
  37: <rustc_metadata::rmeta::encoder::EncodeContext as rustc_hir::intravisit::Visitor>::visit_item
  38: rustc_hir::hir::Crate::visit_all_item_likes
  39: rustc_metadata::rmeta::encoder::encode_metadata_impl
  40: rustc_data_structures::sync::join
  41: rustc_metadata::rmeta::decoder::cstore_impl::<impl rustc_middle::middle::cstore::CrateStore for rustc_metadata::creader::CStore>::encode_metadata
  42: rustc_middle::ty::context::TyCtxt::encode_metadata
  43: rustc_interface::passes::start_codegen
  44: rustc_middle::ty::context::tls::enter_global
  45: rustc_interface::queries::Queries::ongoing_codegen
  46: rustc_interface::interface::run_compiler_in_existing_thread_pool
  47: scoped_tls::ScopedKey<T>::set
  48: rustc_ast::attr::with_globals
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
error: internal compiler error: unexpected panic
note: the compiler unexpectedly panicked. this is a bug.
note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
note: rustc 1.45.0-nightly (75e1463c5 2020-05-13) running on x86_64-unknown-linux-gnu
note: compiler flags: -C prefer-dynamic -C embed-bitcode=no -C debuginfo=2 -C incremental --crate-type proc-macro
note: some of the compiler flags provided by cargo are hidden
query stack during panic:
#0 [optimized_mir] processing `has_layout::generate_impl::{{closure}}#0`
end of query stack
error: could not compile `ink_abi_derive`.
Caused by:
  process didn't exit successfully: `sccache rustc --crate-name ink_abi_derive --edition=2018 abi/derive/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -Cembed-bitcode=no -C debuginfo=2 --cfg 'feature="std"' -C metadata=bb76912a532f61e4 -C extra-filename=-bb76912a532f61e4 --out-dir /ci-cache/ink/targets/robin-add-miri-to-ci/check-std/debug/deps -C incremental=/ci-cache/ink/targets/robin-add-miri-to-ci/check-std/debug/incremental -L dependency=/ci-cache/ink/targets/robin-add-miri-to-ci/check-std/debug/deps --extern proc_macro2=/ci-cache/ink/targets/robin-add-miri-to-ci/check-std/debug/deps/libproc_macro2-94499e8bb61cd8d3.rlib --extern quote=/ci-cache/ink/targets/robin-add-miri-to-ci/check-std/debug/deps/libquote-25fd0e804de82445.rlib --extern syn=/ci-cache/ink/targets/robin-add-miri-to-ci/check-std/debug/deps/libsyn-a49a3cb16f2c92bf.rlib --extern proc_macro` (exit code: 101)
warning: build failed, waiting for other jobs to finish...
error: build failed

Not very clean though because I also have sccache set there

Happy to run some pipelines if you suggest something.