Doctests do not set profile codegen options

[bspeice]

Problem
As currently stands, there's an inconsistency in doctests running with debug_assertions on even when code was compiled through cargo test --release. I would've expected that cargo test --release means that the doctests get compiled in release mode as well, but that doesn't appear to be the case.

Steps

//! Here's a doctest:
//!
//! ```rust
//! use doctest_da::my_function;
//!
//! if cfg!(debug_assertions) {
//!     assert!(my_function().is_ok());
//! } else {
//!     assert!(my_function().is_err());
//! }
//! ```

pub fn my_function() -> Result<(), ()>{
    if cfg!(debug_assertions) {
        Ok(())
    } else {
        Err(())
    }
}

Notes

I'm a bit confused on where exactly this issue may need to be filed. It seems related to #5777 and #5218, but because this hits doctests, #4669 and rust-lang/rust#45599 may take precedence. Let me know if I should move.

Output of cargo version:

cargo 1.32.0 (8610973aa 2019-01-02)

[ehuss]

It looks like there are few issues here.

• Cargo doesn't pass codegen options to rustdoc. I don't immediately see a reason not to.
• In this specific case, it wouldn't help. rustdoc does not parse options the same way rustc does, and seems to always set debug_assertions to true.

@QuietMisdreavus (or @GuillaumeGomez) it looks like there is some command-line parsing and validation done in rustc in build_session_options_and_crate_config that rustdoc does not do. In this example, debug_assertions looks to get unset here. Is there a reason rustdoc doesn't share that code and set up the Options the same?

[Lokathor]

-C target-cpu in the .cargo/config file also doesn't seem to get passed to doc tests.

[ehuss]

I'm not sure what the current status is of rustdoc handling -C flags in doctests. Someone will need to do an audit to make sure that they are all handled the same way as rustc does.

Cargo's handling of these arguments for rustc is done in a function called build_base_args. The flags that are shared between rustc and rustdoc should probably be split off into a separate function. Arguments for doctests are collected here, that place would likely need to also include the -C codegen options that are appropriate for the doctest (by calling the "separate function" that factored out the appropriate codegen options).

S-blocked-external can be removed if we conclude that rustdoc is handling all the options correctly.

There is some risk that adding these flags will start breaking some projects (like if they change behavior on debug_assertions). I don't know if that risk is great enough for us to do something different (like maybe doing a crater run?). I'm inclined to just change it and respond if people run into problems, but I'd like to hear what others think.

For historical perspective: rustdoc originally did not honor the -C flag at all. At some point, it was refectored to use the same option parsing as rustc, but the codegen options themselves weren't wired into the doctest generation. Over time, those have been added, but I am uncertain if all of them are wired up now, or if all the options are handled the same way as rustc (since there are some subtle issues like how some flags imply other flags).

[ehuss]

There might also be performance considerations (either positive or negative). Doctests are particularly slow since they essentially require compiling an executable for every test. For some projects with lots of doctests (like the standard library), this can be relatively slow. Compiling doctests with different codegen settings might be faster or slower to compile, and may be faster or slower to execute. I don't think that will be a significant concern, but something I wanted to highlight as something to keep an eye on.

[Lokathor]

I can say that personally, I've begun actively avoiding all doc tests, and even ripping out existing doc tests, because they are so slow compared to normal tests.

However, assuming your number of doctests doesn't change, then altering the codegen options applied to particular doctests seems unlikely to have a significant effect on compilation time. Any particular doctest is, in general, likely to be extremely small.