cfg(doctest) doesn't work as expected

[GuillaumeGomez]

First, create empty lib crate.

cargo init --lib foo

With use crate::foo::bar:

//! Hello
//!
//! ```
//! use crate::foo::bar;
//! bar();
//! ```

#[cfg(doctest)]
pub fn bar ()
{
  println!("hello");
}

   Compiling foo v0.1.0 (/home/user/tmp/foo)
    Finished test [unoptimized + debuginfo] target(s) in 0.01s
     Running target/debug/deps/foo-d4d7a4ec5556181e

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

   Doc-tests foo

running 1 test
test src/lib.rs -  (line 3) ... FAILED

failures:

---- src/lib.rs -  (line 3) stdout ----
error[E0432]: unresolved import `crate::foo::bar`
 --> src/lib.rs:4:5
  |
4 | use crate::foo::bar;
  |     ^^^^^^^^^^^^^^^ no `bar` in the root

error: aborting due to previous error

For more information about this error, try `rustc --explain E0432`.
Couldn't compile the test.

failures:
    src/lib.rs -  (line 3)

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

error: test failed, to rerun pass '--doc'

With use foo::bar:

//! Hello
//!
//! ```
//! use foo::bar;
//! bar();
//! ```

#[cfg(doctest)]
pub fn bar ()
{
  println!("hello");
}

   Compiling foo v0.1.0 (/home/user/tmp/foo)
    Finished test [unoptimized + debuginfo] target(s) in 0.24s
     Running target/debug/deps/foo-d4d7a4ec5556181e

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

   Doc-tests foo

running 1 test
test src/lib.rs -  (line 3) ... FAILED

failures:

---- src/lib.rs -  (line 3) stdout ----
error[E0432]: unresolved import `foo::bar`
 --> src/lib.rs:4:5
  |
4 | use foo::bar;
  |     ^^^^^^^^ no `bar` in the root

error: aborting due to previous error

For more information about this error, try `rustc --explain E0432`.
Couldn't compile the test.

failures:
    src/lib.rs -  (line 3)

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

error: test failed, to rerun pass '--doc'

//! Hello
//!
//! ```
//! use crate::bar;
//! bar();
//! ```

#[cfg(doctest)]
pub fn bar ()
{
  println!("hello");
}

With use crate::bar:

   Compiling foo v0.1.0 (/home/user/tmp/foo)
    Finished test [unoptimized + debuginfo] target(s) in 0.23s
     Running target/debug/deps/foo-d4d7a4ec5556181e

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

   Doc-tests foo

running 1 test
test src/lib.rs -  (line 3) ... FAILED

failures:

---- src/lib.rs -  (line 3) stdout ----
error[E0432]: unresolved import `crate::bar`
 --> src/lib.rs:4:5
  |
3 | use crate::bar;
  |     ^^^^^^^^^^ no `bar` in the root

error: aborting due to previous error

For more information about this error, try `rustc --explain E0432`.
Couldn't compile the test.

failures:
    src/lib.rs -  (line 3)

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

error: test failed, to rerun pass '--doc'

[GuillaumeGomez]

It seems to be way more complicated than I thought... Apparently, you need the code to be compiled in order to be able to link to it when running tests. However, this first compilation doesn't go through rustdoc at all and therefore, doctest isn't used. So unless I'm missing something, we're kinda stuck because it'd mean that we'd have to recompile the current lib by adding the doctest feature.

cc @rust-lang/compiler @rust-lang/cargo

[GuillaumeGomez]

cc @sunjay (in case you saw something I missed?)

[sunjay]

So unless I'm missing something, we're kinda stuck because it'd mean that we'd have to recompile the current lib by adding the doctest feature.

@GuillaumeGomez I'm not familiar with this stuff at all, but I guess we'd have to do something similar to what we do for #[cfg(test)] then? We would need to compile the crate with cfg(test) and then with cfg(doctest). Then we can run each with their respective versions of the crate. This seems like a reasonable approach. Is it difficult to implement?

My original issue just suggested setting cfg(test) during all tests (including doctests), which would be easier to implement, but probably not desirable given how much it would break things. It's probably best if we figure out a way to recompile the current lib with cfg(doctest).

[GuillaumeGomez]

It needs to be done on cargo then. I had a discussion with @ehuss about this and they told me that it could be a big burden on performances. So To be debated...

[eclipseo]

The following is not working, I'm wondering if this is caused by the present bug of if I'm not using cfg(doctest) as intended?

mod hawktracer {
  cfg_if::cfg_if! {
    // Do not mix tracing and tests
    if #[cfg(all(feature="tracing", not(test), not(doctest)))] {
      pub use rust_hawktracer::*;
    } else {
      pub use noop_proc_macro::hawktracer;
    }
  }
}

[sunjay]

@eclipseo That's because of this bug. The cfg currently only works while collecting tests to run. It does not affect compilation in the way you'd expect.

[GuillaumeGomez]

Ping @rust-lang/cargo

[ehuss]

I don't think we have any additional input to provide. We feel like the cost of rebuilding the library a third time is too high. Unfortunately I don't have any suggestions for alternative approaches.

[GuillaumeGomez]

Could we make this rebuild a bit smarter and only perform it if we have a #[cfg(doctest)] somewhere? (and of course, not run it on the dependencies)

[mgeisler]

Hi @GuillaumeGomez,

I just ran into this also — I wanted to add a helper function for my doctests but I couldn't get it to work at all. I was initially encouraged by your cfg(doctest) is stable and you should use it article, but I now understand that it's only talking about the test collection phase, not the actual test compilation phase.

Could you perhaps update the article with a warning about this (and a link to this bug)? I think that would be helpful for future readers :-)

[GuillaumeGomez]

@mgeisler I updated the blog post.

[mgeisler]

Thanks a lot!

[Fishrock123]

I also ran into the need for this and reported it to zulip: https://rust-lang.zulipchat.com/#narrow/stream/122651-general/topic/.23.5Bcfg(test).5D.20within.20doc-tests

My usage is more-or-less to have the #[async_std::test] attribute also turn on when in doctests.

My specific usage is I would like cargo check to run for this example:
https://docs.rs/preroll/0.2.0/preroll/test_utils/fn.assert_json_error.html

Keep in mind I am trying to provide an accurate example of a test utility.

[domenicquirl]

Hi all, I'd like to add that running into this is somewhat confusing if you don't happen to have read the aforementioned blog post. At least, it wasn't clear to me from this section in the rustdoc book that this doesn't work. Maybe this should be clarified if it's not on track to change anytime soon?

[armanriazi]

yes it doesn't work as expected

1. I created crate check_doc_test.rs in lib module

#[cfg(doctest)]
pub mod check_doc_test {
    pub fn check_doc_macro(){
        doc_comment::doctest!("../README.md");
    }


    #[cfg(doctest)]
    pub fn bar ()
    {
    println!("hello");
    }
}

2. Cargo.toml

[package]
edition = "2021"
name = "kno_lib"
version = "0.1.0"

[dev-dependencies]
doc-comment = "0.3.3"`

3. cargo test --doc --workspace

I can not see any log test as "test result: ok."

4. cargo doc --workspace --message-format short --no-deps --open --color always

` There is not any contents of REAMD.md on doc files in browser

5. with apprericate. I am looking forward

[GuillaumeGomez]

This is a cargo limitation unfortunately.

[tomkarw]

Run into this today.

My original issue just suggested setting cfg(test) during all tests (including doctests), which would be easier to implement, but probably not desirable given how much it would break things. It's probably best if we figure out a way to recompile the current lib with cfg(doctest).

I think this a clean and simple solution to this problem. How would it break things? The only case I can think of is:

#[cfg(test)]
struct ConflictingName;

#[cfg(doctest)]
struct ConflictingName;

but it seems convoluted.

Also, it could be tested with crater if the worries persist.

I think the current behavior is a major shortcoming and needs addressing, as for now there is no way to add doctest specific utils.

[dan-da]

I also ran into this. In my doctests I need functions that are behind cfg(test) for mock/setup purposes. Such fn do not belong in release code.

Judging from the number of "me too" comments here, this seems a pretty common problem for people.

If another compilation for cfg(doctest) cannot be added, then it really seems that cargo test --doc should compile cfg(test) marked blocks at least.