Test using external command hangs

[glts]

Thank you for this useful project.

My project’s test suite includes integration tests that can be enabled with feature miltertest-tests.

cargo test --features miltertest-tests

These tests use the external utility miltertest. miltertest can be installed on Debian and Ubuntu with apt install miltertest or apt install opendkim-tools.

When I try to run these tests with Tarpaulin, they time out after 60 seconds.

cargo tarpaulin --features miltertest-tests

running 2 tests
test basic_stages has been running for over 60 seconds
test eoh_eom_macros has been running for over 60 seconds
Mar 10 08:51:58.266 ERROR cargo_tarpaulin: Failed to get test coverage! Error: Failed to run tests: Error: Timed out waiting for test response

I tried with --follow-exec, too, but it didn’t help. I’m on Ubuntu 20.04 using Rust 1.59.0.

[xd009642]

If you want to try this PR branch #962 it might solve the issue, I'm just trying to solve a flaky test before I can get it merged and do a new release 🙏

[glts]

Tried the branch quickly, it doesn’t help unfortunately. The test still hangs, also with --follow-exec and --follow-exec --forward.

[Nate-Wilkins]

Hi!

I'm having the same issue when using std::Command and the following test:

#[test]
fn test_command_completions_type_zsh() -> Result<(), Box<dyn std::error::Error>> {
    // Given the CLI.
    let mut cmd = Command::cargo_bin("mallardscript")?;

    // When the user generates completions for zsh.
    let result = cmd.arg("completions").arg("--type").arg("zsh").assert();

    result
        // Then no errors occurred.
        .success()
        .stderr(predicate::str::is_empty())
        // Then completions for zsh were outputted.
        .stdout(predicate::str::contains("#compdef mallardscript"));

    return Ok(());
}

but I don't entirely understand how code coverage works except from a highlevel.

When running it against library code it works great!

[xd009642]

You can always try --engine llvm as that's a non-signal based approach

[Nate-Wilkins]

@xd009642 interesting suggestion, thanks!

When trying it locally cargo tarpaulin --engine llvm it says I have 0% test coverage across the CLI tool even though I do have quite a few tests. The tests do run though which was not the case when passing no arguments.

[xd009642]

@Nate-Wilkins do you have a repo I can try? The binaries would need to be built while tarpaulin is running via the build commands tarpaulin runs or by setting rustflags when building beforehand (can add --print-rust-flags to a tarpaulin invocation to see the rustflags you need to set). Would maybe also need to try with --objects and the paths to commands ran if they are built before the tarpaulin run.

Some things with spawning don't really work well with llvm coverage tbh because the writing of coverage data lacks protection from multiple writers so say if you fork a process both child and writer may attempt to write to the same file. Unfortunately this is an issue in the llvm profiling runtime.

ptrace signal stuff is pretty annoying tbh, it's very poorly documented and was only really designed for single threaded applications - plus it behaves significantly differently across multiple distros. I really need to dedicate some time to figure out a approach that works across them all 😢 .

[xd009642]

I tested @glts project with --engine llvm and it works with it (results at end). So that's at least a solution for them. Just need an example from Nate to see if I can track that down and when that's resolved I'll likely close this as there is an issue somewhere about ptrace improvements and project specific support will all be resolved

|| Uncovered Lines:
|| src/callbacks.rs: 46-47, 240, 244-245, 258, 262-263
|| src/connection.rs: 78, 138, 153, 166
|| src/context.rs: 61-65, 176-180, 186-189, 195-196, 378, 409, 415, 418, 509, 514, 524, 530-531, 533-534, 537-539, 542, 544, 546, 556, 559, 564, 574, 579-580, 583-585, 588, 590, 592, 600, 608, 610, 622-624, 627, 629, 631, 634, 638-639, 642-644, 647, 649, 654, 660, 662, 674-675, 677, 680, 684-685, 688-690, 693, 695, 705-706, 738, 741, 782, 785
|| src/ffi_util.rs: 26-27, 38-40, 42, 44
|| src/lib.rs: 151, 161-164, 166-168, 208, 224-225, 227
|| src/listener.rs: 36-37, 42-43, 54-55, 60-61
|| src/macros.rs: 61, 63-67, 77, 79-84, 93-98, 109-112, 115
|| src/message/command.rs: 72, 78, 86, 94, 100, 108, 118, 124, 132, 173-184, 233-239, 241-247, 254, 266, 274-277, 299, 341-342, 344, 364-366, 395, 397, 400, 402, 404, 406, 489, 510, 529, 552-553, 555, 568, 575
|| src/message/mod.rs: 103-104, 126-127, 131-132, 137-138, 174
|| src/message/reply.rs: 77, 80, 83-84, 86, 88-92, 103-105, 107, 110-111, 113, 115-119, 121, 137-143, 235, 237, 240, 242, 245-247, 249, 252, 256, 282, 285, 287-289, 291, 294, 303-305, 307, 316, 318, 320, 322-324, 341-342, 344, 374, 377-379, 381, 383-385, 388, 401-403, 405, 407-409, 411, 425-427, 429, 431-433, 445
|| src/proto_util.rs: 52-53, 59, 119-125, 141-142, 147-148
|| src/session.rs: 94, 123-125, 167-175, 177-178, 180-181, 183-184, 191-195, 207-208, 237, 303, 335, 347, 360, 374, 388, 405, 410, 415, 419, 425, 428, 435, 438, 448, 455, 458, 462, 465, 470, 473, 480, 487, 492, 497-498, 521-523, 526, 528, 537-540, 542, 567, 574, 580, 583, 586-587, 616-617, 632, 652-654, 701, 709-710, 714, 716, 721-724, 726-728, 731, 733, 747
|| Tested/Total Lines:
|| src/callbacks.rs: 34/42
|| src/config.rs: 3/3
|| src/connection.rs: 43/47
|| src/context.rs: 137/219
|| src/ffi_util.rs: 9/16
|| src/lib.rs: 19/31
|| src/listener.rs: 2/10
|| src/macros.rs: 51/75
|| src/message/command.rs: 169/231
|| src/message/mod.rs: 42/51
|| src/message/reply.rs: 96/181
|| src/proto_util.rs: 1/15
|| src/session.rs: 226/316
|| 
67.26% coverage, 832/1237 lines covered

[Nate-Wilkins]

@xd009642 sure thing. This is the repository I'm trying to cover - mallardscript.

[xd009642]

Well I tried it with cargo-llvm-cov and it gets the same results as me - which is grimly promising in the sense it means I'm doing nothing stupid my end and the issue is a trickier one. A bit annoying as it means this won't be an easy fix

[Nate-Wilkins]

Usually I'd suggest to help but I don't know much about code coverage mechanisms...

Still, no worries though. I know you maintain this in your spare time so even just informing me of the gap helps a lot.

Thanks again!

[ChristopherRabotin]

Hi there,

I have a decently large project (55 unit tests and ~130 integration tests, several taking over a minute to run). I'm trying to switch from llvm-cov to tarpaulin because llvm-cov times out. I had a similar issue with tarpaulin, but this thread has helped (using --engine llvm), I think.

However, I'm encountering a compilation error in the Github action: https://github.com/nyx-space/nyx/actions/runs/4121534794/jobs/7117577794 . I'm not getting that issue when I run locally though.

Any idea if this is due to how the Github action is set up? Or something else entirely ?

Thanks

[glts]

@xd009642 Thank you for reproducing the original issue. Indeed it works with --engine llvm! Feel free to close unless you want to use this issue to track similar ones.

[xd009642]

@ChristopherRabotin I believe it's related to this issue #1213 I realised other day llvm engine doesn't seem to work in docker - seems some llvm component isn't provided by the compiler for some reason. I'd recommend using cargo-binstall and installing tarpaulin like that and running the same tarpaulin command