test inconsistent No test targets were found, yet testing was requested

[nmattia]

Description of the problem:

When package package has no test targets, bazel test //package/... sometimes succeeds, sometimes doesn't.

The former case:

> bazel test //package/...
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action

The latter:

2019-01-29T15:15:58.9218544Z INFO: Build completed successfully, 3 total actions
2019-01-29T15:15:58.9227697Z ERROR: No test targets were found, yet testing was requested
2019-01-29T15:15:58.9292161Z INFO: Build completed successfully, 3 total actions
2019-01-29T15:15:58.9524112Z ##[error]Bash exited with code '4'.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I've only seen the ERROR: ... case occur on the rules_haskell CI, with this config:

jobs:
- job: Windows
  pool:
    vmImage: 'vs2017-win2016'
  steps:
  - bash: |
      set -e
      curl -LO https://github.com/bazelbuild/bazel/releases/download/0.21.0/bazel-0.21.0-windows-x86_64.exe
      mv bazel-*.exe bazel.exe
      mkdir /c/bazel
      mv bazel.exe /c/bazel
    displayName: 'Install Bazel'
  - bash: |
      set -e
      export MSYS2_ARG_CONV_EXCL="*"
      ...
      # !!!
      # Replacing `build` with `test` fails the job
      /c/bazel/bazel.exe build --config windows "//tests/data/..."
      ...

    displayName: 'Run Bazel'

What operating system are you running Bazel on?

Windows with vmImage: 'vs2017-win2016'

I'm unable to reproduce the issue on other Windows machines.

What's the output of bazel info release?

release 0.21.0

[laszlocsomor]

I can't make any guesses from the error message. Could you please add --verbose_failures to the flags? That'll show the failing command line.

[nmattia]

CC @gdeest

[nmattia]

@laszlocsomor here's the output:

command]"C:\Program Files\Git\bin\bash.exe" --noprofile --norc -c pwd
/d/a/_temp
[command]"C:\Program Files\Git\bin\bash.exe" --noprofile --norc /d/a/_temp/beaaf7d3-033b-4ab7-9a54-83f2a1e24839.sh
WARNING: The following rc files are no longer being read, please transfer their contents or import their path into one of the standard rc files:
d:\a\1\s/.bazelrc
INFO: Invocation ID: 53673a2a-11dd-4ae1-82c8-99fadd191cf4
Loading: 
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Analyzing: 2 targets (1 packages loaded, 0 targets configured)
Analyzing: 2 targets (2 packages loaded, 15 targets configured)
Analyzing: 2 targets (2 packages loaded, 15 targets configured)
Analyzing: 2 targets (2 packages loaded, 15 targets configured)
Analyzing: 2 targets (2 packages loaded, 15 targets configured)
Analyzing: 2 targets (9 packages loaded, 15 targets configured)
Analyzing: 2 targets (9 packages loaded, 15 targets configured)
Analyzing: 2 targets (19 packages loaded, 103 targets configured)
Analyzing: 2 targets (19 packages loaded, 103 targets configured)
Analyzing: 2 targets (19 packages loaded, 103 targets configured)
Analyzing: 2 targets (19 packages loaded, 103 targets configured)
Analyzing: 2 targets (19 packages loaded, 103 targets configured)
INFO: Analysed 2 targets (21 packages loaded, 138 targets configured).
INFO: Found 2 targets and 0 test targets...
[0 / 2] no action
SUBCOMMAND: # //tests/data:ourclibrary [action 'Compiling tests/data/ourclibrary.c']
cd C:/users/vssadministrator/_bazel_vssadministrator/w3d6ug6o/execroot/io_tweag_rules_haskell
  SET PATH=C:\Program Files\Git\usr\bin;C:\Program Files\Git\bin
    SET PWD=/proc/self/cwd
  external/io_tweag_rules_haskell_ghc_windows_amd64/mingw/bin/gcc -MD -MF bazel-out/x64_windows-fastbuild/bin/tests/data/_objs/ourclibrary/ourclibrary.d -frandom-seed=bazel-out/x64_windows-fastbuild/bin/tests/data/_objs/ourclibrary/ourclibrary.o -iquote . -iquote bazel-out/x64_windows-fastbuild/genfiles -iquote bazel-out/x64_windows-fastbuild/bin -c tests/data/ourclibrary.c -o bazel-out/x64_windows-fastbuild/bin/tests/data/_objs/ourclibrary/ourclibrary.o
SUBCOMMAND: # //tests/data:ourclibrary-static [action 'Compiling tests/data/ourclibrary.c']
cd C:/users/vssadministrator/_bazel_vssadministrator/w3d6ug6o/execroot/io_tweag_rules_haskell
  SET PATH=C:\Program Files\Git\usr\bin;C:\Program Files\Git\bin
    SET PWD=/proc/self/cwd
  external/io_tweag_rules_haskell_ghc_windows_amd64/mingw/bin/gcc -MD -MF bazel-out/x64_windows-fastbuild/bin/tests/data/_objs/ourclibrary-static/ourclibrary.d -frandom-seed=bazel-out/x64_windows-fastbuild/bin/tests/data/_objs/ourclibrary-static/ourclibrary.o -iquote . -iquote bazel-out/x64_windows-fastbuild/genfiles -iquote bazel-out/x64_windows-fastbuild/bin -c tests/data/ourclibrary.c -o bazel-out/x64_windows-fastbuild/bin/tests/data/_objs/ourclibrary-static/ourclibrary.o
SUBCOMMAND: # //tests/data:ourclibrary-static [action 'Linking tests/data/libourclibrary-static.a']
cd C:/users/vssadministrator/_bazel_vssadministrator/w3d6ug6o/execroot/io_tweag_rules_haskell
  SET PATH=C:\Program Files\Git\usr\bin;C:\Program Files\Git\bin
    SET PWD=/proc/self/cwd
  external/io_tweag_rules_haskell_ghc_windows_amd64/mingw/bin/ar rcsD bazel-out/x64_windows-fastbuild/bin/tests/data/libourclibrary-static.a bazel-out/x64_windows-fastbuild/bin/tests/data/_objs/ourclibrary-static/ourclibrary.o
SUBCOMMAND: # //tests/data:ourclibrary [action 'Linking tests/data/libourclibrary.a']
cd C:/users/vssadministrator/_bazel_vssadministrator/w3d6ug6o/execroot/io_tweag_rules_haskell
  SET PATH=C:\Program Files\Git\usr\bin;C:\Program Files\Git\bin
    SET PWD=/proc/self/cwd
  external/io_tweag_rules_haskell_ghc_windows_amd64/mingw/bin/ar rcsD bazel-out/x64_windows-fastbuild/bin/tests/data/libourclibrary.a bazel-out/x64_windows-fastbuild/bin/tests/data/_objs/ourclibrary/ourclibrary.o
INFO: Elapsed time: 148.361s, Critical Path: 0.35s
INFO: 4 processes: 4 local.
INFO: Build completed successfully, 5 total actions
ERROR: No test targets were found, yet testing was requested
INFO: Build completed successfully, 5 total actions

from this build: https://dev.azure.com/tweag/rules_haskell/_build/results?buildId=839
this commit: tweag/rules_haskell@c9964c4
from running this command:

  /c/bazel/bazel.exe test --config windows "//tests/data/..."

where //tests/data/... doesn't have test targets. The command succeeds on other machines.

[annaken]

I'm seeing a similar thing - the same command (bazel test //foo/...) run in the same docker Debian container on different machines gives different results.

The working example output says:
INFO: Build completed successfully, 36 total actions INFO: Build completed successfully, 36 total actions

The not working example says:
INFO: Build completed successfully, 36 total actions ERROR: No test targets were found, yet testing was requested INFO: Build completed successfully, 36 total actions

[annaken]

Sorry, I was mistaken. If I force the full output with --curses no then I get the same result in both cases.

[laszlocsomor]

Thanks!
What's the exit code?
Did you notice any patterns, like is this happening spuriously or you observe anything might influence it? Does the problem go away if you retry the same bazel command again? What's the output of bazel query 'tests(//foo/...)'?

[laszlocsomor]

What's the output of bazel query 'tests(//foo/...)'?

Oh, just realized this might leak your target names. All I'm interested in is whether it returns the tests as it should.

[annaken]

I don't want to hijack this issue, so I'll keep it brief - the example in question gave me an exit code 4 every time, but I was confused about why I was getting different stdout on different machines. Turns out this was because one was essentially doing --curses no and the other --curses yes.

My tests and exit codes are being returned correctly, it was the output/curses behaviour that was confusing me.

[laszlocsomor]

@annaken : Thanks! So you are not affected by this bug, correct?

[annaken]

No, unless the bug is the same confusion over output / suppressed ERROR messages with implicit --curses yes.

[laszlocsomor]

Got it, thanks!

[meisterT]

Is this still an issue with current Bazel?

[davido]

What we are seeing on Bazel@HEAD is:

  $  bazel test Documentation/...
INFO: Invocation ID: bb4c98c4-60fe-4a8e-9462-1914e4e6c536
DEBUG: /home/davido/.cache/bazel/_bazel_davido/27a001f4182820ef315d8d2d4f1edafe/external/bazel_toolchains/rules/version_check.bzl:45:14: 
Current running Bazel is not a release version and one was not defined explicitly in rbe_autoconfig target. Falling back to '0.25.2'
INFO: Analyzed 0 targets (1 packages loaded, 0 targets configured).
INFO: Found 0 test targets...
INFO: Elapsed time: 1.042s, Critical Path: 0.24s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action

But then the return code is: 4. The Bazel code is here: src/main/java/com/google/devtools/build/lib/runtime/commands/TestCommand.java:157

   // TODO(bazel-team): the check above shadows NO_TESTS_FOUND, but switching the conditions breaks
    // more tests
    if (testTargets.isEmpty()) {
      env.getReporter().handle(Event.error(
          null, "No test targets were found, yet testing was requested"));

      DetailedExitCode detailedExitCode =
          buildResult.getSuccess()
              ? DetailedExitCode.justExitCode(ExitCode.NO_TESTS_FOUND)
              : buildResult.getDetailedExitCode();
      env.getEventBus()
          .post(
              new NoTestsFound(
                  detailedExitCode.getExitCode(),
                  buildResult.getStopTime(),
                  buildResult.getWasSuspended()));
      return BlazeCommandResult.detailedExitCode(detailedExitCode);
    }

So that 4 is unconditionally returned, and we would like to have an option, to change that default behavior, something like: --treat_no_tests_as_success_return_code. That way we would not need to add a boilerplate after each and every test command invocation:

  echo 'Running Documentation tests...'
  set +e
  bazelisk test $BAZEL_OPTS Documentation/...
  TEST_RES=$?
  set -e
  if [ ! $TEST_RES -eq 4 ] && [ ! $TEST_RES -eq 0 ]
  then
    echo 'Documentation tests failed'
    exit 1
  fi

See discussion in: [1]

[1]. https://gerrit-review.googlesource.com/c/gerrit-ci-scripts/+/267953

[dmivankov]

--treat_no_tests_as_success_return_code would be great to have, usecase is using bazel query to select test targets, for example to

• pick --jobs N differently for different test sets (based on size for example)
• choose running bazel coverage vs bazel test depending on some query

Filtering all test by tags isn't always expressive enough.

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[github-actions]

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please post @bazelbuild/triage in a comment here and we'll take a look. Thanks!