Run tagged ScalaTest tests also runs all others tests (500USD Bounty)

[Readon]

---

From the maintainer Li Haoyi: I'm putting a 500USD bounty on this issue, payable by bank transfer on a merged PR fixing this.

---

When running testOnly with "-- -n [tag]" by mill it would run all tests including the tagged test.
However, this option is used normally only run the tagged test without more, just as sbt doing.

I have create a minimum example at my fork.
Just run the run.sh, the results would be as below in 0.11.11.

Cmd: sbt 'testOnly -- -n tagged'
works good, only run the tagged test.

tagged run
[info] MyAppSpec:
[info] - Addition test with tag

Cmd: sbt 'testOnly -- -l tagged'
works good, only run tests without specified tag.

untagged run
[info] MyAppSpec:
[info] - Addition should work correctly

Cmd: mill __.testOnly -- -n tagged
Wrong this command should not run tests with tag.

MyAppSpec:
untagged run              //unwanted.
- Addition should work correctly
tagged run
- Addition test with tag

Cmd: mill __.testOnly -- -l tagged
works good, only run tests without specified tag.

MyAppSpec:
untagged run
- Addition should work correctly

[Readon]

mill version 0.11.12 have also been tested, the results is the same.

[lolgab]

@Readon It's probably a scam bot, don't click the link, please.

[Readon]

@Readon It's probably a scam bot, don't click the link, please.

Thanks, fortunately I do not have account for MediaFire.

[lolgab]

@Readon The issue is really strange since the arguments should be handled by the the test runner which is not part of Mill and they seem to be passed correctly. Stranger is the fact that the -l flag works correctly.

[Readon]

I did not check the arguments inside mill. is that possible sbt have done some extra ops on this arguments?

[wb14123]

I can reproduce it in the test directory with the bundled mill script. But when I'm trying to reproduce with the current mill build:

./mill -i dist.run example/scalalib/testing/4-test-tag __.test

It failed to find any tests:

[2523/1] dist.run 
Run completed in 91 milliseconds.
Total number of tests run: 0
Suites: completed 0, aborted 0
Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0

Am I running the correct command?

[wb14123]

I created a PR that I think can fix the issue, but I cannot really test it since the testing command above doesn't work. @lihaoyi or anyone else, could you help me figure out the command to run tests with the dev build?

[lihaoyi]

@wb14123 that command is for manually exercising the example tests, which I think isn't what we want for your PR. For your PR, a unit test added in https://github.com/com-lihaoyi/mill/blob/main/scalalib/test/src/mill/scalalib/TestModuleTests.scala, with the code-under-test in a subfolder of scalalib/test/resources, which you can run via ./mill scalalib.test mill.scalalib.HelloWorldTests.nameoftest

[wb14123]

@lihaoyi Yeah agreed on unit test. But I also want to manually run the testOnly command to test the the build work with the test project. Not sure why it's not able to find the test cases.

[lihaoyi]

@wb14123 your command looks correct. What's the content of 4-test-tag/? Are your tests in ScalaModule layout of test/src/..., or in the SbtModule layout of src/test/scala/...?

[wb14123]

The content is the code mentioned in the issue at the beginning: https://github.com/Readon/mill/tree/main/example/scalalib/testing/4-test-tag . Running ../../../../mill __.test under that directory works.

[wb14123]

Anyway, added tests in the PR. The tests failed without my change and succeed with the change. So the fix should be good.