Specs2 runner that handles Specs (test classes) and Examples (methods)

[hmemcpy]

This allows running a single specs2 test using —test-filter.
Fixes #151

[bazel-io]

Can one of the admins verify this patch?

[googlebot]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If your company signed a CLA, they designated a Point of Contact who decides which employees are authorized to participate. You may need to contact the Point of Contact for your company and ask to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the project maintainer to go/cla#troubleshoot.
• In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again.

[hmemcpy]

I signed it!

[googlebot]

CLAs look good, thanks!

[hmemcpy]

WIP, because I'm still adding a few tests. Also this is an early work that only supports the actual test run.
Location and icon providers will follow in a separate PR :)

[hmemcpy]

Ok, this wasn't trivial, but it solves the problem.
This code generates a --test-filter for specs2 in a way that rule_scala understands.
i.e. for a spec:

package com.example

class MyTest extends SpecificationWithJUnit {
  "Something" should {
     "do something" in {
         ...
     }
  }
}

when clicked on the "do something" element, will produce a test run configuration with the following filter:
--test-filter=com.example.MyTest#Something should::do something, which is the fully-qualified regex pattern that Bazel runs.

@ittaiz

[chaoren]

Did you forget to add this class?
It doesn't currently exist:
https://github.com/bazelbuild/intellij/tree/master/scala/src/com/google/idea/blaze/scala/run

[hmemcpy]

Oops, you're right, but it's not ready yet! I removed it for now...

[chaoren]

Return if null?

[hmemcpy]

Done!

[chaoren]

Here too, return if null.

[hmemcpy]

Also done!

[chaoren]

If I create a failing test at Foo#bar::baz, Foo#bar:: and Foo# succeeds because they're running an empty suite.

This is a problem in the test runner change, not here.

I think bazel expects the test_filter to be a regex. So you can probably just enumerate all of the test cases and match against the filter string.

[hmemcpy]

@chaoren Thank you so much for the comments!
I'll look into the empty suite issue, but the main difference between how specs2 and JUnit implement their JUnit runner, is that JUnit exposes all the "child" elements, so you can do the filtering yourself (e.g. https://github.com/hmemcpy/rules_scala/blob/ad25c9001685f340a1ffb17b0e3d7475ab32ad29/src/java/io/bazel/rulesscala/test_discovery/FilteredRunnerBuilder.scala#L34-L39)

But with specs2 you have to pass an additional filter to the specs2 environment, and it does the filtering internally:
https://github.com/hmemcpy/rules_scala/blob/ad25c9001685f340a1ffb17b0e3d7475ab32ad29/src/java/io/bazel/rulesscala/specs2/Specs2RunnerBuilder.scala#L47

I'll look if I missed handling some cases! 👍

[chaoren]

JUnit exposes all the "child" elements, so you can do the filtering yourself

Yes, I wrote that based on the Java test filter.

with specs2 you have to pass an additional filter to the specs2 environment, and it does the filtering internally:

Can specs2 do the same? Just list every test expression in the form of Class#Test::expression and apply the regex filter to it.

[chaoren]

This converts foo should::bar to just bar right?
I think we want to keep the whole line except replace :: with a space, since only the second part doesn't make sense when read as a sentence.

E.g., this function should return this value instead of just return this value.

[hmemcpy]

There was some issue that the name was displayed as ::bar, for some reason. I haven't investigated yet if this was a problem with the xml. This was a quick fix to strip the :: character from the test name. They are still show up under "should", see image:

before this change, the name would show as ::print hello.

[chaoren]

The specs2 test.xml seems to generate incorrect test suites right now:

com.example.TestClass#foo should::do bar
Produces:

• test suite TestClass (should be fully qualified com.example.TestClass)
• test suite foo should (should be fully qualified com.example.TestClass#foo should or something else instead of #)
• test case foo should::do bar with class name com.example.TestClass (this one seems correct)

As with scala junit test's test.xml it also generates a null test suite at the start. Though it doesn't seem to be a big deal.
This prevents the gutter icon for the test class from displaying the correct test state.

[chaoren]

I'm not terribly familiar with specs2 tests, but is it always of the form:

"foo" should {
  "do bar" in {
    blah
  }
}

can the should be anything else? Is it always two layers of infix expressions (should then in)?

[chaoren]

This name doesn't include the "should" expression.
I'm doing this to get it.
Not sure if there's a cleaner way provided by the scala plugin.

      String testName = pair._2();
      if (testName == null) {
        return null;
      }
      ScInfixExpr testSuite = PsiTreeUtil.getParentOfType(testCase, ScInfixExpr.class);
      if (testSuite != null) {
        String should = testSuite.operation().refName();
        if (should.equals("should")) {
          testName =
              TestConfigurationUtil.getStaticTestName(testSuite.lOp(), false)
                  + should
                  + SmRunnerUtils.TEST_NAME_PARTS_SPLITTER
                  + testName;
        }
      }
      url = "java:test://" + testClass.getQualifiedName() + "." + testName;

[hmemcpy]

Yeah, I ended up doing something similar (found it in the scala plugin as well), see:
https://github.com/hmemcpy/intellij/blob/88f4debcf49bd5f4e9f50d0605f12468ef9a4036/scala/src/com/google/idea/blaze/scala/run/producers/BlazeSpecs2ConfigurationProducer.java#L135-L148

Maybe I could expose this as a utility method, seems useful.

To your question about the infix order, from what we use at Wix is mostly "should" block followed by an "in", but not always. Also valid are >> and, if you're using the immutable specs2 syntax, a ! operator too. The call to TestNodeProvider.isSpecs2Expr(testCase) few lines above checks all that - it's a bit of code in the scala plugin that validates those infix expression order:
https://github.com/JetBrains/intellij-scala/blob/591723c14f1598b97eff720b20b0422f87795679/scala/scala-impl/src/org/jetbrains/plugins/scala/testingSupport/test/structureView/TestNodeProvider.scala#L335

[chaoren]

Another problem on the bazel side:

If you have multiple test scopes with tests of the same name, e.g.,

TestClass
"foo" should "do blah"
"bar" should "do blah"
"baz" should "do blah"

running any of them (e.g., "--filter=TestClass#foo::do blah", "--filter=TestClass#bar::do blah", "--filter=TestClass#baz::do blah"), will only run the first one (i.e., "foo" should "do blah").

I'm assuming because you only match the last part of the full name.

[chaoren]

I'll move the bugs for rules_scala to the other repository so they don't get lost.

[hmemcpy]

Hey, sorry for replying just now, I've been away last week.
Thank you so much for the comments - you're right, there are probably corner cases that are not handled yet. My goal with this PR was to introduce something minimal that would greatly help our teams switch to Bazel - specifically, the ability to run individual specs from IntelliJ (in the way we currently have them specified, like the example above).

I'm sure there are things that aren't handled yet, and I will address them in subsequent PRs.

[hmemcpy]

If you have multiple test scopes with tests of the same name, e.g.,

I've encountered this, but I also asked Eric Torreborre on the gitter channel, if there was a way to filter by the "should" clauses - he said there wasn't currently.

[hmemcpy]

@chaoren hey, I wanted to let you know that I found a bug which caused it to run an empty suite (I was passing an empty string to the specs2 selection filter, instead of a None).
No additional changes were needed in the plugin side.

Is there anything else outstanding in this PR I should fix? Otherwise, it would really help if we could merge this :)

[chaoren]

We can't merge directly from github, so I'll commit the change internally and it'll be synced to github in a week or two.

[ittaiz]

Thanks! Brendan mentioned a plugin release scheduled to land in next Friday (December 1st?). Any idea if this will make it in to that same release? It could really unblock some people here.

…

On Tue, Nov 21, 2017 at 8:18 PM Chaoren Lin ***@***.***> wrote: We can't merge directly from github, so I'll commit the change internally and it'll be synced to github in a week or two. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#155 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIFwzRU8WGkHdgwMMOa0aKC05ngYMuks5s4xPjgaJpZM4QP1Vd> .

[chaoren]

Unfortunately, this change won't make it into next Friday's release.