Filter tests programatically

[nicojs]

Is your feature request related to a problem or a nice-to-have?? Please describe.

For the JavaScript mutation testing framework Stryker we want to be able to filter out individual tests programmatically. The it.only won't work in our case, we need to do it programmatically.

Describe the solution you'd like

Something like Jasmine's specFilter would be awesome.

const Mocha = require('mocha');

function testFilter = function(test) {
  // Use custom logic to determine wether or not to run the test
  return Math.random() < .5;
}

const mocha = new Mocha({ bail: true, testFilter });
mocha.run(() => console.log('done'))

Describe alternatives you've considered

Right now, we're using a horrible hack:

const Mocha = require('mocha');
const realAddTest = Mocha.Suite.prototype.addTest;
Mocha .Suite.prototype.addTest = function() {
    if (Math.random() < .5) {
        realAddTest.apply(this, arguments);
    }
};

It works but has obvious drawbacks:

• It uses a private api which can break any release
• It will not allow us to filter other tests the next run (whenever Mocha can't run tests twice programmatically #2783 is implemented)

Additional context

I want to revisit #2605. Back in the day, @boneskull said this wouldn't be supported since there was no way to configure a testFilter function from the command line and there was no config file support.

no, what I'm concerned about is running with the CLI mainly. it's trivial to use this sort of feature programmatically--there's just currently no reasonable way to run mocha on the command line and hand it a callback to "grep". see bin/_mocha.

Since we now have .mocharc.js support the CLI would also able to support the testFilter:

module.exports = {
  testFilter: (test) => test.fullTitle() === 'it should only run this test';
}

I'd be happy to see how I could help with this!

[juergba]

@nicojs thank you for this issue, it sounds interesting to me.

I would like to understand better the big picture and will need some time to clarify:

• grep feature: all tests are linked into the runner, but at runtime only the matching ones are executed?
• your feature: you only load the matching test, unmatching ones are ignored?
• only feature: could we use the static it.only?
• can we use your feature for re-running failed tests? Another feature requested by some users.
• CLI / programmatic / browser?

It will not allow us to filter other tests the next run (whenever #2783 is implemented)

I don't understand this one. We have a watch feature which re-runs tests (except ESModules).

[juergba]

@nicojs I don't really like Mocha's grep feature, mainly because first it adds all tests to the runner, and then runs only a selection of filtered matching tests.

You would like to do the same, right? Load all unfiltered tests to the runner and re-use the same runner instance applying different filters.

It will not allow us to filter other tests the next run (whenever #2783 is implemented)

[nicojs]

• grep feature: all tests are linked into the runner, but at runtime only the matching ones are executed?
• your feature: you only load the matching test, unmatching ones are ignored?

No, it would basically work the same as grep, but would allow us to determine whether to run the test or not programmatically

• only feature: could we use the static it.only?

No. During mutation testing, we can't instrument/change the tests. They are what they are.

• can we use your feature for re-running failed tests? Another feature requested by some users.

Sounds good. But that does seem #2783 also needs to be implemented.

• CLI / programmatic / browser?

It should run programmatically and should also work in the browser.

It will not allow us to filter other tests the next run (whenever #2783 is implemented)

I don't understand this one. We have a watch feature which re-runs tests (except ESModules).

This isn't useful to us since we might not change the source files at all. Unless we would be able to trick the watch feature that files changed while they didn't? That doesn't sound like a good plan.

You would like to do the same, right? Load all unfiltered tests to the runner and re-use the same runner instance applying different filters.

Yes, exactly.

Let me clarify a bit better what the use case is, seeing as I think that would make it a lot clearer.

During Mutation testing, you are testing the effectiveness of your tests by automatically inserting bugs (or mutants) into your production source code to see if they are spotted by the tests (i.e. result in a failing test).

Take this example:

// add.js
function add(a, b) {
  return a + b;
}

// add.spec.js
it('should add two numbers', () => {
  require('chai').expect(add(4, 0)).eq(4); // great test! ... well, maybe not so much
});

it('some other test', () => {
  require('chai').expect(subtract(4, 0)).eq(4); 
});

A mutation testing framework might instrument add.js to look like this:

// add.js
function add(a, b) {
  if(global.activeMutation === 0) {
    return a - b;
  } else {
    global.mutantCoverage[0] ++;
    return a + b;
  }
}

It will add something we call a "mutation switch". You can turn one mutation on/off at a time. So even though the source files don't change, the test result might very well, since the mutation testing framework decides what to turn on/off. (which is why #2783 is important)

This is where the programmatic test filtering comes in. We want to be able to choose which tests to run in order to not run too many. In this example, if we want to test mutant 0, we don't want to run the second test (some other test), since it is testing something else.

I hope this clarifies it for you. Sorry it took me so long to respond.

[nicojs]

We no longer need this functionality, because Stryker 4 can work with the current --grep regex. Closing this issue. Thanks for taking a look and helping us.