Introduce beforeAll/afterAll module hooks?

[trentmwillis]

I'd like to propose adding in beforeAll/afterAll hooks to test modules. Similar to the functions of the same name in Jasmine or before/after in Mocha. It would be convenient to have a way to access the testing context at the start and end of a module on a case-by-case basis instead of just the sweeping moduleDone and moduleStart callbacks.

I apologize if this has been discussed before, couldn't seem to find any background on the matter.

cc @nathanhammond

[gibson042]

If I correctly understand what you want, this is already possible: http://jsfiddle.net/8zfLqk0r/

QUnit.module( "module", function( obj ) {
    obj.beforeEach( function() {
        this.setup = true;
    } );
    obj.afterEach( function() {
        console.log( "teardown" );
    } );
    QUnit.test( "test", function( assert ) {
        assert.strictEqual( this.setup, true,
        "beforeEach ran" );
    } );
} );

[trentmwillis]

@gibson042 not quite, guess I should've explain myself a bit more clearly.

What I'm looking for are hooks to run at the beginning of a module and at the end of the module, rather than at the beginning and end of each test. Similar to what is done in Jasmine.

[cjmaynar]

This would be pretty useful indeed. I'm looking for something like Python UnitTest's setUpClass/tearDownClass methods that will run once before any tests are run, and then once after all the tests are complete

[leobalter]

@trentmwillis that would be an interesting feature. If it was discussed before - certainly mentioned at least - I believe we should try it.

I would name them before/after module hooks, removing the All, it's easier to match beforeEach and afterEach.

PRs are welcome!

[JamesMGreene]

It is definitely a feature that we've seen requests for before. I myself would like that capability as well.

[JamesMGreene]

P.S. We didn't implement it in the past because there was already a decent plugin available for it: qunit-once.

However, I would imagine that [old] plugin does not currently offer all the niceties that a built-in set of hooks easily could, i.e. guarantee before callbacks to run before any beforeEach callbacks, guarantee after callbacks to run after any afterEach callbacks, first-class support for async, first-class support for promises, etc

[jzaefferer]

From the Jasmine docs:

The beforeAll function is called only once before all the specs in describe are run, and the afterAll function is called after all specs finish. These functions can be used to speed up test suites with expensive setup and teardown.

However, be careful using beforeAll and afterAll! Since they are not reset between specs, it is easy to accidentally leak state between your specs so that they erroneously pass or fail.

This gotcha sounds like a good reason to not offer this feature in the first place. That said, does someone have an example for the "speed up test suites" usecase? Are there other usecases? Usecases with examples would help a lot, both to justify implementing this and given an implementation, to document this feature properly (the "examples" for both Jasmine and Mocha aren't sufficient usecases).

[trentmwillis]

Sorry for letting this sit for so long.

@jzaefferer that gotcha is definitely something to highlight if this feature does move forward.

Currently my team would like this feature for generating mock data at the beginning of a module and tearing it down at the end. While we could definitely do this with beforeEach/afterEach we get a perf benefit from not having to generate a new mock before each test (though we are also very cognizant of potentially polluting other tests and only do this when the mock has no write operations performed on it).

The more interesting use case that I am aware of is an idea in the Ember community. We've talked about moving the acceptance test model to use only one instance of an application and swap out the stateful information between tests instead of tearing down and creating an entire new application. This again has performance benefits. Here is a relevant conversation and example.

[mmun]

@trentmwillis But I don't see why that should be scoped to a module. It could be shared with all acceptance test modules. I would like to avoid test state being managed per module (preferring per test or global state when possible).

That said, I'm not opposed to the idea presented in this issue. I'd prefer that it be named something very explicit though e.g. beforeModule/afterModule even if its not DRY. It shouldn't come up a lot in practice anyways.

[trentmwillis]

@mmun that makes sense and can explore that more in the appropriate Ember channels.

As for this issue, it seems like there are multiple persons interested in this, though I can only think of one potential use case, which could be argued isn't justification for the "gotcha" that @jzaefferer mentioned above. How do we want to proceed?

[leobalter]

My concerns are only on naming it. The gotcha (leaking state to other tests) seems also as a developer's responsibility matter.

The feature makes a lot of sense on ember tests, including using QUnit for acceptance and integration tests.

[trentmwillis]

Sounds good. I'll take a stab at implementation later this week.

As for the name, should we do before/after or beforeModule/afterModule? I think either would be fine, with the former providing more parody with other JS testing frameworks.

[leobalter]

sure! and it seems they also connect well with beforeEach/afterEach semantics.

As I believe this will affect a lot of Ember tests, I would like to know if @rwjblue has any thoughts to add.

[trentmwillis]

@leobalter and others, I started implementing this, but there are two finer details I'm stuck on:

1. For nested modules, should before run once before all tests in the module stack? Or, once before each module? Similar question for after. My thinking is that it should be once for the entire stack of modules.
2. Should we assume that modifications done to this (a.k.a., testEnvironment) in the before hook propagate throughout all subsequent tests? My thinking is yes, since it reasonable to believe that any work done in before is likely setup for all tests.

[leobalter]

For nested modules, should before run once before all tests in the module stack? Or, once before each module?

TBQH, I don't know. It makes a lot of sense to stack the *Each hooks, but it seems better to not stack the modules only hooks. Someone more used to integration and functional tests would answer this better. Once again, @rwjblue might have a better perspective for this being used on ember-qunit.

Should we assume that modifications done to this (a.k.a., testEnvironment) in the before hook propagate throughout all subsequent tests?

Yes, I believe it should work similar to the other hooks and nested modules.

My thinking is yes, since it reasonable to believe that any work done in before is likely setup for all tests.

Exactly

[trentmwillis]

but it seems better to not stack the modules only hooks.

This was my thought, and turns out this is how Mocha does it as well. So if you have two modules, outer and inner, the order of execution goes: outer.before, outer.tests, inner.before, inner.tests, inner.after, outer.after. I'll follow that pattern unless someone presents a valid reason to do otherwise.