BeforeAll/AfterAll?

[modocache]

First of all, thanks so much for this library. It really makes testing in Go a joy.

RSpec includes before(:all) and after(:all) nodes. These nodes are executed once at the beginning or end of an example group, respectively:

require 'rspec'

RSpec.describe 'before and after all nodes' do
  before(:all) { puts "\nbefore(:all)" }
  after(:all) { puts "\nafter(:all)" }
  it 'is an example' { }
  it 'is another example' { }
end

When run, rspec outputs:

before(:all)
..
after(:all)

I was surprised that Ginkgo does not include BeforeAll or AfterAll functions. Is this a conscious omission? Would you consider a pull request which adds these functions?

[onsi]

hey @modocache,

The lack of BeforeAll and AfterAll is intentional, and I'd be curious to see the usecases you have in mind that necessitate it.

*All variants incentivize sharing state across Its -- this is generally not a good idea.

Because Its do not share state Ginkgo can treat them as completely separable uncorrelated entities. This allows Ginkgo to --randomizeAllSpecs - an important feature for sussing out test pollution. It's also the core principle behind Ginkgo's parallelization strategy. The *All variants break this separability by implicitly grouping individual Its via the shared state set up by the BeforeAll and torn down by the AfterAll.

Does this make sense? I'd love to hear your thoughts. FWIW I've avoided RSpec's before(:all) and after(:all) so I don't really miss them.

[modocache]

That makes a lot of sense, thanks for the explanation!

I came across my specific use case when writing this. The post describes testing a HTTP handler. I believe creating the handler is fairly expensive, especially considering it opens a connection to a database, so I was hoping to only do it once. But I need a reference to the handler, so I can't easily move the initialization into a BeforeSuite.

That being said, I've only just started looking into web development with Go, so I'm certain there's a better way to accomplish what I'm doing. I was mainly curious as to why you'd leave out BeforeAll--now I know! Very enlightening, I'll keep it in mind next time I write a testing tool. 👍

[rnapier]

I know you got your answer, but just to add a little: Always try it before
you assume it'll be slow. It's often amazing just how fast it is, and if
you keep things separated correctly, you can run a lot of things in
parallel. Whenever possible, try to move the service local to your machine
(i.e. it's best to run a local database just for your tests rather than
talking to a server if you can set it up). You may even want to mock your
DB entirely. I have some test cases that kick off a small local server to
talk to. I'm able to run multiple instances of my test server by having
them listen on different ports. This keeps my test cases very independent.

If you must share a handle, you really should be able to initialize it in
BeforeSuite. Remember, package variables are shared throughout the package,
so you can create a global variable in one part of your test suite and use
it in another. I'd avoid this if possible, but if it is a stateless
connection, then it shouldn't actually cause trouble to share it. If it's
statefull, then you really shouldn't share it, even if you can figure out
how to.

I'll note that sometimes I'm torn about how many expectations to to put
into an It() just because every It() starts over from scratch. So, for
instance, it does feel crazy at times to run the whole test case once to
test "no error" and once to test the result code is good and again to test
that the returned data is good. But even when I break it down to that kind
of possibly crazy granularity, I'm amazed how fast it is anyway. So I've
gotten over it a bit, and focused on making the tests really clear and I'll
worry about making them fast if they're ever actually slow.

Good luck!

[onsi]

Glad that was helpful @modocache -- and thanks for the sweet blog posts mentioning Ginkgo!

Also, thanks for chiming in @rnapier - I agree with your points. I also prefer smaller Its when possible. The primary exception being around integration-style suites that spin up several external processes and get them to play together -- these tend to be slow and breaking out separate Its when the assertions can be chained into one It becomes counterproductive.

[mml]

I know this is an ancient issue, but I'll mention another way this is an issue. In kubernetes, we have a rather large end-to-end test suite. There is a single suite for the entire project, and so the nested Describe/Context blocks often have very specific needs.

As this is an e2e setup, it's often the case that we need to create certain API objects and wait for them to stabilize in the running server. This is a high-latency operation. Currently we do this in BeforeEach blocks, which works but it strongly encourages test writers to build gigantic It() blocks because of the state teardown/construction. In effect, we have the same amount of test assertions "sharing state" as we would with BeforeAll, but the tests are large, harder to read, and they fail with very little granularity.

BeforeAll/AfterAll or equivalently nested suites would be a useful abstraction for this case.

[and-hom]

I need BeforeAll/AfterAll too. I created integration test for my application. It prepares environment, starts app and performs tests. So I'd like to test my app with different config files and different environment variables. And I have more then one test per each configuration.
Now I have to create a package per each test configuration and one more package for common test utils. It's a workaround and looks ugly.
It would be great to create own setUp/tearDown for Context

[onsi]

@and-hom can you point me at a concrete or specific example?

PSA:

If you want BeforeAll or AfterAll you can always add a guard to a BeforeEach and AfterEach block to ensure it only runs once.

This will not work with paralellized tests or --randomizeAllSpecs. This is one reason why I do not intend to add BeforeAll or AfterAll.

[zeekay]

I think this would be a nice addition. It helps discourage massive It blocks, which are a natural response to BeforeEach blocks with high-latency setup work (as mentioned by @mml). As per @onsi's suggestion above we implemented our own BeforeAll helper like so:

func BeforeAll(fn func()) {
    first := true
    Before(func() {
        if first {
            fn()
            first = false
        }
    })
}

Which works well with our parallelized (and randomized) test setup.

This has been a boon in dealing with some particularly unreliable testing infrastructure (rapid iteration of It blocks causing test server to explode). Judicious use is of course advised, BeforeEach is what you want in most cases...

[b0o]

@onsi

If you want BeforeAll or AfterAll you can always add a guard to a BeforeEach and AfterEach block to ensure it only runs once.

I can see how this makes BeforeAll possible, but how does it help with emulating AfterAll? Adding a guard to an AfterEach would just allow you to run it once after the first It, right?

I really need an AfterAll to reconstruct some high-cost objects.

I've been racking my brain trying to come up with various workarounds, but since the AfterEach will be the final thing to run within a Describe or Context, I don't know how to know that it's really the last one (or if it's even possible to know).

[akutz]

Hi,

I also would like to see BeforeAll implemented for the same reason. Sometimes standing up an environment is expensive.

[glyoko]

+1

[tzvatot]

I know this is an ancient issue, but I'll mention another way this is an issue. In kubernetes, we have a rather large end-to-end test suite. There is a single suite for the entire project, and so the nested Describe/Context blocks often have very specific needs.

As this is an e2e setup, it's often the case that we need to create certain API objects and wait for them to stabilize in the running server. This is a high-latency operation. Currently we do this in BeforeEach blocks, which works but it strongly encourages test writers to build gigantic It() blocks because of the state teardown/construction. In effect, we have the same amount of test assertions "sharing state" as we would with BeforeAll, but the tests are large, harder to read, and they fail with very little granularity.

BeforeAll/AfterAll or equivalently nested suites would be a useful abstraction for this case.

I'm facing the same issue: working with openshift (kubernetes based) I have a test suite devided to several contexts and specs. Some tests needs a setup before their execution. The setup takes a long time, but it's the same setup for all It in the context, which the outcome is a state which the test requires in order to run. Running the setup before each test cause the test duration to exceed the timeout. Besides it consumes a lot of redundant resources.
Having a beforeAll and afterAll would allow me to setup the state, pass that to each It as a context variable and tear it down on at the end of all of them.

[williammartin]

@tzvatot Thanks for your feedback! Is this code somewhere I can see?

[tzvatot]

@tzvatot Thanks for your feedback! Is this code somewhere I can see?

Unfortunately no. But we might open it up sometime in the future.

[wookasz]

+1 on the BeforeAll and AfterAll. We have some integration tests which involve indexing to elasticsearch. It can take over a second for the document to be updated in ES in the test environment. We want to be able to trigger the document update and test several assertions. Multiple assertions in a single It are not ideal because it's unclear which assertion failed and when a single assertion fails, the remaining ones are not executed.

[arxeiss]

Doesn't BeforeSuite() and AfterSuite() solving exactly this problem? http://onsi.github.io/ginkgo/#global-setup-and-teardown-beforesuite-and-aftersuite

Global Setup and Teardown: BeforeSuite and AfterSuite

Sometimes you want to run some set up code once before the entire test suite and some clean up code once after the entire test suite. For example, perhaps you need to spin up and tear down an external database.

Ginkgo provides BeforeSuite and AfterSuite to accomplish this. You typically define these at the top-level in the bootstrap file. For example, say you need to set up an external database:

[tzvatot]

@arxeiss no, it does not solve the problem. As mentioned in the documentation, the BeforeSuite and AfterSuite are global.

For example, assume I have a single test suite called "regression". This test suite contains several test cases. Few of the tests share the same exact setup, which is costly both in time and resources. So instead of forcing each of those tests to do it's own setup, we like to run that setup once, to create some basic resources that will be used by all those tests, and once they finish, we can remove those resources.

Of course, you may argue that those should be in a separated Suite, but since we have several such "groups" of tests that should share the setup and teardown in the "regression" suite, it means that every real life suite will actually be a group of suites (which opens another question about missing BeforeEachSuite and AfterEachSuite).

[liuxh0]

Generally, it does make sense without BeforeAll and AfterAll.
However, while I was writing integration tests recently, I do need them to reduce calling external services, which are time-consuming.

-> Therefore, +1 for BeforeAll / AfterAll. Also would like to know the best practices for integration tests using Ginkgo.

[blgm]

Ginkgo 2.0 (see #711) will hopefully add some additional controls to help with this.

[onsi]

Am happy to share that over 7 years later, the ver2 branch now includes support for Ordered contexts which can include BeforeAll and AfterAll.

[hamadise]

We do use Ginkgo to develop a framework to test applications of Openshift/Kubernetes and having BeforeAll is really sweet option. Otherwise, we're forced to do that ourselves (not in a clean way)
In Openshift, we do need to setup the whole application with all the sessions to different Pods/Nodes.
Then we do run different suites and differents tests. "It" blocks are fully separated and can run in any given order. We can add the whole logic of setting up the app and sessions inside beforeEach. But it takes a significant amount of time.
Reading the comments, people such as (@tzvatot @mml ) are also facing similar issue in k8s/openshift.
our code is open and anyone is welcome to check it and see how ginkgo is really useful https://github.com/test-network-function/test-network-function
We're waiting for official release of Ginkgo v2 :)

[tzvatot]

@onsi how can I find the PR for this? what version will include it?

[onsi]

hey - this shipped in 2.0 and is documented here.

[tzvatot]

@onsi it seems that the BeforeAll/AfterAll is strongly tight with Ordered context. This cause the following issue:
In case of using a DescribeTable with multiple entries, the first entry fails the rest of the entries, because it's Ordered. Removing the Ordered is not an option as it prevents using BeforeAll/AfterAll.
We need an ability to have BeforeAll/AfterAll, but still be able to have DescribeTable run through all of the entries, without skipping them as soon as the first fails.

[onsi]

hey @tzvatot do you mind opening a new issue so we can discuss it there? i'd like to learn more about what you're doing in the table test to figure out what potential solutions for you are.

[tzvatot]

hey @tzvatot do you mind opening a new issue so we can discuss it there? i'd like to learn more about what you're doing in the table test to figure out what potential solutions for you are.

#1103 submitted

[onsi]

cross-posting here since this is the issue that folks thinking about BeforeAll/AfterAll usually land on:

2.7.0 introduces the ContinueOnFailure decorator which resolves #1103 and allows specs in ordered containers to continue running if an earlier spec fails.