One test CLI to rule them all

[epixa]

We have a ton of different ways to test code and features in Kibana, but there's very little consistency in how you run any of the tests. I propose a single test CLI to run any or all of the individual test suites:

tasks

run all tests (default)

$ node scripts/test

run unit-like tests, no slow tests

$ node scripts/test unit --jest --mocha

test end-user interactions, a mix of selenium tests and http request testing (server)

$ node scripts/test app --selenium --server

test all api endpoints (/api)

$ node scripts/test api

test upgrade paths to the current version

$ node scripts/test upgrade --from

test different deployment scenarios (load balanced, local coordinating node, base path)

$ node scripts/test deployment --scenario

guaranteed release blockers

$ node scripts/test smoke

all tasks also accept:
--grep
--bail
--quiet
--verbose
--debug

[epixa]

cc @kjbekkelund @LeeDr @spalger

[cjcenizal]

I imagine that you can run multiple types of tests by adding them as additional arguments, e.g. node scripts/test api unit --jest, right?

[spalger]

I see a few advantages to doing as CJ suggested, but I'm concerned about the difficulty of implementing the feature versus its payoff

• unified reporting when using several configs
• allow failures within tests to not prevent the next test type, but still produce an overall exit status
• possible resource sharing between test types

If these seem like important features then we probably should find a syntax that allows specific arguments to be used for explicit test configs (maybe node scripts/test unit --bail AND api --grep “saved_objects”)

I don't expect it to be super easy and will probably require custom argv parsing since we usually only get a map and won't be able to apply configuration to individual types. I also expect CI to run these tests in independent jobs for concurrency, so it won't utilize this feature. Finally, I don't expect resource sharing to be very straight forward. I would prefer just supporting && for now.

node scripts/test api && node scripts/test unit --jest

[epixa]

++ to keeping the interface simple for now. If there's a real need that && can't provide, then we can always improve on what we have at that time.

[rashmivkulkarni]

cc @liza-mae ( this may be of interest to you)

[elasticmachine]

Pinging @elastic/kibana-operations

[tylersmalley]

This will be done with Bazel - this issue is not needed to track this anymore.