Cypress based E2E tests

[arikfr]

The first goal is to set initial infrastructure to run Cypress based tests both locally and in the CI along with a few example tests.

For the test environment we can launch secondary Docker Compose environment which will be removed post tests. The test script can be something like:

docker-compose -p cypress up -d # -p gives the env a specific name
... create tables ...
... run cypress tests ...
docker-compose -p cypress down

Regardless the Cypress tests should be configured to take the environment URL as a parameter, so we can use it with remote environments as well.

First tests to create

1. Finish Setup flow.
2. Logout.
3. Login.
4. Create data source.

[gabrieldutra]

I think a secondary Docker Compose environment is a good idea, this way we can make sure the test data won't affect other environments.

Something we will need to decide is how we are going to Seed Data for the tests...

Ordering the tests and using data among them may seem the easiest way, however it's not a good practice. Tests should also not fail when run individually.

Another possibility I thought about and noticed on their docs:

• Somehow add the possibility to use a POST /test/seed/entity route, available only on test env, we would have a test written like this:

  describe('Login Page', function () {
    beforeEach(function () {
      cy.exec('drop_tables && create_tables') // create a reset_db may be better
  
      cy.request('POST', '/test/seed/user', { email: 'test@example.com', password: 'goodpassword' })
        .its('body')
        .as('currentUser')
    })
    // tests
  })

  • This approach may generate a problem with remote environments, since they would need to be with a Test env var to enable seeding routes
  • It's good that we can easily see the valid data related to this context

There are, of course, other ways to do that, e.g: using a script to seed data instead of a route.

Cypress also gives us the possibility to stub API responses, but the problem with it is that we are not actually passing through the backend.

Well, let me know what you were thinking for this 😅

[arikfr]

Ordering the tests and using data among them may seem the easiest way, however it's not a good practice. Tests should also not fail when run individually.

That's exactly what I had in mind (to rely on tests order to provide data for the following tests), but your'e right -- that's a bad idea and anti pattern.

In this case, let's start with using Redash's API to create seed data for as long as possible. If we ever feel this isn't enough, we can look into creating some seed helpers elsewhere.

One thing I would consider not doing is to avoid cleaning the database between tests, as this will really slow them down. How about we do some setup before all tests -- like create organization, first admin user and a non admin user, and the rest create per test, but keep it.

[bahmutov]

if test setup with cy.exec and cy.request gets too messy, I suggest using cy.task

[gabrieldutra]

I already have some code here and I'm planning on opening the PR by tomorrow. For me it sounds fine to use Redash's API with cy.request or cy.task (thanks @bahmutov), I'm also trying to make the tests as dynamic as I can, so one does not affect others.

For the test script I'm thinking about using a Node Script with something like:

1. Run docker-compose -p cypress up -d and create db
2. Run setup
   • I could use a POST request for this.
   • if we run the setup here we won't have a setup_spec.js
   • at this point it will also be good to create all other base data we might need, such as a non-admin user
3. Run Cypress with the Module API
   • Could be Cypress UI or only the tests depending on params
4. Run docker-compose -p cypress down

The advantage of using Node Scripts for this is that we won't have anything but JavaScript related, therefore it will be easier for anyone to work with cypress tests.

We could keep setup_spec.js by making sure this script runs before all the others. But I prefer not having this test and make it through the test script.

Please tell me if you have a better idea for any of this 😁

[arikfr]

Sounds like a good plan! My only concern is the use of the Module API instead of the CLI. I'm worried we might find ourselves having to rewrite their CLI eventually... But we can start this way and revisit if needed in the future.

Eager to see this coming together :)

[arikfr]

Closing as this is already implemented 🙌