[Testing] Implement E2E / Integration Testing

[tonyanziano]

It would be great to have E2E tests that we could run on the app. Ideally in packaged form, but unpackaged would be better than what we currently have.

Here are some options:

Spectron - a framework for testing un / packaged Electron apps
Cypress - framework for testing anything that runs in a browser (would take some extra effort to get it working with the Emulator)

UPDATED:

Initial e2e testing infrastructure has been set up using Spectron in PR #1696

This issue will now be used to track implementing integration tests for all the corresponding release checklist items in #1532.

[stevengum]

@tonyanziano Can we add Gov integration tests for just the backend logic to this work item? I have a few bots that are working in US Gov Azure that I can add Inspection Middleware to.

[tonyanziano]

@stevengum I think that's a good idea, but it would be good to think about how we could verify that on the backend.

Spectron's API is primarily focused on exposing the WebDriverIO API (for querying the frontend), and also some of the Electron APIs (for querying the main process / renderer process). However, it lacks the ability for us to reach into a specific code path of the back-end and verify that a function was called. It's better at performing a UI action and then verifying that the frontend changed accordingly or using a Node API like fs to check that the correct file was created / modified.

What we could do is add an additional case to the current create bot e2e test that checks the "Azure for US Government" checkbox when creating the bot, and then verifies that the gov channel service was written to the endpoint service entry in the .bot file.

It's not exactly testing the same thing, but it's something.

[stevengum]

On reflection the test I was thinking of/proposing is not under the tests that would be covered Spectron, but is rather a partial integration test in that it makes sure that the backend of the Emulator can talk with US Gov Azure-created bots. I just threw out the idea as a method of increasing the test coverage, but not related to the incoming Spectron tests 😄

[tonyanziano]

@stevengum ahhh gotcha.

I haven't started any of the work for integration tests, but I like the idea of testing gov bots once that happens.

[tonyanziano]

#1763 Will cover most of the core user scenarios in the Emulator.

Some of the following scenarios are not easily testable due to varying restrictions:

• Inspector bot (need to launch ngrok and configure bot endpoint in Azure portal)
• Logging into Azure (requires exposing credentials)
• Luis inspectors (requires exposing credentials)
• Services pickers (requires exposing credentials / also under-used according to telemetry data)