Skip to content

Commit

Permalink
[SIEM] Run Cypress Tests Against Elastic Cloud & Cypress Command Line…
Browse files Browse the repository at this point in the history
… / Reporting (elastic#42804)

This PR contains enhancements to the `Cypress` end-to-end (e2e) testing support in the SIEM app, as discussed with @spalger

Note: this PR depends on test refactorings in elastic#42766

Cypress tests may now be run against a remote Elastic Cloud instance
(override `baseUrl`), interactively or via the command line.

Credentials are specified via `kibna.dev.yml` or environment variables.

Run tests on the command line (override `baseUrl` and pass credentials via
the `CYPRESS_ELASTICSEARCH_USERNAME` and `CYPRESS_ELASTICSEARCH_PASSWORD`
environment variables), via command line.

Reports are configured to include:

- An HTML report, suitable for e-mail notifications
- Screenshots
- Junit reports (for integration with Kibana CI)
- Videos (optional)

Reports generated when `Cypress` tests are run on the command line are output
to the `target` directory, which follows specific conventions used by the
Kibana CI process:

* Any directory under `target` that begins with `kibana-`, e.g. `kibana-siem` will be uploaded as an artifact to a bucket
* Junit reports are picked up from the `target/junit` directory

The artifacts generated by running `Cypress` tests on the command line conform to the above conventions.

An HTML report (e.g. for email notifications) is output to:

```
target/kibana-siem/cypress/results/output.html
```

Screenshots of failed tests are output to:

```
target/kibana-siem/cypress/screenshots
```

The Kibana CI process reports `junit` test results from the `target/junit` directory.

Cypress `junit` reports are generated in `target/kibana-siem/cypress/results`
and copied to the `target/junit` directory.

Videos are disabled by default, but can optionally be enabled by setting the
`CYPRESS_video=true` environment variable:

```
CYPRESS_video=true yarn cypress:run
```

Videos are (optionally) output to:

```
target/kibana-siem/cypress/videos
```

The full details and instructions for running Cypress tests are in the
`siem/cypress/README.md`, which was updated to reflect the changes
in this in this PR, but in a nutshell, you may run tests interactively with
the following (new) environment variables:

```sh
cd x-pack/legacy/plugins/siem
CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:open
```

Running the command line version of the tests, which will output
the reports described above, is a similar experence:

```sh
cd x-pack/legacy/plugins/siem
CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:run
```

elastic/siem-team#435
elastic/siem-team#437

- regen yarn.lock
  • Loading branch information
andrew-goldstein committed Aug 14, 2019
1 parent f3391bd commit 1aa4f88
Show file tree
Hide file tree
Showing 8 changed files with 483 additions and 73 deletions.
6 changes: 5 additions & 1 deletion x-pack/legacy/plugins/siem/cypress.json
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
{
"baseUrl": "http://localhost:5601"
"baseUrl": "http://localhost:5601",
"screenshotsFolder": "../../../../target/kibana-siem/cypress/screenshots",
"trashAssetsBeforeRuns": false,
"video": false,
"videosFolder": "../../../../target/kibana-siem/cypress/videos"
}
228 changes: 200 additions & 28 deletions x-pack/legacy/plugins/siem/cypress/README.md
Original file line number Diff line number Diff line change
@@ -1,70 +1,242 @@
# Cypress Tests

The `siem/cypress` directory contains end to end tests (specific to the `SIEM` app) that execute via [Cypress](https://www.cypress.io/).
The `siem/cypress` directory contains end to end tests, (plus a few tests
that rely on mocked API calls), that execute via [Cypress](https://www.cypress.io/).

At present, these tests are only executed in a local development environment; they are **not** integrated in the Kibana CI infrastructure, and therefore do **not** run automatically when you submit a PR.
Cypress tests may be run against:

See the `Server and Authentication Requirements` section below for additional details.
- A local Kibana instance, interactively or via the command line. Credentials
are specified via `kibna.dev.yml` or environment variables.
- A remote Elastic Cloud instance (override `baseUrl`), interactively or via
the command line. Again, credentials are specified via `kibna.dev.yml` or
environment variables.
- As part of CI (override `baseUrl` and pass credentials via the
`CYPRESS_ELASTICSEARCH_USERNAME` and `CYPRESS_ELASTICSEARCH_PASSWORD`
environment variables), via command line.

## Organizing Tests and (Mock) Data
At present, Cypress tests are only executed manually. They are **not** yet
integrated in the Kibana CI infrastructure, and therefore do **not** run
automatically when you submit a PR.

- Code and CSS selectors that may be re-used across tests should be added to `siem/cypress/integration/lib`, as described below
- Smoke Tests are located in `siem/cypress/integration/smoke_tests`
- Mocked responses from the server are located in `siem/cypress/fixtures`
## Smoke Tests

### `cypress/integration/lib`
Smoke Tests are located in `siem/cypress/integration/smoke_tests`

The `cypress/integration/lib` folder contains code intended to be re-used across many different tests.
## Test Helpers

- Files named `helpers.ts` (e.g. `siem/cypress/integration/lib/login/helpers.ts`) contain functions (e.g. `login`) that may be imported and invoked from multiple tests.
_Test helpers_ are functions that may be re-used across tests.

- Files named `selectors.ts` export CSS selectors for re-use. For example, `siem/cypress/integration/lib/login/selectors.ts` exports the following selector that matches the Username text area in the Kibana login page:
- Reusable code and CSS selectors should be added to
`siem/cypress/integration/lib`, as described below.

```
### Reusable Test Helper Functions and CSS Selectors

The `cypress/integration/lib` directory contains code intended to be re-used
across many different tests. Add reusable test helper functions and CSS
selectors to directories under `cypress/integration/lib`.

- Files named `helpers.ts` (e.g. `siem/cypress/integration/lib/login/helpers.ts`)
contain functions (e.g. `login`) that may be imported and invoked from multiple tests.

- Files named `selectors.ts` export CSS selectors for re-use. For example,
`siem/cypress/integration/lib/login/selectors.ts` exports the following selector
that matches the Username text area in the Kibana login page:

```sh
export const USERNAME = '[data-test-subj="loginUsername"]';
```

## Server and Authentication Requirements
## Mock Data

We prefer not to mock API responses in most of our smoke tests, but sometimes
it's necessary because a test must assert that a specific value is rendered,
and it's not possible to derive that value based on the data in the
envrionment where tests are running.

Mocked responses API from the server are located in `siem/cypress/fixtures`.

## Authentication

When running tests, there are two ways to specify the credentials used to
authenticate with Kibana:

- Via `kibana.dev.yml` (recommended for developers)
- Via the `CYPRESS_ELASTICSEARCH_USERNAME` and `CYPRESS_ELASTICSEARCH_PASSWORD`
environment variables (recommended for CI), or when testing a remote Kibana
instance, e.g. in Elastic Cloud.

Note: Tests that use the `login()` test helper function for authentication will
automatically use the `CYPRESS_ELASTICSEARCH_USERNAME` and `CYPRESS_ELASTICSEARCH_PASSWORD`
environment variables when they are defined, and fall back to the values in
`config/kibana.dev.yml` when they are unset.

### Content Security Policy (CSP) Settings

Your local or cloud Kibana server must have the `csp.strict: false` setting
configured in `kibana.dev.yml`, or `kibana.yml`, as shown in the example below:

```yaml
csp.strict: false
```
The above setting is required to prevent the _Please upgrade
your browser_ / _This Kibana installation has strict security requirements
enabled that your current browser does not meet._ warning that's displayed for
unsupported user agents, like the one reported by Cypress when running tests.
### Example `kibana.dev.yml`

The current version of the Smoke Tests require running a local Kibana server that connects to an instance of `elasticsearch`. A file named `config/kibana.dev.yml` like the example below is required to run the tests:
If you're a developer running tests interactively or on the command line, the
easiset way to specify the credentials used for authentication is to update
`kibana.dev.yml` per the following example:

```yaml
csp.strict: false
elasticsearch:
username: 'elastic'
password: '<password>'
hosts: ['https://<server>:9200']
```

The `username` and `password` from `config/kibana.dev.yml` will be read by the `login` test helper function when tests authenticate with Kibana.
## Running Tests Interactively

See the `Running Tests Interactively` section for details.
Use the Cypress interactive test runner to develop and debug specific tests
by adding a `.only` to the test you're developing, or click on a specific
spec in the interactive test runner to run just the tests in that spec.

### Content Security Policy (CSP) Settings
To run and debug tests in interactively via the Cypress test runner:

Your local or cloud Kibana server must have the `csp.strict: false` setting
configured in `kibana.dev.yml`, or `kibana.yml`, as shown in the example below:
1. Disable CSP on the local or remote Kibana instance, as described in the
_Content Security Policy (CSP) Settings_ section above.

```yaml
csp.strict: false
2. To specify the credentials required for authentication, configure
`config/kibana.dev.yml`, as described in the _Server and Authentication
Requirements_ section above, or specify them via environment variables
as described later in this section.

3. Start a local instance of the Kibana development server (only if testing against a
local host):

```sh
yarn start --no-base-path
```

## Running Tests Interactively
4. Launch the Cypress interactive test runner via one of the following options:

To run tests in interactively via the Cypress test runner:
- To run tests interactively against the default (local) host specified by
`baseUrl`, as configured in `plugins/siem/cypress.json`:

1. Create and configure a `config/kibana.dev.yml`, as described in the `Server and Authentication Requirements` section above.
```sh
cd x-pack/legacy/plugins/siem
yarn cypress:open
```

- To (optionally) run tests interactively against a different host, pass the
`CYPRESS_baseUrl` environment variable on the command line when launching the
test runner, as shown in the following example:

```sh
cd x-pack/legacy/plugins/siem
CYPRESS_baseUrl=http://localhost:5601 yarn cypress:open
```

2. Start a local instance of the Kibana development server:
- To (optionally) override username and password via environment variables when
running tests interactively:

```sh
cd x-pack/legacy/plugins/siem
CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:open
```

5. Click the `Run all specs` button in the Cypress test runner (after adding
a `.only` to an `it` or `describe` block).

## Running (Headless) Tests on the Command Line

To run (headless) tests on the command line:

1. Disable CSP on the local or remote Kibana instance, as described in the
_Content Security Policy (CSP) Settings_ section above.

2. To specify the credentials required for authentication, configure
`config/kibana.dev.yml`, as described in the _Server and Authentication
Requirements_ section above, or specify them via environment variables
as described later in this section.

3. Start a local instance of the Kibana development server (only if testing against a
local host):

```sh
yarn start --no-base-path
```

3. Launch the Cypress interactive test runner:
4. Launch the Cypress command line test runner via one of the following options:

- To run tests on the command line against the default (local) host specified by
`baseUrl`, as configured in `plugins/siem/cypress.json`:

```sh
cd x-pack/legacy/plugins/siem
yarn cypress:open
yarn cypress:run
```

- To (optionally) run tests on the command line against a different host, pass
`CYPRESS_baseUrl` as an environment variable on the command line, as shown in
the following example:

```sh
cd x-pack/legacy/plugins/siem
CYPRESS_baseUrl=http://localhost:5601 yarn cypress:run
```

4. Click the `Run all specs` button in the Cypress test runner
- To (optionally) override username and password via environment variables when
running via the command line:

```sh
cd x-pack/legacy/plugins/siem
CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:run
```

## Reporting

When Cypress tests are run on the command line via `yarn cypress:run`,
reporting artifacts are generated under the `target` directory in the root
of the Kibana, as detailed for each artifact type in the sections bleow.

### HTML Reports

An HTML report (e.g. for email notifications) is output to:

```
target/kibana-siem/cypress/results/output.html
```

### Screenshots

Screenshots of failed tests are output to:

```
target/kibana-siem/cypress/screenshots
```

### `junit` Reports

The Kibana CI process reports `junit` test results from the `target/junit` directory.

Cypress `junit` reports are generated in `target/kibana-siem/cypress/results`
and copied to the `target/junit` directory.

### Videos (optional)

Videos are disabled by default, but can optionally be enabled by setting the
`CYPRESS_video=true` environment variable:

```
CYPRESS_video=true yarn cypress:run
```

Videos are (optionally) output to:

```
target/kibana-siem/cypress/videos
```
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ import {
} from '../../lib/fields_browser/selectors';
import { logout } from '../../lib/logout';
import { HOSTS_PAGE } from '../../lib/urls';
import { loginAndWaitForPage } from '../../lib/util/helpers';
import { loginAndWaitForPage, DEFAULT_TIMEOUT } from '../../lib/util/helpers';

const defaultHeaders = [
{ id: '@timestamp' },
Expand Down Expand Up @@ -200,9 +200,9 @@ describe('Fields Browser', () => {

cy.get(`[data-test-subj="headers-group"]`).then(headersDropArea => drop(headersDropArea));

clickOutsideFieldsBrowser();

cy.get(`[data-test-subj="header-text-${toggleField}"]`).should('exist');
cy.get(`[data-test-subj="header-text-${toggleField}"]`, { timeout: DEFAULT_TIMEOUT }).should(
'exist'
);
});

it('resets all fields in the timeline when `Reset Fields` is clicked', () => {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import { populateTimeline } from '../../lib/fields_browser/helpers';
import { logout } from '../../lib/logout';
import { toggleFirstEventDetails } from '../../lib/timeline/helpers';
import { HOSTS_PAGE } from '../../lib/urls';
import { loginAndWaitForPage } from '../../lib/util/helpers';
import { loginAndWaitForPage, DEFAULT_TIMEOUT } from '../../lib/util/helpers';

describe('toggle column in timeline', () => {
beforeEach(() => {
Expand Down Expand Up @@ -72,6 +72,8 @@ describe('toggle column in timeline', () => {

cy.get(`[data-test-subj="headers-group"]`).then(headersDropArea => drop(headersDropArea));

cy.get(`[data-test-subj="header-text-${idField}"]`).should('exist');
cy.get(`[data-test-subj="header-text-${idField}"]`, { timeout: DEFAULT_TIMEOUT }).should(
'exist'
);
});
});
23 changes: 20 additions & 3 deletions x-pack/legacy/plugins/siem/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,20 +6,37 @@
"license": "Elastic-License",
"scripts": {
"build-graphql-types": "node scripts/generate_types_from_graphql.js",
"cypress:open": "cypress open"
"cypress:open": "cypress open",
"cypress:run": "cypress run --spec ./cypress/integration/**/*.spec.ts --reporter mocha-multi-reporters --reporter-options configFile=./reporter_config.json; mochawesome-merge --reportDir ../../../../target/kibana-siem/cypress/results > ../../../../target/kibana-siem/cypress/results/output.json; marge ../../../../target/kibana-siem/cypress/results/output.json --reportDir ../../../../target/kibana-siem/cypress/results; mkdir -p ../../../../target/junit && cp ../../../../target/kibana-siem/cypress/results/*.xml ../../../../target/junit/"
},
"devDependencies": {
"@cypress/webpack-preprocessor": "^4.1.0",
"@types/js-yaml": "^3.12.1",
"@types/lodash": "^4.14.110",
"@types/react-beautiful-dnd": "^10.0.1",
"cypress": "^3.3.1",
"cypress": "^3.4.1",
"js-yaml": "^3.13.1",
"mocha-junit-reporter": "^1.23.1",
"mocha-multi-reporters": "^1.1.7",
"mochawesome": "^4.0.1",
"mochawesome-merge": "^2.0.1",
"mochawesome-report-generator": "^4.0.1",
"ts-loader": "^6.0.4"
},
"dependencies": {
"lodash": "^4.17.13",
"react-beautiful-dnd": "^10.0.1",
"react-markdown": "^4.0.6"
},
"workspaces": {
"packages": [
"packages/*"
],
"nohoist": [
"**/mochawesome",
"**/mochawesome/**",
"**/mocha-multi-reporters",
"**/mocha-multi-reporters/**"
]
}
}
}
10 changes: 10 additions & 0 deletions x-pack/legacy/plugins/siem/reporter_config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"reporterEnabled": "mochawesome, mocha-junit-reporter",
"reporterOptions": {
"html": false,
"json": true,
"mochaFile": "../../../../target/kibana-siem/cypress/results/results-[hash].xml",
"overwrite": false,
"reportDir": "../../../../target/kibana-siem/cypress/results"
}
}
Loading

0 comments on commit 1aa4f88

Please sign in to comment.