Make expectSnapshot available in all functional test runs #82932

dgieselaar · 2020-11-09T10:51:35Z

Haven't figured out a way to test everything as jest-snapshot sidesteps mocking of fs, added some smoke tests where possible.

Closes elastic#80292.

dgieselaar · 2020-11-09T13:13:03Z

Hmm, I think I'll need some help with the types - looks like the Mocha types are conflicting with the Jest types.

…ing-ftr

spalger

This looks great, I plan to run this and try it out in some test locally later today but the code looks great.

packages/kbn-test/src/functional_test_runner/lib/mocha/load_test_files.js

…ing-ftr

spalger

Only thing I feel like I'm missing from this implementation of snapshots is the summary at the bottom of the test output that indicates how many snapshots were checked/failed/updated, etc. If you can produce some summary information that's tracked by the mocha reporter then you should be able to include information like that in the epilogue

spalger · 2020-11-10T16:50:47Z

packages/kbn-test/src/functional_test_runner/cli.ts

@@ -141,6 +150,7 @@ export function runFtrCli() {
        --exclude-tag=tag  a tag to be excluded, pass multiple times for multiple tags
        --test-stats       print the number of tests (included and excluded) to STDERR
        --updateBaselines  replace baseline screenshots with whatever is generated from the test
+        --updateSnapshots  replace inline and file snapshots with whatever is generated from the test


I definitely reached for the -u flag here, maybe it would be nice to have an --update,-u flag that sets both updateBaselines and updateSnapshots.

done - I tried to add -u only where necessary, but let me know if I missed anything.

x-pack/test/snapshot_declarations.d.ts

…ing-ftr

dgieselaar · 2020-11-11T09:38:10Z

@spalger I can't quite figure out how to collect stats. I'm not sure if I can get access to the reporter, or emit events that the reporter can use. Any clues where I should begin?

dgieselaar · 2020-11-16T09:02:30Z

@spalger Looks like the typings in test/typings are not picked up in x-pack, is that correct or am I holding this wrong?

spalger

LGTM!

packages/kbn-test/src/functional_test_runner/lib/snapshots/decorate_snapshot_ui.ts

…em correctly

spalger · 2020-11-18T19:12:18Z

test/tsconfig.json

@@ -4,7 +4,7 @@
    "tsBuildInfoFile": "../build/tsbuildinfo/test",
    "types": ["node", "mocha", "flot"]
  },
-  "include": ["**/*", "../typings/elastic__node_crypto.d.ts", "typings/**/*"],
+  "include": ["**/*", "../typings/elastic__node_crypto.d.ts", "typings/**/*", "../packages/kbn-test/types/ftr_globals/**/*"],


Went ahead and unified the global mocha and snapshot types under this directory in @kbn/test and included them in both the test/tsconfig.json and x-pack/test/tsconfig.json projects.

kibanamachine · 2020-11-18T20:55:28Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 610387f

Metrics [docs]

✅ unchanged

History

💔 Build #88667 failed c8583e7
💔 Build #88661 failed 2825618
💔 Build #87305 failed 148982d
💔 Build #87299 failed 779ab6b
💔 Build #87291 failed 22bd132

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

dgieselaar · 2020-11-19T07:38:44Z

Thanks for all the help @spalger!

) Co-authored-by: spalger <spalger@users.noreply.github.com> # Conflicts: # packages/kbn-test/src/functional_test_runner/cli.ts

* master: (60 commits) Forward any registry cache-control header for files (elastic#83680) Revert "[Alerting] Add `alert.updatedAt` field to represent date of last user edit (elastic#83578)" [Security Solution][Detections] Fix adding an action to detection rules (elastic#83722) Make expectSnapshot available in all functional test runs (elastic#82932) Skip failing cypress test Increase bulk request timeout during esArchiver load (elastic#83657) [data.search] Server-side background session service (elastic#81099) [maps] convert VectorStyleEditor to TS (elastic#83582) Revert "[App Search] Engine overview layout stub (elastic#83504)" Adding documentation for global action configuration options (elastic#83557) [Metrics UI] Optimizations for Snapshot and Inventory Metadata (elastic#83596) chore(NA): update lmdb store to v0.8.15 (elastic#83726) [App Search] Engine overview layout stub (elastic#83504) [Workplace Search] Update SourceIcon to match latest changes in ent-search (elastic#83714) [Enterprise Search] Rename React Router helpers (elastic#83718) [Maps] Add 'crossed' & 'exited' events to tracking alert (elastic#82463) Updating code-owners to use new core/app-services team names (elastic#83731) Add Managed label to data streams and a view switch for the table (elastic#83049) [Maps] Add query bar inputs to geo threshold alerts tracked points & boundaries (elastic#80871) fix(NA): search examples kibana version declaration (elastic#83182) ...

…ode-details * 'master' of github.com:elastic/kibana: fixed pagination in connectors list (elastic#83638) Forward any registry cache-control header for files (elastic#83680) Revert "[Alerting] Add `alert.updatedAt` field to represent date of last user edit (elastic#83578)" [Security Solution][Detections] Fix adding an action to detection rules (elastic#83722) Make expectSnapshot available in all functional test runs (elastic#82932) Skip failing cypress test Increase bulk request timeout during esArchiver load (elastic#83657) [data.search] Server-side background session service (elastic#81099) [maps] convert VectorStyleEditor to TS (elastic#83582) Revert "[App Search] Engine overview layout stub (elastic#83504)" Adding documentation for global action configuration options (elastic#83557) [Metrics UI] Optimizations for Snapshot and Inventory Metadata (elastic#83596) chore(NA): update lmdb store to v0.8.15 (elastic#83726) [App Search] Engine overview layout stub (elastic#83504) [Workplace Search] Update SourceIcon to match latest changes in ent-search (elastic#83714) [Enterprise Search] Rename React Router helpers (elastic#83718)

Co-authored-by: spalger <spalger@users.noreply.github.com>

) Co-authored-by: spalger <spalger@users.noreply.github.com>

dgieselaar force-pushed the snapshot-testing-ftr branch 3 times, most recently from 6f898b0 to f2adbb4 Compare November 9, 2020 10:56

Make expectSnapshot available in all functional test runs

6e95ee7

Closes elastic#80292.

dgieselaar force-pushed the snapshot-testing-ftr branch from f2adbb4 to 6e95ee7 Compare November 9, 2020 11:12

dgieselaar marked this pull request as ready for review November 9, 2020 12:42

dgieselaar requested a review from a team as a code owner November 9, 2020 12:42

dgieselaar added release_note:skip Skip the PR/issue when compiling release notes v7.11.0 labels Nov 9, 2020

Add & update tests

7cee115

dgieselaar force-pushed the snapshot-testing-ftr branch from 4fc3d38 to 7cee115 Compare November 9, 2020 12:45

dgieselaar added 2 commits November 9, 2020 15:33

Merge branch 'master' of github.com:elastic/kibana into snapshot-test…

3bb35e5

…ing-ftr

Make sure unused snapshot check only runs once

8f24c32

spalger reviewed Nov 9, 2020

View reviewed changes

packages/kbn-test/src/functional_test_runner/lib/mocha/load_test_files.js Outdated Show resolved Hide resolved

dgieselaar added 5 commits November 9, 2020 20:56

Merge branch 'master' of github.com:elastic/kibana into snapshot-test…

2076517

…ing-ftr

Remove console.log statement

f6f84b5

Fix failing test

7105392

Merge branch 'master' of github.com:elastic/kibana into snapshot-test…

5d9e332

…ing-ftr

Define own Mocha types to prevent Jest conflicts

feb156a

spalger reviewed Nov 10, 2020

View reviewed changes

dgieselaar added 7 commits November 10, 2020 20:22

Merge branch 'master' of github.com:elastic/kibana into snapshot-test…

c88ce41

…ing-ftr

Move snapshot typings file to test/typings

61ed37b

Merge branch 'master' of github.com:elastic/kibana into snapshot-test…

56f0c9b

…ing-ftr

Add -u shorthand option for updating baseline screenshots and snapshots

22bd132

move types

4342dd6

Use existing fake mocha types

779ab6b

Remove unused type import

a12d3ea

Fix type errors

148982d

spalger added 3 commits November 18, 2020 11:08

Merge branch 'master' of github.com:elastic/kibana into pr/82932

af9e222

unify global ftr types under kbn-test

2825618

remove unnecessary arrow -> function convert

78e261a

spalger approved these changes Nov 18, 2020

View reviewed changes

packages/kbn-test/src/functional_test_runner/lib/snapshots/decorate_snapshot_ui.ts Outdated Show resolved Hide resolved

spalger added 4 commits November 18, 2020 11:32

avoid using typescript specific import paths

c8583e7

Merge branch 'master' of github.com:elastic/kibana into pr/82932

0ef5514

don't import ftr globals in kbn-test package

d7e295d

include the files in ftr projects so that project check associates th…

610387f

…em correctly

spalger reviewed Nov 18, 2020

View reviewed changes

dgieselaar merged commit 3a8ea29 into elastic:master Nov 19, 2020

dgieselaar deleted the snapshot-testing-ftr branch November 19, 2020 07:38

dgieselaar mentioned this pull request Nov 19, 2020

[7.x] Make expectSnapshot available (#82932) #83770

Merged

dgieselaar added a commit that referenced this pull request Nov 19, 2020

[7.x] Make expectSnapshot available (#82932) (#83770)

4697b7e

Co-authored-by: spalger <spalger@users.noreply.github.com>

chrisronline pushed a commit to chrisronline/kibana that referenced this pull request Nov 19, 2020

Make expectSnapshot available in all functional test runs (elastic#82932

6bfe02a

) Co-authored-by: spalger <spalger@users.noreply.github.com>

lukeelmers mentioned this pull request Nov 20, 2020

[expressions] Convert interpreter_functional test suite to use new unified expectSnapshot implementation. #83955

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make expectSnapshot available in all functional test runs #82932

Make expectSnapshot available in all functional test runs #82932

dgieselaar commented Nov 9, 2020 •

edited

Loading

dgieselaar commented Nov 9, 2020

spalger left a comment

spalger left a comment

spalger Nov 10, 2020

dgieselaar Nov 11, 2020

dgieselaar commented Nov 11, 2020

dgieselaar commented Nov 16, 2020

spalger left a comment

spalger Nov 18, 2020

kibanamachine commented Nov 18, 2020

dgieselaar commented Nov 19, 2020

Make expectSnapshot available in all functional test runs #82932

Make expectSnapshot available in all functional test runs #82932

Conversation

dgieselaar commented Nov 9, 2020 • edited Loading

dgieselaar commented Nov 9, 2020

spalger left a comment

Choose a reason for hiding this comment

spalger left a comment

Choose a reason for hiding this comment

spalger Nov 10, 2020

Choose a reason for hiding this comment

dgieselaar Nov 11, 2020

Choose a reason for hiding this comment

dgieselaar commented Nov 11, 2020

dgieselaar commented Nov 16, 2020

spalger left a comment

Choose a reason for hiding this comment

spalger Nov 18, 2020

Choose a reason for hiding this comment

kibanamachine commented Nov 18, 2020

💚 Build Succeeded

Metrics [docs]

History

dgieselaar commented Nov 19, 2020

dgieselaar commented Nov 9, 2020 •

edited

Loading