Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flaky: lens smokescreen tests should allow creation of lens visualizations #71203

Closed
timroes opened this issue Jul 9, 2020 · 8 comments
Closed
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Feature:Lens Team:Visualizations Visualization editors, elastic-charts and infrastructure test-failure-flaky

Comments

@timroes
Copy link
Contributor

timroes commented Jul 9, 2020

It seems the following lens smokescreen test is flaky:

x-pack/test/functional/apps/lens/smokescreen·ts.lens app lens smokescreen tests should allow creation of lens visualizations

Example failure:

https://kibana-ci.elastic.co/job/elastic+kibana+pipeline-pull-request/59736/testReport//junit/Chrome%20X-Pack%20UI%20Functional%20Tests/x-pack_test_functional_apps_lens_smokescreen%C2%B7ts/lens_app__lens_smokescreen_tests_should_allow_creation_of_lens_visualizations

Failure stats of the last 60 days:

screenshot-20200709-091123

cc @flash1293 @mbondyra @wylieconlon

@timroes timroes added Team:Visualizations Visualization editors, elastic-charts and infrastructure failed-test A test failure on a tracked branch, potentially flaky-test Feature:Lens :KibanaApp/fix-it-week test-failure-flaky labels Jul 9, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-app (Team:KibanaApp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-test-triage (failed-test)

@timroes timroes changed the title Flaky Lens smoketest: Flaky: lens smokescreen tests should allow creation of lens visualizations Jul 9, 2020
@wylieconlon
Copy link
Contributor

Some of the "flakiness" in this test was caused by real changes in the functional test environment on cloud that caused our tests to fail. We've made a number of changes to it over the last 60 days to fix it, including:

#69519
#68677
#67175

This particular example failed with the stack trace

Error: retry.try timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="lnsXY_splitDimensionPanel"] [data-test-subj="lns-empty-dimension"])
Wait timed out after 10020ms
    at /dev/shm/workspace/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
    at process._tickCallback (internal/process/next_tick.js:68:7)
    at onFailure (/dev/shm/workspace/parallel/20/kibana/test/common/services/retry/retry_for_success.ts:28:9)
    at retryForSuccess (/dev/shm/workspace/parallel/20/kibana/test/common/services/retry/retry_for_success.ts:68:13)

And had the screenshot:

lens app lens smokescreen tests should allow creation of lens visualizations

Notice the "Break down by" is empty, which shouldn't happen normally.

I suspect that many of the flaky issues we're seeing in Lens tests are because of the general issue with needing to double-click elements to get focus.

@timroes
Copy link
Contributor Author

timroes commented Jul 9, 2020

@wylieconlon do we have any analysis for why the double click needs to happen? (maybe it's related to elements having an EuiTooltip or Popover attached, actually require two clicks on a mobile device to be triggered - potentially similar issue here?)

If this happens in a lot of places I wonder, if this is something we can/should somehow address more in the service layer of our functional testing framework in a more ubiquitous manner?

@wylieconlon
Copy link
Contributor

It's definitely related to EUI elements, and does seem to happen when the buttons are in popovers/modals most often.

@wylieconlon
Copy link
Contributor

Opened an issue for it #71256

@flash1293
Copy link
Contributor

@wylieconlon I think this can be closed due to #71986 , right?

@wylieconlon
Copy link
Contributor

I think so. If the issue comes back, we can reopen this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Feature:Lens Team:Visualizations Visualization editors, elastic-charts and infrastructure test-failure-flaky
Projects
None yet
Development

No branches or pull requests

4 participants