[7.17] Some UA & Stack Management tests fail when run against ES 8.0

[lukeelmers]

When testing 7.17 Kibana against 8.0 Elasticsearch, we get some UA & stack mgmt-related failures from CI:

Index Management

[job] [logs] Default CI Group #13 / Index Management app feature controls security global dashboard read with index_management_user should render the "Data" section with index management
[job] [logs] Default CI Group #13 / Index Management app feature controls security global dashboard read with index_management_user should render the "Data" section with index management

CJ believes these are known issues with Index Management's use of a deprecated ES API (#122933). Seb might know more about this.

Upgrade Assistant

[job] [logs] Default Accessibility Tests / Upgrade Assistant "after all" hook in "Upgrade Assistant"
[job] [logs] Default Accessibility Tests / Upgrade Assistant "after all" hook in "Upgrade Assistant"
[job] [logs] Default Accessibility Tests / Upgrade Assistant "before all" hook in "Upgrade Assistant"
[job] [logs] Default Accessibility Tests / Upgrade Assistant "before all" hook in "Upgrade Assistant"
[job] [logs] Default CI Group #4 / Upgrade Assistant Deprecation pages "after all" hook for "renders the Kibana deprecations page"
[job] [logs] Default CI Group #4 / Upgrade Assistant Deprecation pages "after all" hook for "renders the Kibana deprecations page"
[job] [logs] Default CI Group #4 / Upgrade Assistant Deprecation pages "before all" hook for "renders the Elasticsearch deprecations page"
[job] [logs] Default CI Group #4 / Upgrade Assistant Deprecation pages "before all" hook for "renders the Elasticsearch deprecations page"

CJ suspects that these failures are due to the test design. He doesn't think they were designed to check for the interstitial states that block the UI during rolling upgrades (#112907). Someone from the team needs to investigate this.

https://buildkite.com/elastic/kibana-7-dot-latest-es-8-dot-0-forward-compatibility/builds/15

In 7.17, basic Kibana functions are still expected to work when run against ES 8.0, because during an upgrade process users will upgrade ES first.

So the ask here is to:

• investigate the failures from the build
• determine if they are simply bugs in our tests, or legit issues
• if they are legit issues, we should determine if they need to be raised as 7.17 blockers

To test locally (be sure to update the config paths & string to grep for):

git checkout 7.17
TEST_ES_BRANCH=8.0.0 node scripts/functional_tests_server.js --config=path/to/suite/config.ts
# in another terminal session
node scripts/functional_test_runner --config=path/to/suite/config.ts --grep="whatever"

[elasticmachine]

Pinging @elastic/platform-deployment-management (Team:Deployment Management)

[cjcenizal]

@lukeelmers If these turn out to be issues with test design, do we need to fix them and if so do we have a deadline?

[cjcenizal]

@elastic/platform-deployment-management Let's investigate these failures, figure out what's causing them, and record that information here, but hold off on writing any code until we have a better understanding of what's required.

[alisonelizabeth]

I'm not able to find the branch no-include_type_name mentioned in the issue description to run these tests locally. However, based on the logs, I believe the Upgrade Assistant tests are failing because they are trying to set up deprecated features in a before() hook in order to test the deprecation pages. However, on 8.0, these deprecated features are blocked and therefore the test setup fails.

[rudolf]

We merged the PR and deleted that branch, but I've updated the instructions in the issue to use the 7.17 branch instead.

[alisonelizabeth]

I can confirm my comment above is correct regarding the test failures. CJ is also correct in that we do not test for the interstitial states.

I see Spencer has already opened #123312 to filter tests that should only run on a 7.17 cluster, so I’m not sure any additional action is needed at this time.

[alisonelizabeth]

I'm going to close this issue as I believe the tests have been fixed as commented in #123083 (comment).

#122933 tracks any follow-up work needed for Index Management, and #123063 for reworking the tests in UA.