[APM] e2e are failing when running them locally and CI [7.x]

[v1v]

Kibana version:
7.x branch

commit 8dbd3af63bf905bca9127cf13a93d856c6a6b156 (HEAD -> 7.x)
Author: Dario Gieselaar <dario.gieselaar@elastic.co>
Date:   Tue Jun 2 14:25:59 2020 +0200

Elasticsearch version:
master branch

{
name: "70f5565bcc8b",
cluster_name: "docker-cluster",
cluster_uuid: "oJF8ilNAQYqwjWbyf70dHw",
version: {
number: "8.0.0-SNAPSHOT",
build_flavor: "default",
build_type: "docker",
build_hash: "d7b31a8a355c713bbf9c5f5f5887d75e5452dd62",
build_date: "2020-06-02T10:42:48.564111Z",
build_snapshot: true,
lucene_version: "8.6.0",
minimum_wire_compatibility_version: "7.9.0",
minimum_index_compatibility_version: "7.0.0"
},
tagline: "You Know, for Search"
}

Server OS version:
macosx 10.15.4 (19E287)

Browser version:
N/A

Browser OS version:
N/A

Original install method (e.g. download page, yum, from source, etc.):
From source
node --version - v10.19.0
nvm --version - 0.35.3

Description of the problem including expected versus actual behavior:

Kibana does not start

Steps to reproduce:

Run all the below steps from scratch:

cd /tmp
git clone https://github.com/elastic/kibana --depth 1
git checkout 7.x
docker ps -a
cd kibana
nvm use $(cat .node-version)

cd x-pack/plugins/apm/e2e
yarn install
cd -
yarn kbn clean && yarn kbn bootstrap
nohup node ./scripts/kibana --no-base-path --dev --no-dev-config --config x-pack/plugins/apm/e2e/ci/kibana.e2e.yml > /tmp/kibana.log 2>&1 &
x-pack/plugins/apm/e2e/run-e2e.sh

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

➜  kibana git:(7.x) node ./scripts/kibana --no-base-path --dev --no-dev-config --config x-pack/plugins/apm/e2e/ci/kibana.e2e.yml
  log   [08:03:59.098] [debug][config] Marking config path as handled: server
  log   [08:03:59.098] [debug][config] Marking config path as handled: csp
  log   [08:03:59.098] [debug][config] Marking config path as handled: plugins
...
...

  log   [08:20:48.542] [debug][metrics] Refreshing metrics
server    log   [08:20:48.638] [debug][http] Kibana server is not ready yet get:/api/status.
server    log   [08:20:49.138] [debug][http] Kibana server is not ready yet get:/api/status.
server    log   [08:20:49.638] [debug][http] Kibana server is not ready yet get:/api/status.
server    log   [08:20:50.139] [debug][http] Kibana server is not ready yet get:/api/status.
server    log   [08:20:50.645] [debug][http] Kibana server is not ready yet get:/api/status.
...
server    log   [09:00:09.352] [debug][http] Kibana server is not ready yet get:/api/status.
server    log   [09:00:09.854] [debug][http] Kibana server is not ready yet get:/api/status.
  log   [09:00:10.176] [debug][metrics] Refreshing metrics
server    log   [09:00:10.358] [debug][http] Kibana server is not ready yet get:/api/status.
server    log   [09:00:10.861] [debug][http] Kibana server is not ready yet get:/api/status.
server    log   [09:00:11.361] [debug][http] Kibana server is not ready yet get:/api/status.
server    log   [09:00:11.863] [debug][http] Kibana server is not ready yet get:/api/status.
server    log   [09:00:12.367] [debug][http] Kibana server is not ready yet get:/api/status.
optmzr    log   [09:00:12.533] [debug][metrics] Refreshing metrics
server    log   [09:00:12.869] [debug][http] Kibana server is not ready yet get:/api/status.
server    log   [09:00:13.370] [debug][http] Kibana server is not ready yet get:/api/status.
server    log   [09:00:13.870] [debug][http] Kibana server is not ready yet get:/api/status.
server    log   [09:00:14.370] [debug][http] Kibana server is not ready yet get:/api/status.
server    log   [09:00:14.871] [debug][http] Kibana server is not ready yet get:/api/status.
  log   [09:00:15.178] [debug][metrics] Refreshing metrics
server    log   [09:00:15.375] [debug][http] Kibana server is not ready yet get:/api/status.
server    log   [09:00:15.876] [debug][http] Kibana server is not ready yet get:/api/status.
server    log   [09:00:16.376] [debug][http] Kibana server is not ready yet get:/api/status.
server    log   [09:00:16.877] [debug][http] Kibana server is not ready yet get:/api/status.
optmzr    log   [09:00:17.536] [debug][metrics] Refreshing metrics

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[sorenlouv]

I cannot repro locally. @v1v
Can you look into whether this is a kibana issue or a problem with the tests?

[v1v]

I'm afraid this happens also in the CI:

• build 5
• kibana-logs
• apm-its-logs

NOTE: the mp4 file is actually the one that should be deleted with the PR #67949

[ogupte]

Locally, I'm seeing what appears to be a version mismatch between Kibana and ES: [error][savedobjects-service] This version of Kibana (v7.9.0) is incompatible with the following Elasticsearch nodes in your cluster: v8.0.0 @ 172.18.0.2:9200 (172.18.0.2)

[ogupte]

On this line: https://github.com/elastic/kibana/blob/master/x-pack/plugins/apm/e2e/run-e2e.sh#L66

The tests run and pass when I change ${APM_IT_DIR}/scripts/compose.py start master -> ${APM_IT_DIR}/scripts/compose.py start 7.9.0 on the 7.x branch. How should we make this version dynamic?