Update stack versions after 8.11 stack release

[rhr323]

This PR updates the stack versions tested to 8.11.1 and 8.12.0-SNAPSHOT.

[rhr323]

Thanks @thbkrkr , I applied the changes you recommended.

I also found more 8.11.0, which I guess we should also update: 406489e (let me know if not 😉 )

[thbkrkr]

I also found more 8.11.0, which I guess we should also update: 406489e (let me know if not 😉 )

We can but it doesn't matter, it's just for testing that if we set x.y.z, we get x.y.z.

I missed that we also need to update the examples, not only Chart.yaml, so:

while read f; do \
    sed -i.bak -E "s|0\.8\.0-SNAPSHOT|0.9.0-SNAPSHOT|g" "$f"; rm "$f.bak"; \
done < <(find deploy/eck-stack -name Chart.yaml)

[rhr323]

buildkite test this -f p=kind -m s=8.11.1,s=8.12.0-SNAPSHOT

[rhr323]

buildkite test this -f p=kind -m s=8.11.1,s=8.12.0-SNAPSHOT

[thbkrkr]

buildkite test this -f p=gke -m s=8.11.1,s=8.12.0-SNAPSHOT

https://buildkite.com/elastic/cloud-on-k8s-operator/builds/6295

Test failures:
🐞 TestMetricbeatStackMonitoringRecipe/ES_data_should_pass_validations ~ gke-8-11-1
🐞 TestEntSearchAssociation/ES_cluster_health_should_eventually_be_green ~ gke-8-12-0-snaps
🐞 TestKBStackMonitoring/ES_cluster_health_should_eventually_be_green#02 ~ gke-8-12-0-snaps
🐞 TestKibanaAssociationWhenReferencedESDisappears/ES_cluster_health_should_eventually_be_green ~ gke-8-12-0-snaps
🐞 TestKibanaStandalone/ES_cluster_health_should_eventually_be_green ~ gke-8-12-0-snaps
🐞 TestKillKibanaDeployment/ES_endpoint_should_eventually_be_reachable#01 ~ gke-8-12-0-snaps
🐞 TestKillKibanaPod/ES_cluster_health_should_eventually_be_green#01 ~ gke-8-12-0-snaps
🐞 TestMetricbeatStackMonitoringRecipe/ES_data_should_pass_validations ~ gke-8-12-0-snaps

[rhr323]

buildkite test this -f p=gke -m s=8.11.1,s=8.12.0-SNAPSHOT

https://buildkite.com/elastic/cloud-on-k8s-operator/builds/6323

Test failures (as in previous run):
🐞 TestMetricbeatStackMonitoringRecipe/ES_data_should_pass_validations ~ gke-8-11-1
🐞 TestEntSearchAssociation/ES_cluster_health_should_eventually_be_green ~ gke-8-12-0-snaps
🐞 TestKibanaAssociationWhenReferencedESDisappears/ES_cluster_health_should_eventually_be_green ~ gke-8-12-0-snaps
🐞 TestKillKibanaDeployment/ES_cluster_health_should_eventually_be_green#01 ~ gke-8-12-0-snaps
🐞 TestKillKibanaPod/ES_cluster_health_should_eventually_be_green#01 ~ gke-8-12-0-snaps
🐞 TestMetricbeatStackMonitoringRecipe/ES_data_should_pass_validations ~ gke-8-12-0-snaps

New test failures (compared to previous run):
🐞 TestAPMKibanaAssociation/ES_cluster_health_should_eventually_be_green ~ gke-8-12-0-snaps
🐞 TestCrossNSAssociation/ES_cluster_health_should_eventually_be_green ~ gke-8-12-0-snaps

Old test failures (only in the previous run, possibly flaky):
🐞 TestKBStackMonitoring/ES_cluster_health_should_eventually_be_green#02 ~ gke-8-12-0-snaps
🐞 TestKibanaStandalone/ES_cluster_health_should_eventually_be_green ~ gke-8-12-0-snaps

[thbkrkr]

TestMetricbeatStackMonitoringRecipe failed because of #7322, reverted in #7328.

[rhr323]

buildkite test this -f p=gke -m s=8.11.1,s=8.12.0-SNAPSHOT

=> https://buildkite.com/elastic/cloud-on-k8s-operator/builds/6404

Test failures:
🐞 TestKBStackMonitoring/ES_cluster_health_should_eventually_be_green#02 ~ gke-8-12-0-snaps
🐞 TestUpdateKibanaSecureSettings/Kibana_Pods_should_eventually_be_ready ~ gke-8-12-0-snaps

will look into them to see if they are the same issue with data stream indices having replicas.

---

Update:

🐞 TestKBStackMonitoring/ES_cluster_health_should_eventually_be_green#02 ~ gke-8-12-0-snaps:

.ds-metrics-fleet_server.agent_status-default index is created with a replica

🐞 TestUpdateKibanaSecureSettings/Kibana_Pods_should_eventually_be_ready ~ gke-8-12-0-snaps

keystore_initialized_flag=/usr/share/kibana/config/elastic-internal-init-keystore.ok
Initializing keystore.

• [[ -f /usr/share/kibana/config/elastic-internal-init-keystore.ok ]]
• echo 'Initializing keystore.'
• /usr/share/kibana/bin/kibana-keystore create
  Error: ENOENT: no such file or directory, open '/usr/share/kibana/config/kibana.yml'
  at Object.openSync (node:fs:603:3)
  at readFileSync (node:fs:471:35)
  at readYaml (/usr/share/kibana/node_modules/@kbn/config/src/raw/read_config.js:20:69)
  at getConfigFromFiles (/usr/share/kibana/node_modules/@kbn/config/src/raw/read_config.js:56:18)
  at buildDataPaths (/usr/share/kibana/node_modules/@kbn/utils/src/path/index.js:41:82)
  at getDataPath (/usr/share/kibana/node_modules/@kbn/utils/src/path/index.js:71:36)
  at getKeystore (/usr/share/kibana/src/cli/keystore/get_keystore.js:21:63)
  at Object. (/usr/share/kibana/src/cli_keystore/cli_keystore.js:25:71)
  at Module._compile (node:internal/modules/cjs/loader:1256:14)
  at Object.Module._extensions..js (node:internal/modules/cjs/loader:1310:10)