Merge branch 'filebeat-deployment-support-feature' of github.com:oper…

…atorequals/helm-charts into filebeat-deployment-support-feature

.ci/jobs/elastic+helm-charts+pull-request+cluster-cleanup.yml

@@ -3,6 +3,7 @@
     name: elastic+helm-charts+pull-request+cluster-cleanup
     display-name: elastic / helm-charts - pull-request - cluster cleanup
     description: Pull request - cluster cleanup
+    concurrent: true
     scm:
     - git:
         refspec: +refs/pull/*:refs/remotes/origin/pr/*

.ci/jobs/elastic+helm-charts+pull-request+cluster-creation.yml

@@ -3,6 +3,7 @@
     name: elastic+helm-charts+pull-request+cluster-creation
     display-name: elastic / helm-charts - pull-request - cluster creation
     description: Pull request - cluster creation
+    concurrent: true
     scm:
     - git:
         refspec: +refs/pull/*:refs/remotes/origin/pr/*

.ci/jobs/elastic+helm-charts+pull-request+integration-apm-server.yml

@@ -3,6 +3,7 @@
     name: elastic+helm-charts+pull-request+integration-apm-server
     display-name: elastic / helm-charts - pull-request - integration apm-server
     description: Pull request - integration apm-server
+    concurrent: true
     scm:
     - git:
         refspec: +refs/pull/*:refs/remotes/origin/pr/*

.ci/jobs/elastic+helm-charts+pull-request+integration-elasticsearch.yml

@@ -3,6 +3,7 @@
     name: elastic+helm-charts+pull-request+integration-elasticsearch
     display-name: elastic / helm-charts - pull-request - integration elasticsearch
     description: Pull request - integration elasticsearch
+    concurrent: true
     scm:
     - git:
         refspec: +refs/pull/*:refs/remotes/origin/pr/*

.ci/jobs/elastic+helm-charts+pull-request+integration-filebeat.yml

@@ -3,6 +3,7 @@
     name: elastic+helm-charts+pull-request+integration-filebeat
     display-name: elastic / helm-charts - pull-request - integration filebeat
     description: Pull request - integration filebeat
+    concurrent: true
     scm:
     - git:
         refspec: +refs/pull/*:refs/remotes/origin/pr/*

.ci/jobs/elastic+helm-charts+pull-request+integration-kibana.yml

@@ -3,6 +3,7 @@
     name: elastic+helm-charts+pull-request+integration-kibana
     display-name: elastic / helm-charts - pull-request - integration kibana
     description: Pull request - integration kibana
+    concurrent: true
     scm:
     - git:
         refspec: +refs/pull/*:refs/remotes/origin/pr/*

.ci/jobs/elastic+helm-charts+pull-request+integration-logstash.yml

@@ -3,6 +3,7 @@
     name: elastic+helm-charts+pull-request+integration-logstash
     display-name: elastic / helm-charts - pull-request - integration logstash
     description: Pull request - integration logstash
+    concurrent: true
     scm:
     - git:
         refspec: +refs/pull/*:refs/remotes/origin/pr/*

.ci/jobs/elastic+helm-charts+pull-request+integration-metricbeat.yml

@@ -3,6 +3,7 @@
     name: elastic+helm-charts+pull-request+integration-metricbeat
     display-name: elastic / helm-charts - pull-request - integration metricbeat
     description: Pull request - integration metricbeat
+    concurrent: true
     scm:
     - git:
         refspec: +refs/pull/*:refs/remotes/origin/pr/*

.ci/jobs/elastic+helm-charts+pull-request+lint-python.yml

@@ -3,6 +3,7 @@
     name: elastic+helm-charts+pull-request+lint-python
     display-name: elastic / helm-charts - pull-request - lint python
     description: Pull request - lint python
+    concurrent: true
     scm:
     - git:
         refspec: +refs/pull/*:refs/remotes/origin/pr/*

.ci/jobs/elastic+helm-charts+pull-request+template-testing.yml

@@ -3,6 +3,7 @@
     name: elastic+helm-charts+pull-request+template-testing
     display-name: elastic / helm-charts - pull-request - template testing
     description: Pull request - template testing
+    concurrent: true
     scm:
     - git:
         refspec: +refs/pull/*:refs/remotes/origin/pr/*

.github/ISSUE_TEMPLATE/bug_report.md

@@ -18,7 +18,7 @@ e.g. `helm get elasticsearch` (replace `elasticsearch` with the name of your hel
 
 *Be careful to obfuscate every secrets (credentials, token, public IP, ...) that could be visible in the output before copy-pasting.*
 
-*If you find some secrets in plain text in `helm get release` output you should use [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) to managed them is a secure way (see [Security Example](https://github.com/elastic/helm-charts/blob/master/elasticsearch/examples/security/security.yml#L23-L38)).*
+*If you find some secrets in plain text in `helm get release` output you should use [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) to managed them is a secure way (see [Security Example](https://github.com/elastic/helm-charts/blob/master/elasticsearch/examples/security/values.yaml#L23-L38)).*
 
 <details>
 <summary>Output of helm get release</summary>

BREAKING_CHANGES.md

@@ -3,6 +3,11 @@
 <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
 
 
+  - [7.10.0](#7100)
+    - [Migration to Helm 3](#migration-to-helm-3)
+      - [Metricbeat upgrade](#metricbeat-upgrade)
+      - [Elasticsearch upgrade with persistence.labels.enabled](#elasticsearch-upgrade-with-persistencelabelsenabled)
+      - [Rendered manifests contain a resource that already exists error](#rendered-manifests-contain-a-resource-that-already-exists-error)
   - [7.9.3 - 2020/10/22](#793---20201022)
     - [Fix Logstash headless Service (end)](#fix-logstash-headless-service-end)
   - [6.8.13 - 2020/10/22](#6813---20201022)
@@ -39,6 +44,87 @@
 <!-- Use this to update TOC: -->
 <!-- docker run --rm -it -v $(pwd):/usr/src jorgeandrada/doctoc --github -->
 
+## 7.10.0
+
+### Migration to Helm 3
+
+Starting from the 7.10.0 release, Helm 3 is fully supported in Elastic Helm charts
+and Helm 2 is deprecated.
+
+In most cases, [Helm 2to3][] can be used to migrate from previous charts
+releases deployed with Helm 2:
+
+```shell
+# Install Helm 3
+
+# Install 2to3 plugin
+helm plugin install https://github.com/helm/helm-2to3.git
+
+# Migrate Helm 2 local config
+helm3 2to3 move config
+
+# Migrate Helm 2 releases
+helm3 2to3 convert <release-name>
+
+# Upgrade to 7.10.0
+helm upgrade <release-name> elastic/<chart-name> --version 7.10.0
+
+# Cleanup Helm 2 datas
+helm3 2to3 cleanup
+```
+
+Migration to Helm 3 with 7.10.0 charts release should work smoothly for the
+following charts.
+
+- apm-server >= 7.6.0
+- elasticsearch >= 7.4.0 (except when `persistence.labels.enabled` is true)
+- filebeat >= 7.9.0
+- kibana >= 7.4.0
+- logstash >= 7.9.0
+
+#### Metricbeat upgrade
+
+Metricbeat 7.10.0 introduce a breaking change in [#516][] to make it compatible
+with Helm 3.
+
+The removing of some `heritage` labels in Metricbeat deployment make upgrade
+fail with the following error:
+
+```
+UPGRADE FAILED
+Error: Deployment.apps "mb-metricbeat-metrics" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app":"mb-metricbeat-metrics", "release":"mb"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
+Error: UPGRADE FAILED: Deployment.apps "mb-metricbeat-metrics" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app":"mb-metricbeat-metrics", "release":"mb"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
+```
+
+Unfortunately using `helm upgrade --force` with Helm 3 is not enough. This chart
+will need to be uninstalled and re-installed.
+
+#### Elasticsearch upgrade with persistence.labels.enabled
+
+If you are using `persistence.labels.enabled=true` with Elasticsearch, upgrade
+will fail even with `--force`.
+
+You'll need to deploy a new release with the same `clusterName` using Helm 3,
+migrate your data, then remove the old release.
+
+#### Rendered manifests contain a resource that already exists error
+
+We experimented some `rendered manifests contain a resource that already exists`
+errors with some charts upgrade, mostly for charts deploying `ClusterRole` and
+`ClusterRoleBinding`resources.
+
+Helm 3 is automatically adding some new annotations (`meta.helm.sh/release-name`
+and `meta.helm.sh/release-namespace`) and labels (`app.kubernetes.io/managed-by`)
+to the charts resources during upgrade. However it sometimes fail to add the annotations with the below error:
+
+```
+Error: UPGRADE FAILED: rendered manifests contain a resource that already exists. Unable to continue with update: ClusterRole "apm-apm-server-cluster-role" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "apm"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "default"
+```
+
+The workaround is to manually add these annotations and labels to the existing failing resources
+using `kubectl edit` for example, then relaunch the upgrade command.
+
+
 ## 7.9.3 - 2020/10/22
 
 ### Fix Logstash headless Service (end)
@@ -57,7 +143,7 @@ See [7.9.3 - 2020/10/22](#793---20201022) and [7.9.1 - 2020/09/03](#791---202009
 ### Fix Logstash headless Service
 
 [#776][] fixed an issue with headless `Service` when using `extraPorts` value
-(see [Add headless Service for StatefulSet](#add-headless-service-for-statefulset) 
+(see [Add headless Service for StatefulSet](#add-headless-service-for-statefulset)
 for more details). Unfortunately, it introduced a new bug when using a `NodePort`
 `Service` ([#807][]). This is fixed by [#839][] in 7.9.3 (and 6.8.13).
 
@@ -290,6 +376,7 @@ volumeClaimTemplate:
 [#352]: https://github.com/elastic/helm-charts/pull/352
 [#437]: https://github.com/elastic/helm-charts/pull/437
 [#458]: https://github.com/elastic/helm-charts/pull/458
+[#516]: https://github.com/elastic/helm/pull/516
 [#540]: https://github.com/elastic/helm-charts/pull/540
 [#568]: https://github.com/elastic/helm-charts/pull/568
 [#572]: https://github.com/elastic/helm-charts/pull/572
@@ -307,6 +394,7 @@ volumeClaimTemplate:
 [elastic elasticsearch chart]: https://github.com/elastic/helm-charts/tree/master/elasticsearch
 [elastic helm repo]: https://helm.elastic.co
 [github releases]: https://github.com/elastic/helm-charts/releases
+[helm 2to3]: https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/
 [migration guide]: https://github.com/elastic/helm-charts/blob/master/elasticsearch/examples/migration/README.md
 [new branching model]: https://github.com/elastic/helm-charts/blob/master/CONTRIBUTING.md#branching
 [kube-state-metrics]: https://github.com/helm/charts/tree/master/stable/kube-state-metrics

CHANGELOG.md

@@ -1,5 +1,29 @@
 # Changelog
 
+## 7.10.0
+
+* 7.10.0 as default version.
+
+
+| PR | Author | Title |
+| --- | --- | --- |
+| [#927](https://github.com/elastic/helm-charts/pull/927) | [@jmlrt](https://github.com/jmlrt) | [meta] Remove version from dev install section title  |
+| [#922](https://github.com/elastic/helm-charts/pull/922) | [@jmlrt](https://github.com/jmlrt) | [meta] Remove support for k8s <1.14 & helm <2.17.0 (#916) [7.10]  |
+| [#919](https://github.com/elastic/helm-charts/pull/919) | [@jmlrt](https://github.com/jmlrt) | [meta] Upgrade test (#907) [7.10]  |
+| [#914](https://github.com/elastic/helm-charts/pull/914) | [@jmlrt](https://github.com/jmlrt) | [meta] Initiate 7.10 branch  |
+| [#897](https://github.com/elastic/helm-charts/pull/897) | [@cospeedster](https://github.com/cospeedster) | [elasticsearch] Fix spelling  |
+| [#911](https://github.com/elastic/helm-charts/pull/911) | [@jmlrt](https://github.com/jmlrt) | [elasticsearch] Update test hook annotations  |
+| [#910](https://github.com/elastic/helm-charts/pull/910) | [@jmlrt](https://github.com/jmlrt) | [meta] Add link to eck chart doc  |
+| [#902](https://github.com/elastic/helm-charts/pull/902) | [@jmlrt](https://github.com/jmlrt) | [meta] Helm 3 (#516)  |
+| [#891](https://github.com/elastic/helm-charts/pull/891) | [@jmlrt](https://github.com/jmlrt) | [meta] Increase helm timeout  |
+| [#890](https://github.com/elastic/helm-charts/pull/890) | [@jmlrt](https://github.com/jmlrt) | [meta] Update rbac.authorization.k8s.io api  |
+| [#887](https://github.com/elastic/helm-charts/pull/887) | [@nkammah](https://github.com/nkammah) | [meta] Add warning comment placeholder (7.x branch)  |
+| [#882](https://github.com/elastic/helm-charts/pull/882) | [@jmlrt](https://github.com/jmlrt) | [metricbeat] Use relocated stable repo for kube-state-metrics  |
+| [#880](https://github.com/elastic/helm-charts/pull/880) | [@jmlrt](https://github.com/jmlrt) | [meta] Add support for helm 2.17.0 and k8s 1.17  |
+| [#854](https://github.com/elastic/helm-charts/pull/854) | [@jmlrt](https://github.com/jmlrt) | [elasticsearch] Add coordinator node to multi test  |
+| [#860](https://github.com/elastic/helm-charts/pull/860) | [@nkammah](https://github.com/nkammah) | [meta] Simplify doc in 7.x branch  |
+
+
 ## 7.9.3
 
 * 7.9.3 as default version.

CONTRIBUTING.md

@@ -195,8 +195,6 @@ Charts are released from version branchs (example `7.7` branch).
 
 [Elastic Helm repository][] is updated only during releases.
 
-The current release process is documented in [release.md][].
-
 
 ## Testing
 
@@ -267,6 +265,5 @@ make goss
 [serverspec]: https://serverspec.org
 [templating test example]: https://github.com/elastic/helm-charts/blob/master/elasticsearch/tests/elasticsearch_test.py
 [templating tests section]: #templating-tests
-[release.md]: https://github.com/elastic/helm-charts/blob/master/helpers/release.md
 [releases section]: #releases
 [requirements.txt]: https://github.com/elastic/helm-charts/blob/master/requirements.txt

README.md

@@ -11,7 +11,6 @@
   - [Support Matrix](#support-matrix)
   - [Kubernetes Versions](#kubernetes-versions)
   - [Helm versions](#helm-versions)
-    - [Helm 3 beta](#helm-3-beta)
 - [ECK](#eck)
 
 <!-- END doctoc generated TOC please keep comment here to allow auto update -->
@@ -36,14 +35,14 @@ The `master` version of these charts are intended to support the latest pre-rele
 versions of our products, and therefore may or may not work with current released
 versions.
 
-| Chart                                      | Docker documentation                                                            |Latest 7 Version|Latest 6 Version|
-|--------------------------------------------|---------------------------------------------------------------------------------|-----------|-----------|
-| [APM-Server](./apm-server/README.md)       | https://www.elastic.co/guide/en/apm/server/current/running-on-docker.html       |[`7.9.3`][apm-7] |[`6.8.13`][apm-6] |
-| [Elasticsearch](./elasticsearch/README.md) | https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html     |[`7.9.3`][elasticsearch-7] |[`6.8.13`][elasticsearch-6] |
-| [Filebeat](./filebeat/README.md)           | https://www.elastic.co/guide/en/beats/filebeat/current/running-on-docker.html   |[`7.9.3`][filebeat-7] |[`6.8.13`][filebeat-6] |
-| [Kibana](./kibana/README.md)               | https://www.elastic.co/guide/en/kibana/current/docker.html                      |[`7.9.3`][kibana-7] |[`6.8.13`][kibana-6] |
-| [Logstash](./logstash/README.md)           | https://www.elastic.co/guide/en/logstash/current/docker.html                    |[`7.9.3`][logstash-7] |[`6.8.13`][logstash-6] |
-| [Metricbeat](./metricbeat/README.md)       | https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-docker.html |[`7.9.3`][metricbeat-7] |[`6.8.13`][metricbeat-6] |
+| Chart                                      | Docker documentation                                                            | Latest 7 Version           | Latest 6 Version            |
+|--------------------------------------------|---------------------------------------------------------------------------------|----------------------------|-----------------------------|
+| [APM-Server](./apm-server/README.md)       | https://www.elastic.co/guide/en/apm/server/current/running-on-docker.html       | [`7.10.0`][apm-7]           | [`6.8.13`][apm-6]           |
+| [Elasticsearch](./elasticsearch/README.md) | https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html     | [`7.10.0`][elasticsearch-7] | [`6.8.13`][elasticsearch-6] |
+| [Filebeat](./filebeat/README.md)           | https://www.elastic.co/guide/en/beats/filebeat/current/running-on-docker.html   | [`7.10.0`][filebeat-7]      | [`6.8.13`][filebeat-6]      |
+| [Kibana](./kibana/README.md)               | https://www.elastic.co/guide/en/kibana/current/docker.html                      | [`7.10.0`][kibana-7]        | [`6.8.13`][kibana-6]        |
+| [Logstash](./logstash/README.md)           | https://www.elastic.co/guide/en/logstash/current/docker.html                    | [`7.10.0`][logstash-7]      | [`6.8.13`][logstash-6]      |
+| [Metricbeat](./metricbeat/README.md)       | https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-docker.html | [`7.10.0`][metricbeat-7]    | [`6.8.13`][metricbeat-6]    |
 
 ## Supported Configurations
 
@@ -77,40 +76,37 @@ exact versions are defined under `KUBERNETES_VERSIONS` in
 ### Helm versions
 
 While we are checking backward compatibility, the charts are only tested with
-Helm version mentioned in [helm-tester Dockerfile][] (currently 2.16.12).
+Helm version mentioned in [helm-tester Dockerfile][] (currently 3.4.1).
 
-#### Helm 3 beta
-
-While we don't have automated tests for [Helm 3][] yet, we fixed the main
-blockers to use it. We now have enough feedbacks from internal and external
-users to add support in beta.
 
 ## ECK
 
 In addition of these Helm charts, Elastic also provides
 [Elastic Cloud on Kubernetes][] which is based on [Operator pattern][] and is
 Elastic recommended way to deploy Elasticsearch, Kibana and APM Server on
-Kubernetes.
+Kubernetes. There is a dedicated Helm chart for ECK which can be found
+[in ECK repo][eck-chart] ([documentation][eck-chart-doc]).
 
 
 [currently tested]: https://devops-ci.elastic.co/job/elastic+helm-charts+master/
+[eck-chart]: https://github.com/elastic/cloud-on-k8s/tree/master/deploy
+[eck-chart-doc]: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-install-helm.html
 [elastic cloud on kubernetes]: https://github.com/elastic/cloud-on-k8s
 [elastic helm repo]: https://helm.elastic.co
 [github releases]: https://github.com/elastic/helm-charts/releases
-[helm 3]: https://v3.helm.sh
 [helm-tester Dockerfile]: https://github.com/elastic/helm-charts/blob/master/helpers/helm-tester/Dockerfile
 [helpers/matrix.yml]: https://github.com/elastic/helm-charts/blob/master/helpers/matrix.yml
 [operator pattern]: https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
 [elasticsearch-771]: https://github.com/elastic/helm-charts/tree/7.7.1/elasticsearch/
-[apm-7]: https://github.com/elastic/helm-charts/tree/7.9.3/apm-server/README.md
+[apm-7]: https://github.com/elastic/helm-charts/tree/7.10.0/apm-server/README.md
 [apm-6]: https://github.com/elastic/helm-charts/tree/6.8.13/apm-server/README.md
-[elasticsearch-7]: https://github.com/elastic/helm-charts/tree/7.9.3/elasticsearch/README.md
+[elasticsearch-7]: https://github.com/elastic/helm-charts/tree/7.10.0/elasticsearch/README.md
 [elasticsearch-6]: https://github.com/elastic/helm-charts/tree/6.8.13/elasticsearch/README.md
-[filebeat-7]: https://github.com/elastic/helm-charts/tree/7.9.3/filebeat/README.md
+[filebeat-7]: https://github.com/elastic/helm-charts/tree/7.10.0/filebeat/README.md
 [filebeat-6]: https://github.com/elastic/helm-charts/tree/6.8.13/filebeat/README.md
-[kibana-7]: https://github.com/elastic/helm-charts/tree/7.9.3/kibana/README.md
+[kibana-7]: https://github.com/elastic/helm-charts/tree/7.10.0/kibana/README.md
 [kibana-6]: https://github.com/elastic/helm-charts/tree/6.8.13/kibana/README.md
-[logstash-7]: https://github.com/elastic/helm-charts/tree/7.9.3/logstash/README.md
+[logstash-7]: https://github.com/elastic/helm-charts/tree/7.10.0/logstash/README.md
 [logstash-6]: https://github.com/elastic/helm-charts/tree/6.8.13/logstash/README.md
-[metricbeat-7]: https://github.com/elastic/helm-charts/tree/7.9.3/metricbeat/README.md
+[metricbeat-7]: https://github.com/elastic/helm-charts/tree/7.10.0/metricbeat/README.md
 [metricbeat-6]: https://github.com/elastic/helm-charts/tree/6.8.13/metricbeat/README.md