From 7dc34ef86876b835ae0b679e436a45555ae88d66 Mon Sep 17 00:00:00 2001 From: azro352 Date: Thu, 31 Aug 2023 22:49:07 +0200 Subject: [PATCH 01/16] Add logstash helm chart skeleton --- .../eck-stack/charts/eck-logstash/.helmignore | 24 +++++ .../eck-stack/charts/eck-logstash/Chart.yaml | 10 ++ deploy/eck-stack/charts/eck-logstash/LICENSE | 93 +++++++++++++++++++ .../charts/eck-logstash/templates/NOTES.txt | 6 ++ .../eck-logstash/templates/_helpers.tpl | 51 ++++++++++ .../eck-logstash/templates/logstash.yaml | 33 +++++++ .../eck-stack/charts/eck-logstash/values.yaml | 58 ++++++++++++ 7 files changed, 275 insertions(+) create mode 100644 deploy/eck-stack/charts/eck-logstash/.helmignore create mode 100644 deploy/eck-stack/charts/eck-logstash/Chart.yaml create mode 100644 deploy/eck-stack/charts/eck-logstash/LICENSE create mode 100644 deploy/eck-stack/charts/eck-logstash/templates/NOTES.txt create mode 100644 deploy/eck-stack/charts/eck-logstash/templates/_helpers.tpl create mode 100644 deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml create mode 100644 deploy/eck-stack/charts/eck-logstash/values.yaml diff --git a/deploy/eck-stack/charts/eck-logstash/.helmignore b/deploy/eck-stack/charts/eck-logstash/.helmignore new file mode 100644 index 0000000000..f1568daf25 --- /dev/null +++ b/deploy/eck-stack/charts/eck-logstash/.helmignore @@ -0,0 +1,24 @@ +# Patterns to ignore when building packages. +# This supports shell glob matching, relative path matching, and +# negation (prefixed with !). Only one pattern per line. +.DS_Store +# Common VCS dirs +.git/ +.gitignore +.bzr/ +.bzrignore +.hg/ +.hgignore +.svn/ +# Common backup files +*.swp +*.bak +*.tmp +*.orig +*~ +# Various IDEs +.project +.idea/ +*.tmproj +.vscode/ +templates/tests diff --git a/deploy/eck-stack/charts/eck-logstash/Chart.yaml b/deploy/eck-stack/charts/eck-logstash/Chart.yaml new file mode 100644 index 0000000000..b7143c9750 --- /dev/null +++ b/deploy/eck-stack/charts/eck-logstash/Chart.yaml @@ -0,0 +1,10 @@ +apiVersion: v2 +name: eck-logstash +description: Logstash managed by the ECK operator +kubeVersion: ">= 1.21.0-0" +type: application +version: 0.8.0-SNAPSHOT +sources: + - https://github.com/elastic/cloud-on-k8s + - https://github.com/elastic/logstash +icon: https://helm.elastic.co/icons/logstash.png diff --git a/deploy/eck-stack/charts/eck-logstash/LICENSE b/deploy/eck-stack/charts/eck-logstash/LICENSE new file mode 100644 index 0000000000..92503a7217 --- /dev/null +++ b/deploy/eck-stack/charts/eck-logstash/LICENSE @@ -0,0 +1,93 @@ +Elastic License 2.0 + +URL: https://www.elastic.co/licensing/elastic-license + +## Acceptance + +By using the software, you agree to all of the terms and conditions below. + +## Copyright License + +The licensor grants you a non-exclusive, royalty-free, worldwide, +non-sublicensable, non-transferable license to use, copy, distribute, make +available, and prepare derivative works of the software, in each case subject to +the limitations and conditions below. + +## Limitations + +You may not provide the software to third parties as a hosted or managed +service, where the service provides users with access to any substantial set of +the features or functionality of the software. + +You may not move, change, disable, or circumvent the license key functionality +in the software, and you may not remove or obscure any functionality in the +software that is protected by the license key. + +You may not alter, remove, or obscure any licensing, copyright, or other notices +of the licensor in the software. Any use of the licensor’s trademarks is subject +to applicable law. + +## Patents + +The licensor grants you a license, under any patent claims the licensor can +license, or becomes able to license, to make, have made, use, sell, offer for +sale, import and have imported the software, in each case subject to the +limitations and conditions in this license. This license does not cover any +patent claims that you cause to be infringed by modifications or additions to +the software. If you or your company make any written claim that the software +infringes or contributes to infringement of any patent, your patent license for +the software granted under these terms ends immediately. If your company makes +such a claim, your patent license ends immediately for work on behalf of your +company. + +## Notices + +You must ensure that anyone who gets a copy of any part of the software from you +also gets a copy of these terms. + +If you modify the software, you must include in any modified copies of the +software prominent notices stating that you have modified the software. + +## No Other Rights + +These terms do not imply any licenses other than those expressly granted in +these terms. + +## Termination + +If you use the software in violation of these terms, such use is not licensed, +and your licenses will automatically terminate. If the licensor provides you +with a notice of your violation, and you cease all violation of this license no +later than 30 days after you receive that notice, your licenses will be +reinstated retroactively. However, if you violate these terms after such +reinstatement, any additional violation of these terms will cause your licenses +to terminate automatically and permanently. + +## No Liability + +*As far as the law allows, the software comes as is, without any warranty or +condition, and the licensor will not be liable to you for any damages arising +out of these terms or the use or nature of the software, under any kind of +legal claim.* + +## Definitions + +The **licensor** is the entity offering these terms, and the **software** is the +software the licensor makes available under these terms, including any portion +of it. + +**you** refers to the individual or entity agreeing to these terms. + +**your company** is any legal entity, sole proprietorship, or other kind of +organization that you work for, plus all organizations that have control over, +are under the control of, or are under common control with that +organization. **control** means ownership of substantially all the assets of an +entity, or the power to direct its management and policies by vote, contract, or +otherwise. Control can be direct or indirect. + +**your licenses** are all the licenses granted to you for the software under +these terms. + +**use** means anything you do with the software requiring one of your licenses. + +**trademark** means trademarks, service marks, and similar rights. \ No newline at end of file diff --git a/deploy/eck-stack/charts/eck-logstash/templates/NOTES.txt b/deploy/eck-stack/charts/eck-logstash/templates/NOTES.txt new file mode 100644 index 0000000000..c2f255af99 --- /dev/null +++ b/deploy/eck-stack/charts/eck-logstash/templates/NOTES.txt @@ -0,0 +1,6 @@ + +1. Check Logstash status + $ kubectl get logstash {{ include "logstash.fullname" . }} -n {{ .Release.Namespace }} + +2. Check Logstash pod status + $ kubectl get pods --namespace={{ .Release.Namespace }} -l logstash.k8s.elastic.co/name={{ include "logstash.fullname" . }} diff --git a/deploy/eck-stack/charts/eck-logstash/templates/_helpers.tpl b/deploy/eck-stack/charts/eck-logstash/templates/_helpers.tpl new file mode 100644 index 0000000000..7efd669fdb --- /dev/null +++ b/deploy/eck-stack/charts/eck-logstash/templates/_helpers.tpl @@ -0,0 +1,51 @@ +{{/* +Expand the name of the chart. +*/}} +{{- define "logstash.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Create a default fully qualified app name. +We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). +If release name contains chart name it will be used as a full name. +*/}} +{{- define "logstash.fullname" -}} +{{- if .Values.fullnameOverride }} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- $name := default .Chart.Name .Values.nameOverride }} +{{- if contains $name .Release.Name }} +{{- .Release.Name | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} +{{- end }} +{{- end }} +{{- end }} + +{{/* +Create chart name and version as used by the chart label. +*/}} +{{- define "logstash.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Common labels +*/}} +{{- define "logstash.labels" -}} +helm.sh/chart: {{ include "logstash.chart" . }} +{{ include "logstash.selectorLabels" . }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- if .Values.labels }} +{{ toYaml .Values.labels }} +{{- end }} +{{- end }} + +{{/* +Selector labels +*/}} +{{- define "logstash.selectorLabels" -}} +app.kubernetes.io/name: {{ include "logstash.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} diff --git a/deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml b/deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml new file mode 100644 index 0000000000..0d25724241 --- /dev/null +++ b/deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml @@ -0,0 +1,33 @@ +--- +apiVersion: logstash.k8s.elastic.co/v1 +kind: Logstash +metadata: + name: {{ include "logstash.fullname" . }} + labels: + {{- include "logstash.labels" . | nindent 4 }} + annotations: + eck.k8s.elastic.co/license: enterprise + {{- with .Values.annotations }} + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + version: {{ required "A Logstash version is required" .Values.version }} + count: {{ required "An amount of pod is required" .Values.count }} + elasticsearchRefs: + {{ toYaml .Values.elasticsearchRefs | nindent 4 }} + {{- with .Values.config }} + config: + {{- toYaml . | nindent 4 }} + {{- end }} + pipelines: + {{ toYaml .Values.pipelines | nindent 4 }} + volumeClaimTemplates: + {{ toYaml .Values.volumeClaimTemplates | nindent 4 }} + {{- with .Values.podTemplate }} + podTemplate: + {{- toYaml . | nindent 4 }} + {{- end }} + {{- with .Values.monitoring }} + monitoring: + {{- toYaml . | nindent 4 }} + {{- end }} diff --git a/deploy/eck-stack/charts/eck-logstash/values.yaml b/deploy/eck-stack/charts/eck-logstash/values.yaml new file mode 100644 index 0000000000..306b4f0d46 --- /dev/null +++ b/deploy/eck-stack/charts/eck-logstash/values.yaml @@ -0,0 +1,58 @@ +--- +# Default values for eck-logstash. +# This is a YAML-formatted file. + +# Overridable names of the Logstash resource. +# By default, this is the Release name set for the chart, +# followed by 'eck-logstash'. +# +# nameOverride will override the name of the Chart with the name set here, +# so nameOverride: quickstart, would convert to '{{ Release.name }}-quickstart' +# +# nameOverride: "quickstart" +# +# fullnameOverride will override both the release name, and the chart name, +# and will name the Logstash resource exactly as specified. +# +# fullnameOverride: "quickstart" + +# Version of Logstash. +# +version: 8.10.0-SNAPSHOT + +# Labels that will be applied to Logstash. +# +labels: {} + +# Annotations that will be applied to Logstash. +# +annotations: {} + +# Controlling the number of pods. +# ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-logstash-scaling-logstash.html +# +count: 1 + +elasticsearchRefs: [] + +# The logstash configuration, the ECK equivalent to logstash.yml +# ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-logstash-configuration.html#k8s-logstash-configuring-logstash +# +config: {} + +# The logstash pipelines, the ECK equivalent to pipelines.yml +# ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-logstash-configuration.html#k8s-logstash-pipelines +# +pipelines: [] + +# volumeClaimTemplates +# ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-logstash-configuration.html#k8s-volume-claim-settings +# +volumeClaimTemplates: [] + +# Set podTemplate to customize the pod used by Logstash +# ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-customize-pods.html +# +podTemplate: {} + +monitoring: {} From 61d70b22e02df018146a006cf62879457a000800 Mon Sep 17 00:00:00 2001 From: azro352 Date: Thu, 31 Aug 2023 23:24:05 +0200 Subject: [PATCH 02/16] Add examples --- .../eck-logstash/examples/basic-eck.yaml | 43 +++++++++++++++++++ .../charts/eck-logstash/examples/es-role.yaml | 24 +++++++++++ .../eck-logstash/templates/logstash.yaml | 16 ++++--- .../eck-stack/charts/eck-logstash/values.yaml | 16 ++++--- 4 files changed, 86 insertions(+), 13 deletions(-) create mode 100644 deploy/eck-stack/charts/eck-logstash/examples/basic-eck.yaml create mode 100644 deploy/eck-stack/charts/eck-logstash/examples/es-role.yaml diff --git a/deploy/eck-stack/charts/eck-logstash/examples/basic-eck.yaml b/deploy/eck-stack/charts/eck-logstash/examples/basic-eck.yaml new file mode 100644 index 0000000000..b096d36349 --- /dev/null +++ b/deploy/eck-stack/charts/eck-logstash/examples/basic-eck.yaml @@ -0,0 +1,43 @@ +--- +version: 8.10.0-SNAPSHOT + +elasticsearchRefs: + - clusterName: eck + name: elasticsearch + +pipelines: + - pipeline.id: main + config.string: | + input { + beats { + port => 5044 + } + } + filter { + grok { + match => { "message" => "%{HTTPD_COMMONLOG}"} + } + geoip { + source => "[source][address]" + target => "[source]" + } + } + output { + elasticsearch { + hosts => [ "${ECK_ES_HOSTS}" ] + user => "${ECK_ES_USER}" + password => "${ECK_ES_PASSWORD}" + ssl_certificate_authorities => "${ECK_ES_SSL_CERTIFICATE_AUTHORITY}" + } + } + +services: + - name: beats + service: + spec: + type: ClusterIP + ports: + - port: 5044 + name: "filebeat" + protocol: TCP + targetPort: 5044 diff --git a/deploy/eck-stack/charts/eck-logstash/examples/es-role.yaml b/deploy/eck-stack/charts/eck-logstash/examples/es-role.yaml new file mode 100644 index 0000000000..971eb33dd8 --- /dev/null +++ b/deploy/eck-stack/charts/eck-logstash/examples/es-role.yaml @@ -0,0 +1,24 @@ +--- +version: 8.10.0-SNAPSHOT + +elasticsearchRefs: + - clusterName: eck + name: elasticsearch + +pipelines: + - pipeline.id: main + config.string: | + input { exec { command => "uptime" interval => 10 } } + output { + elasticsearch { + hosts => [ "${ECK_ES_HOSTS}" ] + ssl_enabled => true + ssl_certificate_authorities => "${ECK_ES_SSL_CERTIFICATE_AUTHORITY}" + user => "${ECK_ES_USER}" + password => "${ECK_ES_PASSWORD}" + index => "my-index" + data_stream => false + ilm_enabled => false + manage_template => false + } + } diff --git a/deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml b/deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml index 0d25724241..d8bc8f1863 100644 --- a/deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml +++ b/deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml @@ -13,16 +13,11 @@ metadata: spec: version: {{ required "A Logstash version is required" .Values.version }} count: {{ required "An amount of pod is required" .Values.count }} - elasticsearchRefs: - {{ toYaml .Values.elasticsearchRefs | nindent 4 }} + {{- with .Values.config }} config: {{- toYaml . | nindent 4 }} {{- end }} - pipelines: - {{ toYaml .Values.pipelines | nindent 4 }} - volumeClaimTemplates: - {{ toYaml .Values.volumeClaimTemplates | nindent 4 }} {{- with .Values.podTemplate }} podTemplate: {{- toYaml . | nindent 4 }} @@ -31,3 +26,12 @@ spec: monitoring: {{- toYaml . | nindent 4 }} {{- end }} + + pipelines: + {{ toYaml .Values.pipelines | nindent 4 }} + volumeClaimTemplates: + {{ toYaml .Values.volumeClaimTemplates | nindent 4 }} + elasticsearchRefs: + {{ toYaml .Values.elasticsearchRefs | nindent 4 }} + services: + {{ toYaml .Values.services | nindent 4 }} diff --git a/deploy/eck-stack/charts/eck-logstash/values.yaml b/deploy/eck-stack/charts/eck-logstash/values.yaml index 306b4f0d46..93ce8ee805 100644 --- a/deploy/eck-stack/charts/eck-logstash/values.yaml +++ b/deploy/eck-stack/charts/eck-logstash/values.yaml @@ -33,13 +33,18 @@ annotations: {} # count: 1 -elasticsearchRefs: [] - # The logstash configuration, the ECK equivalent to logstash.yml # ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-logstash-configuration.html#k8s-logstash-configuring-logstash # config: {} +# Set podTemplate to customize the pod used by Logstash +# ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-customize-pods.html +# +podTemplate: {} + +monitoring: {} + # The logstash pipelines, the ECK equivalent to pipelines.yml # ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-logstash-configuration.html#k8s-logstash-pipelines # @@ -50,9 +55,6 @@ pipelines: [] # volumeClaimTemplates: [] -# Set podTemplate to customize the pod used by Logstash -# ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-customize-pods.html -# -podTemplate: {} +elasticsearchRefs: [] -monitoring: {} +services: [] From 970e887cecd89de8943310686e87b83c0ef0c090 Mon Sep 17 00:00:00 2001 From: azro352 Date: Fri, 1 Sep 2023 08:14:55 +0200 Subject: [PATCH 03/16] Add examples and links to matching recipes --- .../eck-logstash/examples/basic-eck.yaml | 1 + .../charts/eck-logstash/examples/es-role.yaml | 1 + .../eck-logstash/examples/monitored.yaml | 49 ++++++++ .../charts/eck-logstash/examples/multi.yaml | 78 +++++++++++++ .../charts/eck-logstash/examples/volumes.yaml | 107 ++++++++++++++++++ 5 files changed, 236 insertions(+) create mode 100644 deploy/eck-stack/charts/eck-logstash/examples/monitored.yaml create mode 100644 deploy/eck-stack/charts/eck-logstash/examples/multi.yaml create mode 100644 deploy/eck-stack/charts/eck-logstash/examples/volumes.yaml diff --git a/deploy/eck-stack/charts/eck-logstash/examples/basic-eck.yaml b/deploy/eck-stack/charts/eck-logstash/examples/basic-eck.yaml index b096d36349..6db1a1d601 100644 --- a/deploy/eck-stack/charts/eck-logstash/examples/basic-eck.yaml +++ b/deploy/eck-stack/charts/eck-logstash/examples/basic-eck.yaml @@ -1,4 +1,5 @@ --- +# values corresponding to config/recipes/logstash/logstash-eck.yaml version: 8.10.0-SNAPSHOT elasticsearchRefs: diff --git a/deploy/eck-stack/charts/eck-logstash/examples/es-role.yaml b/deploy/eck-stack/charts/eck-logstash/examples/es-role.yaml index 971eb33dd8..45a1f282a8 100644 --- a/deploy/eck-stack/charts/eck-logstash/examples/es-role.yaml +++ b/deploy/eck-stack/charts/eck-logstash/examples/es-role.yaml @@ -1,4 +1,5 @@ --- +# values corresponding to config/recipes/logstash/logstash-es-role.yaml version: 8.10.0-SNAPSHOT elasticsearchRefs: diff --git a/deploy/eck-stack/charts/eck-logstash/examples/monitored.yaml b/deploy/eck-stack/charts/eck-logstash/examples/monitored.yaml new file mode 100644 index 0000000000..9bbb061ad0 --- /dev/null +++ b/deploy/eck-stack/charts/eck-logstash/examples/monitored.yaml @@ -0,0 +1,49 @@ +--- +# values corresponding to config/recipes/logstash/logstash-monitored.yaml +version: 8.10.0-SNAPSHOT + +monitoring: + metrics: + elasticsearchRefs: + - name: elasticsearch-monitoring + +pipelines: + - pipeline.id: main + config.string: | + input { + beats { + port => 5044 + } + } + filter { + grok { + match => { "message" => "%{HTTPD_COMMONLOG}"} + } + geoip { + source => "[source][address]" + target => "[source]" + } + } + output { + elasticsearch { + hosts => [ "${ECK_ES_HOSTS}" ] + user => "${ECK_ES_USER}" + password => "${ECK_ES_PASSWORD}" + ssl_certificate_authorities => "${ECK_ES_SSL_CERTIFICATE_AUTHORITY}" + } + } + +elasticsearchRefs: + - clusterName: eck + name: elasticsearch + +services: + - name: beats + service: + spec: + type: ClusterIP + ports: + - port: 5044 + name: "filebeat" + protocol: TCP + targetPort: 5044 diff --git a/deploy/eck-stack/charts/eck-logstash/examples/multi.yaml b/deploy/eck-stack/charts/eck-logstash/examples/multi.yaml new file mode 100644 index 0000000000..2e40063e6d --- /dev/null +++ b/deploy/eck-stack/charts/eck-logstash/examples/multi.yaml @@ -0,0 +1,78 @@ +--- +# values corresponding to config/recipes/logstash/logstash-multi.yaml +version: 8.10.0-SNAPSHOT + +pipelines: + - pipeline.id: main + config.string: | + input { + beats { + port => 5044 + } + } + filter { + grok { + match => { "message" => "%{HTTPD_COMMONLOG}"} + } + geoip { + source => "[source][address]" + target => "[source]" + } + } + output { + pipeline { + send_to => 'prod' + } + pipeline { + send_to => 'qa' + } + } + - pipeline.id: production + config.string: | + input { + pipeline { + address => 'prod' + } + } + output { + elasticsearch { + hosts => [ "${PROD_ES_ES_HOSTS}" ] + user => "${PROD_ES_ES_USER}" + password => "${PROD_ES_ES_PASSWORD}" + ssl_certificate_authorities => "${PROD_ES_ES_SSL_CERTIFICATE_AUTHORITY}" + } + } + - pipeline.id: qa + config.string: | + input { + pipeline { + address => 'qa' + } + } + output { + elasticsearch { + hosts => [ "${QA_ES_ES_HOSTS}" ] + user => "${QA_ES_ES_USER}" + password => "${QA_ES_ES_PASSWORD}" + ssl_certificate_authorities => "${QA_ES_ES_SSL_CERTIFICATE_AUTHORITY}" + } + } + +elasticsearchRefs: + - clusterName: prod-es + name: production + - clusterName: qa-es + name: qa + namespace: qa + +services: + - name: beats + service: + spec: + type: ClusterIP + ports: + - port: 5044 + name: "filebeat" + protocol: TCP + targetPort: 5044 + diff --git a/deploy/eck-stack/charts/eck-logstash/examples/volumes.yaml b/deploy/eck-stack/charts/eck-logstash/examples/volumes.yaml new file mode 100644 index 0000000000..f4495ce12c --- /dev/null +++ b/deploy/eck-stack/charts/eck-logstash/examples/volumes.yaml @@ -0,0 +1,107 @@ +--- +# values corresponding to config/recipes/logstash/logstash-volumes.yaml +version: 8.10.0-SNAPSHOT + +config: + log.level: info + queue.type: persisted + path.queue: /usr/share/logstash/pq + +podTemplate: + spec: + containers: + - name: logstash + volumeMounts: + - mountPath: /usr/share/logstash/pq + name: pq + readOnly: false + - mountPath: /usr/share/logstash/dlq + name: dlq + readOnly: false + +pipelines: + - pipeline.id: dlq_read + dead_letter_queue.enable: false + config.string: | + input { + dead_letter_queue { + path => "/usr/share/logstash/dlq" + commit_offsets => true + pipeline_id => "beats" + clean_consumed => true + } + } + filter { + mutate { + remove_field => "[geoip][location]" + } + } + output { + elasticsearch { + hosts => [ "${ECK_ES_HOSTS}" ] + user => "${ECK_ES_USER}" + password => "${ECK_ES_PASSWORD}" + ssl_certificate_authorities => "${ECK_ES_SSL_CERTIFICATE_AUTHORITY}" + } + } + - pipeline.id: beats + dead_letter_queue.enable: true + path.dead_letter_queue: /usr/share/logstash/dlq + config.string: | + input { + beats { + port => 5044 + } + } + filter { + grok { + match => { "message" => "%{HTTPD_COMMONLOG}"} + } + geoip { + source => "[source][address]" + target => "[source]" + } + } + output { + elasticsearch { + hosts => [ "${ECK_ES_HOSTS}" ] + user => "${ECK_ES_USER}" + password => "${ECK_ES_PASSWORD}" + ssl_certificate_authorities => "${ECK_ES_SSL_CERTIFICATE_AUTHORITY}" + } + } + +volumeClaimTemplates: + - metadata: + name: pq + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + - metadata: + name: dlq + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 5Gi + + +elasticsearchRefs: + - clusterName: eck + name: elasticsearch + +services: + - name: beats + service: + spec: + type: ClusterIP + ports: + - port: 5044 + name: "filebeat" + protocol: TCP + targetPort: 5044 + From ff374fd0a71736d81c30a3210cc25b0904467451 Mon Sep 17 00:00:00 2001 From: Michael Montgomery Date: Mon, 18 Sep 2023 09:46:12 -0500 Subject: [PATCH 04/16] Update eck-stack chart to include logstash chart. Update eck-stack values to include logstash chart, disabled by default. Include logstash tests. Signed-off-by: Michael Montgomery --- deploy/eck-stack/Chart.yaml | 3 + .../templates/tests/logstash_test.yaml | 146 ++++++++++++++++++ deploy/eck-stack/values.yaml | 5 + 3 files changed, 154 insertions(+) create mode 100644 deploy/eck-stack/charts/eck-logstash/templates/tests/logstash_test.yaml diff --git a/deploy/eck-stack/Chart.yaml b/deploy/eck-stack/Chart.yaml index ede8ebe399..a7412c4a20 100644 --- a/deploy/eck-stack/Chart.yaml +++ b/deploy/eck-stack/Chart.yaml @@ -21,3 +21,6 @@ dependencies: - name: eck-beats condition: eck-beats.enabled version: "0.8.0-SNAPSHOT" + - name: eck-logstash + condition: eck-logstash.enabled + version: "0.8.0-SNAPSHOT" diff --git a/deploy/eck-stack/charts/eck-logstash/templates/tests/logstash_test.yaml b/deploy/eck-stack/charts/eck-logstash/templates/tests/logstash_test.yaml new file mode 100644 index 0000000000..59c9128a05 --- /dev/null +++ b/deploy/eck-stack/charts/eck-logstash/templates/tests/logstash_test.yaml @@ -0,0 +1,146 @@ +suite: test logstash +templates: + - templates/logstash.yaml +tests: + - it: should render quickstart properly + set: + version: 8.2.0 + release: + name: quickstart + asserts: + - isKind: + of: Logstash + - equal: + path: metadata.name + value: quickstart-eck-logstash + - equal: + path: spec.version + value: 8.2.0 + - it: name override should work properly + set: + nameOverride: override + release: + name: quickstart + asserts: + - isKind: + of: Logstash + - equal: + path: metadata.name + value: quickstart-override + - it: fullname override should work properly + set: + fullnameOverride: override + release: + name: quickstart + asserts: + - isKind: + of: Logstash + - equal: + path: metadata.name + value: override + - it: setting volumeClaimDeletePolicy should work properly + set: + volumeClaimTemplates: + - metadata: + name: pq + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + release: + name: quickstart + asserts: + - isKind: + of: Logstash + - equal: + path: spec.volumeClaimTemplates + value: + - metadata: + name: pq + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + - it: should render labels, annotations, and additional custom values properly + set: + labels: + test: label + annotations: + test: annotation + monitoring: + metrics: + elasticsearchRefs: + - name: monitoring + namespace: observability + logs: + elasticsearchRefs: + - name: monitoring + namespace: observability + services: + - name: beats + service: + spec: + type: ClusterIP + ports: + - port: 5044 + name: "filebeat" + protocol: TCP + targetPort: 5044 + release: + name: quickstart + asserts: + - isKind: + of: Logstash + - equal: + path: metadata.labels + value: + app.kubernetes.io/instance: quickstart + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/name: eck-logstash + helm.sh/chart: eck-logstash-0.8.0-SNAPSHOT + test: label + - equal: + path: metadata.annotations + value: + eck.k8s.elastic.co/license: enterprise + test: annotation + - equal: + path: spec.services + value: + - name: beats + service: + spec: + type: ClusterIP + ports: + - port: 5044 + name: "filebeat" + protocol: TCP + targetPort: 5044 + # Uncomment when image, and serviceAccountName have been added to values/templates. + # Also probably want to add checks for configref, and pipelinesRef when they are added. + # + # - it: should render image properly + # set: + # image: my.regis.try/logstash:8 + # asserts: + # - equal: + # path: spec.image + # value: my.regis.try/logstash:8 + # - it: should render image properly + # set: + # image: my.registry.com/elastic/logstash:8.9.0 + # asserts: + # - equal: + # path: spec.image + # value: my.registry.com/elastic/logstash:8.9.0 + # - it: should render serviceAccountName properly + # set: + # serviceAccountName: my-sa + # asserts: + # - equal: + # path: spec.serviceAccountName + # value: my-sa diff --git a/deploy/eck-stack/values.yaml b/deploy/eck-stack/values.yaml index d7753ac223..381566c632 100644 --- a/deploy/eck-stack/values.yaml +++ b/deploy/eck-stack/values.yaml @@ -34,3 +34,8 @@ eck-fleet-server: # eck-beats: enabled: false + +# If enabled, will use the eck-logstash chart and deploy a Logstash resource. +# +eck-logstash: + enabled: false From c81acd24de29b1b10dcad3a93dcd211fabfd287f Mon Sep 17 00:00:00 2001 From: azro352 <35503478+azro352@users.noreply.github.com> Date: Tue, 19 Sep 2023 20:14:42 +0200 Subject: [PATCH 05/16] Update deploy/eck-stack/charts/eck-logstash/values.yaml Co-authored-by: Michael Montgomery --- deploy/eck-stack/charts/eck-logstash/values.yaml | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/deploy/eck-stack/charts/eck-logstash/values.yaml b/deploy/eck-stack/charts/eck-logstash/values.yaml index 93ce8ee805..447c31c411 100644 --- a/deploy/eck-stack/charts/eck-logstash/values.yaml +++ b/deploy/eck-stack/charts/eck-logstash/values.yaml @@ -43,7 +43,18 @@ config: {} # podTemplate: {} +# Settings for configuring stack monitoring. +# ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-stack-monitoring.html +# monitoring: {} + # metrics: + # elasticsearchRefs: + # - name: monitoring + # namespace: observability + # logs: + # elasticsearchRefs: + # - name: monitoring + # namespace: observability # The logstash pipelines, the ECK equivalent to pipelines.yml # ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-logstash-configuration.html#k8s-logstash-pipelines From 181890408710ebc5344bd460553b995390338174 Mon Sep 17 00:00:00 2001 From: azro352 Date: Tue, 19 Sep 2023 21:16:24 +0200 Subject: [PATCH 06/16] Add image, serviceAccountName, revisionHistoryLimit, configRef, pipelineRef, secureSettings into the values, chart and tests --- .../eck-logstash/templates/logstash.yaml | 19 +++++ .../templates/tests/logstash_test.yaml | 70 ++++++++++++------- .../eck-stack/charts/eck-logstash/values.yaml | 31 ++++++++ 3 files changed, 96 insertions(+), 24 deletions(-) diff --git a/deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml b/deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml index d8bc8f1863..ce720911a7 100644 --- a/deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml +++ b/deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml @@ -13,11 +13,28 @@ metadata: spec: version: {{ required "A Logstash version is required" .Values.version }} count: {{ required "An amount of pod is required" .Values.count }} + {{- with .Values.image }} + image: {{ . }} + {{- end }} + {{- with .Values.serviceAccountName }} + serviceAccountName: {{ . }} + {{- end }} + {{- with .Values.revisionHistoryLimit }} + revisionHistoryLimit: {{ . }} + {{- end }} {{- with .Values.config }} config: {{- toYaml . | nindent 4 }} {{- end }} + {{- with .Values.configRef }} + configRef: + {{- toYaml . | nindent 4 }} + {{- end }} + {{- with .Values.pipelineRef }} + pipelineRef: + {{- toYaml . | nindent 4 }} + {{- end }} {{- with .Values.podTemplate }} podTemplate: {{- toYaml . | nindent 4 }} @@ -35,3 +52,5 @@ spec: {{ toYaml .Values.elasticsearchRefs | nindent 4 }} services: {{ toYaml .Values.services | nindent 4 }} + secureSettings: + {{ toYaml .Values.secureSettings | nindent 4 }} diff --git a/deploy/eck-stack/charts/eck-logstash/templates/tests/logstash_test.yaml b/deploy/eck-stack/charts/eck-logstash/templates/tests/logstash_test.yaml index 59c9128a05..f4fb4ad332 100644 --- a/deploy/eck-stack/charts/eck-logstash/templates/tests/logstash_test.yaml +++ b/deploy/eck-stack/charts/eck-logstash/templates/tests/logstash_test.yaml @@ -120,27 +120,49 @@ tests: name: "filebeat" protocol: TCP targetPort: 5044 - # Uncomment when image, and serviceAccountName have been added to values/templates. - # Also probably want to add checks for configref, and pipelinesRef when they are added. - # - # - it: should render image properly - # set: - # image: my.regis.try/logstash:8 - # asserts: - # - equal: - # path: spec.image - # value: my.regis.try/logstash:8 - # - it: should render image properly - # set: - # image: my.registry.com/elastic/logstash:8.9.0 - # asserts: - # - equal: - # path: spec.image - # value: my.registry.com/elastic/logstash:8.9.0 - # - it: should render serviceAccountName properly - # set: - # serviceAccountName: my-sa - # asserts: - # - equal: - # path: spec.serviceAccountName - # value: my-sa + - it: should render image properly + set: + image: my.regis.try/logstash:8 + asserts: + - equal: + path: spec.image + value: my.regis.try/logstash:8 + - it: should render image properly + set: + image: my.registry.com/elastic/logstash:8.9.0 + asserts: + - equal: + path: spec.image + value: my.registry.com/elastic/logstash:8.9.0 + - it: should render serviceAccountName properly + set: + serviceAccountName: my-sa + asserts: + - equal: + path: spec.serviceAccountName + value: my-sa + - it: should render revisionHistoryLimit properly + set: + revisionHistoryLimit: 5 + asserts: + - equal: + path: spec.revisionHistoryLimit + value: 5 + - it: should render configRef properly + set: + configRef: + secretRef: + secretName: configLogstashSecretName + asserts: + - equal: + path: spec.configRef.secretRef.secretName + value: configLogstashSecretName + - it: should render pipelineRef properly + set: + pipelineRef: + secretRef: + secretName: pipelineLogstashSecretName + asserts: + - equal: + path: spec.pipelineRef.secretRef.secretName + value: pipelineLogstashSecretName diff --git a/deploy/eck-stack/charts/eck-logstash/values.yaml b/deploy/eck-stack/charts/eck-logstash/values.yaml index 93ce8ee805..267c43469a 100644 --- a/deploy/eck-stack/charts/eck-logstash/values.yaml +++ b/deploy/eck-stack/charts/eck-logstash/values.yaml @@ -20,6 +20,15 @@ # version: 8.10.0-SNAPSHOT +# Logstash Docker image to deploy +# +# image: + +# Used to check access from the current resource to a resource (for ex. a remote Elasticsearch cluster) in a different namespace. +# Can only be used if ECK is enforcing RBAC on references. +# +# serviceAccountName: "" + # Labels that will be applied to Logstash. # labels: {} @@ -28,6 +37,11 @@ labels: {} # annotations: {} +# Number of revisions to retain to allow rollback in the underlying StatefulSets. +# By default, if not set, Kubernetes sets 10. +# +# revisionHistoryLimit: 2 + # Controlling the number of pods. # ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-logstash-scaling-logstash.html # @@ -38,6 +52,10 @@ count: 1 # config: {} +configRef: {} +# secretRef: +# secretName: '' + # Set podTemplate to customize the pod used by Logstash # ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-customize-pods.html # @@ -50,11 +68,24 @@ monitoring: {} # pipelines: [] +pipelineRef: {} +# secretRef: +# secretName: '' + + # volumeClaimTemplates # ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-logstash-configuration.html#k8s-volume-claim-settings # volumeClaimTemplates: [] +# ElasticsearchRefs are references to Elasticsearch clusters running in the same Kubernetes cluster. elasticsearchRefs: [] +# - namespace: '' +# name: '' +# serviceName: '' +# secretName: '' services: [] + +# SecureSettings is a list of references to Kubernetes Secrets containing sensitive configuration options for the Logstash +secureSettings: [] From 0f127b7bc9f2227f56b2583fba2d57a4c15634ca Mon Sep 17 00:00:00 2001 From: Michael Montgomery Date: Wed, 11 Oct 2023 18:44:07 -0500 Subject: [PATCH 07/16] Update apiversion to v1alpha1 Signed-off-by: Michael Montgomery --- .../charts/eck-logstash/templates/logstash.yaml | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml b/deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml index ce720911a7..eeb0d6a253 100644 --- a/deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml +++ b/deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml @@ -1,5 +1,5 @@ --- -apiVersion: logstash.k8s.elastic.co/v1 +apiVersion: logstash.k8s.elastic.co/v1alpha1 kind: Logstash metadata: name: {{ include "logstash.fullname" . }} @@ -44,13 +44,8 @@ spec: {{- toYaml . | nindent 4 }} {{- end }} - pipelines: - {{ toYaml .Values.pipelines | nindent 4 }} - volumeClaimTemplates: - {{ toYaml .Values.volumeClaimTemplates | nindent 4 }} - elasticsearchRefs: - {{ toYaml .Values.elasticsearchRefs | nindent 4 }} - services: - {{ toYaml .Values.services | nindent 4 }} - secureSettings: - {{ toYaml .Values.secureSettings | nindent 4 }} + pipelines: {{ toYaml .Values.pipelines | nindent 4 }} + volumeClaimTemplates: {{ toYaml .Values.volumeClaimTemplates | nindent 4 }} + elasticsearchRefs: {{ toYaml .Values.elasticsearchRefs | nindent 4 }} + services: {{ toYaml .Values.services | nindent 4 }} + secureSettings: {{ toYaml .Values.secureSettings | nindent 4 }} From 040f6b1402ef6dfaab628219cc99c932a1a487e5 Mon Sep 17 00:00:00 2001 From: Michael Montgomery Date: Thu, 12 Oct 2023 09:30:14 -0500 Subject: [PATCH 08/16] Add logstash to docs. Update Helm stack readme to include logstash. Add logstash example. Signed-off-by: Michael Montgomery --- deploy/eck-stack/README.md | 2 + .../examples/logstash/basic-eck.yaml | 104 ++++++++++++++++++ .../stack-helm-chart.asciidoc | 15 ++- hack/helm/release/README.md | 1 + 4 files changed, 121 insertions(+), 1 deletion(-) create mode 100644 deploy/eck-stack/examples/logstash/basic-eck.yaml diff --git a/deploy/eck-stack/README.md b/deploy/eck-stack/README.md index 31cc0bdb34..e0b4bc7ca9 100644 --- a/deploy/eck-stack/README.md +++ b/deploy/eck-stack/README.md @@ -12,6 +12,7 @@ The following Elastic Stack resources are currently supported. - Elastic Agent - Fleet Server - Beats +- Logstash Additional resources will be supported in future releases of this Helm Chart. @@ -75,6 +76,7 @@ The following table lists the configurable parameters of the eck-stack chart and | `eck-kibana.enabled` | If `true`, create a Kibana resource (using the eck-kibana Chart) | `true` | | `eck-agent.enabled` | If `true`, create an Elastic Agent resource (using the eck-agent Chart) | `false` | | `eck-fleet-server.enabled` | If `true`, create a Fleet Server resource (using the eck-fleet-server Chart) | `false` | +| `eck-logstash.enabled` | If `true`, create a Logstash resource (using the eck-logstash Chart) | `false` | Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. diff --git a/deploy/eck-stack/examples/logstash/basic-eck.yaml b/deploy/eck-stack/examples/logstash/basic-eck.yaml new file mode 100644 index 0000000000..200afdb86e --- /dev/null +++ b/deploy/eck-stack/examples/logstash/basic-eck.yaml @@ -0,0 +1,104 @@ +--- +eck-elasticsearch: + version: 8.9.0 + nodeSets: + - name: default + count: 3 + config: + node.store.allow_mmap: false + podTemplate: + spec: + containers: + - name: elasticsearch + resources: + limits: + memory: 2Gi + requests: + memory: 2Gi +eck-beats: + enabled: true + version: 8.9.0 + spec: + type: filebeat + daemonSet: null + config: + filebeat.inputs: + - type: log + paths: + - /data/logstash-tutorial.log + processors: + - add_host_metadata: {} + - add_cloud_metadata: {} + output.logstash: + hosts: ["logstash-ls-beats:5044"] + deployment: + podTemplate: + spec: + automountServiceAccountToken: true + initContainers: + - name: download-tutorial + image: curlimages/curl + command: ["/bin/sh"] + args: ["-c", "curl -L https://download.elastic.co/demos/logstash/gettingstarted/logstash-tutorial.log.gz | gunzip -c > /data/logstash-tutorial.log"] + volumeMounts: + - name: data + mountPath: /data + containers: + - name: filebeat + securityContext: + runAsUser: 1000 + volumeMounts: + - name: data + mountPath: /data + - name: beat-data + mountPath: /usr/share/filebeat/data + volumes: + - name: data + emptydir: {} + - name: beat-data + emptydir: {} +eck-logstash: + enabled: true + + version: 8.9.0 + + elasticsearchRefs: + - clusterName: eck + name: elasticsearch + + pipelines: + - pipeline.id: main + config.string: | + input { + beats { + port => 5044 + } + } + filter { + grok { + match => { "message" => "%{HTTPD_COMMONLOG}"} + } + geoip { + source => "[source][address]" + target => "[source]" + } + } + output { + elasticsearch { + hosts => [ "${ECK_ES_HOSTS}" ] + user => "${ECK_ES_USER}" + password => "${ECK_ES_PASSWORD}" + ssl_certificate_authorities => "${ECK_ES_SSL_CERTIFICATE_AUTHORITY}" + } + } + + services: + - name: beats + service: + spec: + type: ClusterIP + ports: + - port: 5044 + name: "filebeat" + protocol: TCP + targetPort: 5044 diff --git a/docs/orchestrating-elastic-stack-applications/stack-helm-chart.asciidoc b/docs/orchestrating-elastic-stack-applications/stack-helm-chart.asciidoc index 2e6373968b..cab74d02eb 100644 --- a/docs/orchestrating-elastic-stack-applications/stack-helm-chart.asciidoc +++ b/docs/orchestrating-elastic-stack-applications/stack-helm-chart.asciidoc @@ -60,6 +60,19 @@ helm install eck-stack-with-fleet elastic/eck-stack \ --values https://raw.githubusercontent.com/elastic/cloud-on-k8s/{eck_release_branch}/deploy/eck-stack/examples/agent/fleet-agents.yaml -n elastic-stack ---- +[float] +[id="{p}-install-logstash-elasticsearch-kibana-helm"] +== Installing Logstash along with Elasticsearch and Kibana using the eck-stack Helm Chart + +The following section builds upon the previous sections, and allows installing Logstash along with Elasticsearch and Kibana. + +[source,sh,subs="attributes"] +---- +# Install an eck-managed Elasticsearch, Kibana, Fleet Server, and managed Elastic Agents using custom values. +helm install eck-stack-with-logstash elastic/eck-stack \ + --values https://raw.githubusercontent.com/elastic/cloud-on-k8s/{eck_release_branch}/deploy/eck-stack/examples/logstash/basic-eck.yaml -n elastic-stack +---- + [float] [id="{p}-eck-stack-individual-components"] === Installing individual components of the Elastic Stack using the Helm Charts @@ -81,4 +94,4 @@ helm install es-quickstart elastic/eck-stack -n elastic-stack --create-namespace [source,sh] ---- helm install es-quickstart elastic/eck-elasticsearch -n elastic-stack --create-namespace ----- \ No newline at end of file +---- diff --git a/hack/helm/release/README.md b/hack/helm/release/README.md index a849498a98..7b0fe66b0a 100644 --- a/hack/helm/release/README.md +++ b/hack/helm/release/README.md @@ -35,6 +35,7 @@ ECK Helm charts are grouped under 2 parent charts: `eck-operator` and `eck-stack │   │   ├── eck-elasticsearch │   │   ├── eck-fleet-server │   │   └── eck-kibana +│   │   └── eck-logstash ``` All subcharts are managed locally through their parent's `charts/` directory. From f26037e134bdfa90d25f2c1c4ade8e23c6b45013 Mon Sep 17 00:00:00 2001 From: Michael Montgomery Date: Thu, 12 Oct 2023 10:35:05 -0500 Subject: [PATCH 09/16] Add kibana to example to match docs. Add some notes to the example. Signed-off-by: Michael Montgomery --- .../examples/logstash/basic-eck.yaml | 21 ++++++++++++------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/deploy/eck-stack/examples/logstash/basic-eck.yaml b/deploy/eck-stack/examples/logstash/basic-eck.yaml index 200afdb86e..d6ebdcec2c 100644 --- a/deploy/eck-stack/examples/logstash/basic-eck.yaml +++ b/deploy/eck-stack/examples/logstash/basic-eck.yaml @@ -1,6 +1,5 @@ --- eck-elasticsearch: - version: 8.9.0 nodeSets: - name: default count: 3 @@ -15,9 +14,14 @@ eck-elasticsearch: memory: 2Gi requests: memory: 2Gi +eck-kibana: + enabled: true + spec: + count: 1 + elasticsearchRef: + name: elasticsearch eck-beats: enabled: true - version: 8.9.0 spec: type: filebeat daemonSet: null @@ -30,7 +34,8 @@ eck-beats: - add_host_metadata: {} - add_cloud_metadata: {} output.logstash: - hosts: ["logstash-ls-beats:5044"] + # This needs to be {{logstash-name}}-ls-beats:5044 + hosts: ["logstash-ls-beats-ls-beats:5044"] deployment: podTemplate: spec: @@ -59,13 +64,14 @@ eck-beats: emptydir: {} eck-logstash: enabled: true - - version: 8.9.0 - + # This is required to be able to set the logstash + # output of beats in a consistent manner. + fullnameOverride: "logstash-ls-beats" elasticsearchRefs: + # This clusterName is required to match the environment variables + # used in the below config.string output section. - clusterName: eck name: elasticsearch - pipelines: - pipeline.id: main config.string: | @@ -91,7 +97,6 @@ eck-logstash: ssl_certificate_authorities => "${ECK_ES_SSL_CERTIFICATE_AUTHORITY}" } } - services: - name: beats service: From 7678c1f895d86cb8c0fdf28aef866ac81776b57d Mon Sep 17 00:00:00 2001 From: Michael Montgomery Date: Thu, 12 Oct 2023 10:38:31 -0500 Subject: [PATCH 10/16] Update logstash example versions. Signed-off-by: Michael Montgomery --- deploy/eck-stack/charts/eck-logstash/examples/basic-eck.yaml | 2 +- deploy/eck-stack/charts/eck-logstash/examples/es-role.yaml | 2 +- deploy/eck-stack/charts/eck-logstash/examples/monitored.yaml | 2 +- deploy/eck-stack/charts/eck-logstash/examples/multi.yaml | 2 +- deploy/eck-stack/charts/eck-logstash/examples/volumes.yaml | 2 +- deploy/eck-stack/charts/eck-logstash/values.yaml | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) diff --git a/deploy/eck-stack/charts/eck-logstash/examples/basic-eck.yaml b/deploy/eck-stack/charts/eck-logstash/examples/basic-eck.yaml index 6db1a1d601..ea7349005c 100644 --- a/deploy/eck-stack/charts/eck-logstash/examples/basic-eck.yaml +++ b/deploy/eck-stack/charts/eck-logstash/examples/basic-eck.yaml @@ -1,6 +1,6 @@ --- # values corresponding to config/recipes/logstash/logstash-eck.yaml -version: 8.10.0-SNAPSHOT +version: 8.11.0-SNAPSHOT elasticsearchRefs: - clusterName: eck diff --git a/deploy/eck-stack/charts/eck-logstash/examples/es-role.yaml b/deploy/eck-stack/charts/eck-logstash/examples/es-role.yaml index 45a1f282a8..6e725fae17 100644 --- a/deploy/eck-stack/charts/eck-logstash/examples/es-role.yaml +++ b/deploy/eck-stack/charts/eck-logstash/examples/es-role.yaml @@ -1,6 +1,6 @@ --- # values corresponding to config/recipes/logstash/logstash-es-role.yaml -version: 8.10.0-SNAPSHOT +version: 8.11.0-SNAPSHOT elasticsearchRefs: - clusterName: eck diff --git a/deploy/eck-stack/charts/eck-logstash/examples/monitored.yaml b/deploy/eck-stack/charts/eck-logstash/examples/monitored.yaml index 9bbb061ad0..dc6b52ae53 100644 --- a/deploy/eck-stack/charts/eck-logstash/examples/monitored.yaml +++ b/deploy/eck-stack/charts/eck-logstash/examples/monitored.yaml @@ -1,6 +1,6 @@ --- # values corresponding to config/recipes/logstash/logstash-monitored.yaml -version: 8.10.0-SNAPSHOT +version: 8.11.0-SNAPSHOT monitoring: metrics: diff --git a/deploy/eck-stack/charts/eck-logstash/examples/multi.yaml b/deploy/eck-stack/charts/eck-logstash/examples/multi.yaml index 2e40063e6d..cebc9c9023 100644 --- a/deploy/eck-stack/charts/eck-logstash/examples/multi.yaml +++ b/deploy/eck-stack/charts/eck-logstash/examples/multi.yaml @@ -1,6 +1,6 @@ --- # values corresponding to config/recipes/logstash/logstash-multi.yaml -version: 8.10.0-SNAPSHOT +version: 8.11.0-SNAPSHOT pipelines: - pipeline.id: main diff --git a/deploy/eck-stack/charts/eck-logstash/examples/volumes.yaml b/deploy/eck-stack/charts/eck-logstash/examples/volumes.yaml index f4495ce12c..f36cf2c127 100644 --- a/deploy/eck-stack/charts/eck-logstash/examples/volumes.yaml +++ b/deploy/eck-stack/charts/eck-logstash/examples/volumes.yaml @@ -1,6 +1,6 @@ --- # values corresponding to config/recipes/logstash/logstash-volumes.yaml -version: 8.10.0-SNAPSHOT +version: 8.11.0-SNAPSHOT config: log.level: info diff --git a/deploy/eck-stack/charts/eck-logstash/values.yaml b/deploy/eck-stack/charts/eck-logstash/values.yaml index 45f287d313..66ed760be1 100644 --- a/deploy/eck-stack/charts/eck-logstash/values.yaml +++ b/deploy/eck-stack/charts/eck-logstash/values.yaml @@ -18,7 +18,7 @@ # Version of Logstash. # -version: 8.10.0-SNAPSHOT +version: 8.11.0-SNAPSHOT # Logstash Docker image to deploy # From dca93d7e13d3441d3e5449b4be6710064f8156ee Mon Sep 17 00:00:00 2001 From: Michael Montgomery Date: Thu, 12 Oct 2023 10:44:55 -0500 Subject: [PATCH 11/16] Adjusting values file with more links/docs. Signed-off-by: Michael Montgomery --- deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml | 2 +- deploy/eck-stack/charts/eck-logstash/values.yaml | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml b/deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml index eeb0d6a253..9ce8048bb4 100644 --- a/deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml +++ b/deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml @@ -12,7 +12,7 @@ metadata: {{- end }} spec: version: {{ required "A Logstash version is required" .Values.version }} - count: {{ required "An amount of pod is required" .Values.count }} + count: {{ required "A pod count is required" .Values.count }} {{- with .Values.image }} image: {{ . }} {{- end }} diff --git a/deploy/eck-stack/charts/eck-logstash/values.yaml b/deploy/eck-stack/charts/eck-logstash/values.yaml index 66ed760be1..065918e03c 100644 --- a/deploy/eck-stack/charts/eck-logstash/values.yaml +++ b/deploy/eck-stack/charts/eck-logstash/values.yaml @@ -90,9 +90,13 @@ pipelineRef: {} volumeClaimTemplates: [] # ElasticsearchRefs are references to Elasticsearch clusters running in the same Kubernetes cluster. +# Ensure that the 'clusterName' field matches what is referenced in the pipeline. +# ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-logstash-configuration.html#k8s-logstash-pipelines-es +# elasticsearchRefs: [] # - namespace: '' # name: '' +# clusterName: '' # serviceName: '' # secretName: '' From 5775d5cc1ad681e6e8f829f2f5b6d6c70b3effde Mon Sep 17 00:00:00 2001 From: Michael Montgomery Date: Thu, 12 Oct 2023 10:46:53 -0500 Subject: [PATCH 12/16] Add the standard note for allow_mmap Signed-off-by: Michael Montgomery --- deploy/eck-stack/examples/logstash/basic-eck.yaml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/deploy/eck-stack/examples/logstash/basic-eck.yaml b/deploy/eck-stack/examples/logstash/basic-eck.yaml index d6ebdcec2c..00f0b94f9e 100644 --- a/deploy/eck-stack/examples/logstash/basic-eck.yaml +++ b/deploy/eck-stack/examples/logstash/basic-eck.yaml @@ -4,6 +4,11 @@ eck-elasticsearch: - name: default count: 3 config: + # Comment out when setting the vm.max_map_count via initContainer, as these are mutually exclusive. + # For production workloads, it is strongly recommended to increase the kernel setting vm.max_map_count to 262144 + # and leave node.store.allow_mmap unset. + # ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-virtual-memory.html + # node.store.allow_mmap: false podTemplate: spec: From 84a4c3d0cd10322d40f080cfb145e6c79b964788 Mon Sep 17 00:00:00 2001 From: azro352 Date: Sun, 15 Oct 2023 12:20:59 +0200 Subject: [PATCH 13/16] Fix pipelineRef to pipelinesRef --- .../eck-stack/charts/eck-logstash/templates/logstash.yaml | 4 ++-- .../charts/eck-logstash/templates/tests/logstash_test.yaml | 6 +++--- deploy/eck-stack/charts/eck-logstash/values.yaml | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml b/deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml index 9ce8048bb4..36131f010e 100644 --- a/deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml +++ b/deploy/eck-stack/charts/eck-logstash/templates/logstash.yaml @@ -31,8 +31,8 @@ spec: configRef: {{- toYaml . | nindent 4 }} {{- end }} - {{- with .Values.pipelineRef }} - pipelineRef: + {{- with .Values.pipelinesRef }} + pipelinesRef: {{- toYaml . | nindent 4 }} {{- end }} {{- with .Values.podTemplate }} diff --git a/deploy/eck-stack/charts/eck-logstash/templates/tests/logstash_test.yaml b/deploy/eck-stack/charts/eck-logstash/templates/tests/logstash_test.yaml index f4fb4ad332..9a8584a5a5 100644 --- a/deploy/eck-stack/charts/eck-logstash/templates/tests/logstash_test.yaml +++ b/deploy/eck-stack/charts/eck-logstash/templates/tests/logstash_test.yaml @@ -157,12 +157,12 @@ tests: - equal: path: spec.configRef.secretRef.secretName value: configLogstashSecretName - - it: should render pipelineRef properly + - it: should render pipelinesRef properly set: - pipelineRef: + pipelinesRef: secretRef: secretName: pipelineLogstashSecretName asserts: - equal: - path: spec.pipelineRef.secretRef.secretName + path: spec.pipelinesRef.secretRef.secretName value: pipelineLogstashSecretName diff --git a/deploy/eck-stack/charts/eck-logstash/values.yaml b/deploy/eck-stack/charts/eck-logstash/values.yaml index 065918e03c..f738aeeb4f 100644 --- a/deploy/eck-stack/charts/eck-logstash/values.yaml +++ b/deploy/eck-stack/charts/eck-logstash/values.yaml @@ -79,7 +79,7 @@ monitoring: {} # pipelines: [] -pipelineRef: {} +pipelinesRef: {} # secretRef: # secretName: '' From 5d7b848e73d464fb77e00512ac8c51ef3027a443 Mon Sep 17 00:00:00 2001 From: azro352 <35503478+azro352@users.noreply.github.com> Date: Sun, 15 Oct 2023 12:22:00 +0200 Subject: [PATCH 14/16] Update deploy/eck-stack/charts/eck-logstash/values.yaml Co-authored-by: Peter Brachwitz --- deploy/eck-stack/charts/eck-logstash/values.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deploy/eck-stack/charts/eck-logstash/values.yaml b/deploy/eck-stack/charts/eck-logstash/values.yaml index f738aeeb4f..10324c419a 100644 --- a/deploy/eck-stack/charts/eck-logstash/values.yaml +++ b/deploy/eck-stack/charts/eck-logstash/values.yaml @@ -74,7 +74,7 @@ monitoring: {} # - name: monitoring # namespace: observability -# The logstash pipelines, the ECK equivalent to pipelines.yml +# The Logstash pipelines, the ECK equivalent to pipelines.yml # ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-logstash-configuration.html#k8s-logstash-pipelines # pipelines: [] From d7c4a6b4582c36dfdbf11026c2ba21bdc547c482 Mon Sep 17 00:00:00 2001 From: Michael Montgomery Date: Mon, 16 Oct 2023 08:00:25 -0500 Subject: [PATCH 15/16] Fix wording in docs. Signed-off-by: Michael Montgomery --- .../stack-helm-chart.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/orchestrating-elastic-stack-applications/stack-helm-chart.asciidoc b/docs/orchestrating-elastic-stack-applications/stack-helm-chart.asciidoc index cab74d02eb..caeb34ef23 100644 --- a/docs/orchestrating-elastic-stack-applications/stack-helm-chart.asciidoc +++ b/docs/orchestrating-elastic-stack-applications/stack-helm-chart.asciidoc @@ -68,7 +68,7 @@ The following section builds upon the previous sections, and allows installing L [source,sh,subs="attributes"] ---- -# Install an eck-managed Elasticsearch, Kibana, Fleet Server, and managed Elastic Agents using custom values. +# Install an eck-managed Elasticsearch, Kibana, Beats and Logstash using custom values. helm install eck-stack-with-logstash elastic/eck-stack \ --values https://raw.githubusercontent.com/elastic/cloud-on-k8s/{eck_release_branch}/deploy/eck-stack/examples/logstash/basic-eck.yaml -n elastic-stack ---- From a375f890bddd700a168a997108ca931a28a0c041 Mon Sep 17 00:00:00 2001 From: Michael Montgomery Date: Mon, 16 Oct 2023 08:08:20 -0500 Subject: [PATCH 16/16] Update wording in docs. Signed-off-by: Michael Montgomery --- .../stack-helm-chart.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/orchestrating-elastic-stack-applications/stack-helm-chart.asciidoc b/docs/orchestrating-elastic-stack-applications/stack-helm-chart.asciidoc index caeb34ef23..d9adf02778 100644 --- a/docs/orchestrating-elastic-stack-applications/stack-helm-chart.asciidoc +++ b/docs/orchestrating-elastic-stack-applications/stack-helm-chart.asciidoc @@ -62,9 +62,9 @@ helm install eck-stack-with-fleet elastic/eck-stack \ [float] [id="{p}-install-logstash-elasticsearch-kibana-helm"] -== Installing Logstash along with Elasticsearch and Kibana using the eck-stack Helm Chart +== Installing Logstash along with Elasticsearch, Kibana and Beats using the eck-stack Helm Chart -The following section builds upon the previous sections, and allows installing Logstash along with Elasticsearch and Kibana. +The following section builds upon the previous sections, and allows installing Logstash along with Elasticsearch, Kibana and Beats. [source,sh,subs="attributes"] ----