Elasticsearch index must be lowercase #16081

urso · 2020-02-04T18:21:54Z

Bug

What does this PR do?

This change ensure that index names are always converted to lowercase.
Static strings are converted to lowercase upfront, while dynamic strings
will be post-processed.

Why is it important?

When indexing into Elasticsearch index names must always be lowercase.
If the index or indices setting are configured to produce non-lowercase
strings (e.g. by extracting part of the index name from the event
contents), we need to normalize them to be lowercase.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~- [ ] I have made corresponding changes to the documentation~~
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works

Author's Checklist

[ ]

How to test this PR locally

Index with different index/indices configurations into Elasticsearch:

use upper case in index setting
extract index name from event, that has the field value as upper case string
configure a mapping via indices where the target index is upper case

Run same checks with Redis, Kafka, Logstash outputs (the index name should consistently be lower case for all outputs).

Related issues

Use cases

Screenshots

Logs

CHANGELOG.next.asciidoc

kvch · 2020-02-04T19:09:15Z

libbeat/outputs/outil/select.go

-				str = otherwise
-			}
-
-			if str == "" {


I might be too early to click on approve. Don't we need to set sel to nilSelector if otherwise is an empty string?

We have had this pattern in multiple locations. I moved some of the logic into the expression constructors. e.g. constSelectorExpr will check if str is "" and create a nilSelector or constSelector. This change makes code (hopefully) a little easier to read, while reducing duplication and chance of errors.

When indexing into Elasticsearch index names must always be lowercase. If the index or indices setting are configured to produce non-lowercase strings (e.g. by extracting part of the index name from the event contents), we need to normalize them to be lowercase. This change ensure that index names are always converted to lowercase. Static strings are converted to lowercase upfront, while dynamic strings will be post-processed.

urso · 2020-02-06T17:55:19Z

Failing tests due to fixes in the ES timestamp parsing. These will be fixed by: #16139

* Index names must be lowercase When indexing into Elasticsearch index names must always be lowercase. If the index or indices setting are configured to produce non-lowercase strings (e.g. by extracting part of the index name from the event contents), we need to normalize them to be lowercase. This change ensure that index names are always converted to lowercase. Static strings are converted to lowercase upfront, while dynamic strings will be post-processed. * update kafka/redis/LS output to guarantee lowercase index * add godoc (cherry picked from commit 7ddcb1e)

* Elasticsearch index must be lowercase (#16081) * Index names must be lowercase When indexing into Elasticsearch index names must always be lowercase. If the index or indices setting are configured to produce non-lowercase strings (e.g. by extracting part of the index name from the event contents), we need to normalize them to be lowercase. This change ensure that index names are always converted to lowercase. Static strings are converted to lowercase upfront, while dynamic strings will be post-processed. * update kafka/redis/LS output to guarantee lowercase index * add godoc (cherry picked from commit 7ddcb1e)

* [Filebeat] move create-[module,fileset,fields] to mage (#15836) - move create-[module,fileset,fields] to mage - make mage create commands available in x-pack/filebeat - change Makefile to use mage for create commands * Elasticsearch index must be lowercase (#16081) * Index names must be lowercase When indexing into Elasticsearch index names must always be lowercase. If the index or indices setting are configured to produce non-lowercase strings (e.g. by extracting part of the index name from the event contents), we need to normalize them to be lowercase. This change ensure that index names are always converted to lowercase. Static strings are converted to lowercase upfront, while dynamic strings will be post-processed. * update kafka/redis/LS output to guarantee lowercase index * add godoc * Regenerate expected files after changes in date parsing (#16139) Elasticsearch has modified the behaviour on date parsing when the date doesn't include timezone data. Regenerate a couple of golden files that are affected by this change. * Add autodiscover for aws_ec2 (#14823) * Add autodiscover for aws_ec2 * Add aws.ec2.* to autodiscover template * Fix a connection error in httpjson input (#16123) * Fix a connection error in httpjson input * Include document_id in decode_json_fields allowed fields (#16156) * ci: run test on Windows (#15570) * feat: run test on Windows * chore: parameter to enable/disable windows test * deleteDir before of the checkout * Update Jenkinsfile * Update Jenkinsfile * Update Jenkinsfile * Update Jenkinsfile * Update Jenkinsfile * Apply suggestions from code review * feat: apply dependency hierarchies * fit: use isChanged for all matches, and add the libbeat match where it is needed * feat: add x-pack/winlogbeat windows unit tests * fix: duplicate when condition * Update Jenkinsfile Co-Authored-By: Andrew Kroh <andrew.kroh@elastic.co> Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> * improve kubernetes.pod.cpu.usage.limit.pct field description (#16128) * upgrade github.com/gogo/protobuf/... to v1.3.1 (#16138) * [Filebeat] Add ECS tls & categorization fields to apache module (#16121) * Add ECS tls & categorization fields to apache module - tls.cipher (access) - tls.protocol (access) - tls.protocol_version (access) - event.kind (access) - event.category (access) - event.outcome (access) - lowercase http.request.method for ECS compliance (access) - event.kind (error) - event.category (error) - event.type (error) Closes #16032 * [Metricbeat] Add Overview dashboard to Tomcat module * [Metricbeat] Fix PostgreSQL Dashboard (#16132) * [Metricbeat] Fix PostgreSQL Dashboard * Update version * Fix: imports order (#16207) * [Metricbeat]kube-state-metrics: add storage class support (#16145) * add ksm storage class support * [Journalbeat] Improve parsing of syslog.pid in journalbeat to strip the username when present (#16116) * Improve parsing of syslog.pid in journalbeat to strip the username in pid when present. * Add entry to changelog with pull ID. * Improve the comment on the username strip. * [Agent] Allow CA cert pinning on the Elasticsearch output or any code that user tlscommon.TLSConfig builder. (#16019) * Add a sha256 pin for the CA Certificate When multiples CA are presents on the system we cannot ensure that a specific one was used to validates the chains exposer by the server. This PRs adds a `ca_sha256` option to the `tlscommon.TLSConfig` that is used by all the code that has to create a TCP client with TLS support. When the option is set, it will hook a new callback in the validation chains that will inspect the verified and validated chains by Go to ensure that a lets a certificate in the chains match the provided sha256. Usage example for the Elasticsearch output. ``` output.elasticsearch: hosts: [127.0.0.1:9200] ssl.ca_sha256: <base64_encoded_sha1> ``` You can generate the pin using the **openssl** binary with the following command: ``` openssl x509 -in ca.crt -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64 ``` OpenSSL's [documentation](https://www.openssl.org/docs/manmaster/man1/dgst.html) You will need to start Elasticsearch with the following options ```yaml xpack.security.enabled: true indices.id_field_data.enabled: true xpack.license.self_generated.type: trial xpack.security.http.ssl.enabled: true xpack.security.http.ssl.key: /etc/pki/localhost/localhost.key" xpack.security.http.ssl.certificate: /etc/pki/localhost/localhost.crt" xpack.security.http.ssl.certificate_authorities: /etc/pki/ca/ca.crt" ``` This pull request also include a new service in the docker-compose.yml that will start a new Elasticsearch server with TLS and security configured. * [docs] Add 7.6 breaking changes and release highlights (#16202) * [docs] Add early draft of Elastic Log Driver docs (#15799) * Index template will only be loaded if the configured output is Elasticsearch or Elastic Cloud (#16124) (#16225) Minor update to be more explicit on the index template loading requirement. Co-authored-by: romain-chanu <51113389+romain-chanu@users.noreply.github.com> * Remove spaces in prometheus commented out option (#16233) * Fix: don't miss address scheme (#16205) * Fix: don't miss address scheme * Add unit test * Adjust source after code review * Add comment to method * Freeze virtualenv version until issue with CI is resolved (#16235) * [docs] Fix install command to match instructions on docker hub (#16249) * [docs] Add link to observability release blog (#16246) * ci(jenkins): enable fix-permissions to be executed without running make too (#16130) * ci(jenkins): enable fix-permissions to be executed without running make too * ci(jenkins): go modules are stored in the HOME path * ci(jenkins): fix permissions should run only if docker is enabled * Upgrade go-ucfg to version 0.8.2 (#16199) * Upgrade go-ucfg to master, for testing before 0.8.2 release. * Update notice. * Fix tests. * Update to the v0.8.2 release tag and remake NOTICE.txt. * Improve test name. * Add ingress nginx controller fileset (#16197) * update notice Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com> Co-authored-by: Steffen Siering <steffen.siering@elastic.co> Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: Lei Qiu <lei.qiu@elastic.co> Co-authored-by: Fae Charlton <fae.charlton@elastic.co> Co-authored-by: Ivan Fernandez Calvo <kuisathaverat@users.noreply.github.com> Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> Co-authored-by: Chris Mark <chrismarkou92@gmail.com> Co-authored-by: Gil Raphaelli <g@raphaelli.com> Co-authored-by: Mario Castro <mariocaster@gmail.com> Co-authored-by: Dimitri Mazmanov <sorantis@gmail.com> Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com> Co-authored-by: Pablo Mercado <pablo.mercado@elastic.co> Co-authored-by: Blake Rouse <blake.rouse@elastic.co> Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com> Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: romain-chanu <51113389+romain-chanu@users.noreply.github.com> Co-authored-by: Michal Pristas <michal.pristas@gmail.com> Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'. ## Why is it important? Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (#16081. #6342). As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (#18640). With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs.

Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'. ## Why is it important? Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (elastic#16081. elastic#6342). As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (elastic#18640). With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs. (cherry picked from commit 28f7aca)

…19118) Make selector string casing configurable (#18854) Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'. Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (#16081. #6342). As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (#18640). With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs. (cherry picked from commit 28f7aca)

Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'. Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (elastic#16081. elastic#6342). As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (elastic#18640). With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs. (cherry picked from commit 28f7aca)

Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'. ## Why is it important? Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (elastic#16081. elastic#6342). As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (elastic#18640). With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs. (cherry picked from commit 28f7aca)

Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'. ## Why is it important? Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (elastic#16081. elastic#6342). As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (elastic#18640). With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs.

* [Filebeat] move create-[module,fileset,fields] to mage (elastic#15836) - move create-[module,fileset,fields] to mage - make mage create commands available in x-pack/filebeat - change Makefile to use mage for create commands * Elasticsearch index must be lowercase (elastic#16081) * Index names must be lowercase When indexing into Elasticsearch index names must always be lowercase. If the index or indices setting are configured to produce non-lowercase strings (e.g. by extracting part of the index name from the event contents), we need to normalize them to be lowercase. This change ensure that index names are always converted to lowercase. Static strings are converted to lowercase upfront, while dynamic strings will be post-processed. * update kafka/redis/LS output to guarantee lowercase index * add godoc * Regenerate expected files after changes in date parsing (elastic#16139) Elasticsearch has modified the behaviour on date parsing when the date doesn't include timezone data. Regenerate a couple of golden files that are affected by this change. * Add autodiscover for aws_ec2 (elastic#14823) * Add autodiscover for aws_ec2 * Add aws.ec2.* to autodiscover template * Fix a connection error in httpjson input (elastic#16123) * Fix a connection error in httpjson input * Include document_id in decode_json_fields allowed fields (elastic#16156) * ci: run test on Windows (elastic#15570) * feat: run test on Windows * chore: parameter to enable/disable windows test * deleteDir before of the checkout * Update Jenkinsfile * Update Jenkinsfile * Update Jenkinsfile * Update Jenkinsfile * Update Jenkinsfile * Apply suggestions from code review * feat: apply dependency hierarchies * fit: use isChanged for all matches, and add the libbeat match where it is needed * feat: add x-pack/winlogbeat windows unit tests * fix: duplicate when condition * Update Jenkinsfile Co-Authored-By: Andrew Kroh <andrew.kroh@elastic.co> Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> * improve kubernetes.pod.cpu.usage.limit.pct field description (elastic#16128) * upgrade github.com/gogo/protobuf/... to v1.3.1 (elastic#16138) * [Filebeat] Add ECS tls & categorization fields to apache module (elastic#16121) * Add ECS tls & categorization fields to apache module - tls.cipher (access) - tls.protocol (access) - tls.protocol_version (access) - event.kind (access) - event.category (access) - event.outcome (access) - lowercase http.request.method for ECS compliance (access) - event.kind (error) - event.category (error) - event.type (error) Closes elastic#16032 * [Metricbeat] Add Overview dashboard to Tomcat module * [Metricbeat] Fix PostgreSQL Dashboard (elastic#16132) * [Metricbeat] Fix PostgreSQL Dashboard * Update version * Fix: imports order (elastic#16207) * [Metricbeat]kube-state-metrics: add storage class support (elastic#16145) * add ksm storage class support * [Journalbeat] Improve parsing of syslog.pid in journalbeat to strip the username when present (elastic#16116) * Improve parsing of syslog.pid in journalbeat to strip the username in pid when present. * Add entry to changelog with pull ID. * Improve the comment on the username strip. * [Agent] Allow CA cert pinning on the Elasticsearch output or any code that user tlscommon.TLSConfig builder. (elastic#16019) * Add a sha256 pin for the CA Certificate When multiples CA are presents on the system we cannot ensure that a specific one was used to validates the chains exposer by the server. This PRs adds a `ca_sha256` option to the `tlscommon.TLSConfig` that is used by all the code that has to create a TCP client with TLS support. When the option is set, it will hook a new callback in the validation chains that will inspect the verified and validated chains by Go to ensure that a lets a certificate in the chains match the provided sha256. Usage example for the Elasticsearch output. ``` output.elasticsearch: hosts: [127.0.0.1:9200] ssl.ca_sha256: <base64_encoded_sha1> ``` You can generate the pin using the **openssl** binary with the following command: ``` openssl x509 -in ca.crt -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64 ``` OpenSSL's [documentation](https://www.openssl.org/docs/manmaster/man1/dgst.html) You will need to start Elasticsearch with the following options ```yaml xpack.security.enabled: true indices.id_field_data.enabled: true xpack.license.self_generated.type: trial xpack.security.http.ssl.enabled: true xpack.security.http.ssl.key: /etc/pki/localhost/localhost.key" xpack.security.http.ssl.certificate: /etc/pki/localhost/localhost.crt" xpack.security.http.ssl.certificate_authorities: /etc/pki/ca/ca.crt" ``` This pull request also include a new service in the docker-compose.yml that will start a new Elasticsearch server with TLS and security configured. * [docs] Add 7.6 breaking changes and release highlights (elastic#16202) * [docs] Add early draft of Elastic Log Driver docs (elastic#15799) * Index template will only be loaded if the configured output is Elasticsearch or Elastic Cloud (elastic#16124) (elastic#16225) Minor update to be more explicit on the index template loading requirement. Co-authored-by: romain-chanu <51113389+romain-chanu@users.noreply.github.com> * Remove spaces in prometheus commented out option (elastic#16233) * Fix: don't miss address scheme (elastic#16205) * Fix: don't miss address scheme * Add unit test * Adjust source after code review * Add comment to method * Freeze virtualenv version until issue with CI is resolved (elastic#16235) * [docs] Fix install command to match instructions on docker hub (elastic#16249) * [docs] Add link to observability release blog (elastic#16246) * ci(jenkins): enable fix-permissions to be executed without running make too (elastic#16130) * ci(jenkins): enable fix-permissions to be executed without running make too * ci(jenkins): go modules are stored in the HOME path * ci(jenkins): fix permissions should run only if docker is enabled * Upgrade go-ucfg to version 0.8.2 (elastic#16199) * Upgrade go-ucfg to master, for testing before 0.8.2 release. * Update notice. * Fix tests. * Update to the v0.8.2 release tag and remake NOTICE.txt. * Improve test name. * Add ingress nginx controller fileset (elastic#16197) * update notice Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com> Co-authored-by: Steffen Siering <steffen.siering@elastic.co> Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: Lei Qiu <lei.qiu@elastic.co> Co-authored-by: Fae Charlton <fae.charlton@elastic.co> Co-authored-by: Ivan Fernandez Calvo <kuisathaverat@users.noreply.github.com> Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> Co-authored-by: Chris Mark <chrismarkou92@gmail.com> Co-authored-by: Gil Raphaelli <g@raphaelli.com> Co-authored-by: Mario Castro <mariocaster@gmail.com> Co-authored-by: Dimitri Mazmanov <sorantis@gmail.com> Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com> Co-authored-by: Pablo Mercado <pablo.mercado@elastic.co> Co-authored-by: Blake Rouse <blake.rouse@elastic.co> Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com> Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: romain-chanu <51113389+romain-chanu@users.noreply.github.com> Co-authored-by: Michal Pristas <michal.pristas@gmail.com> Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

urso added review libbeat :Outputs test-plan Add this PR to be manual test plan Team:Beats v7.7.0 labels Feb 4, 2020

urso changed the title ~~Es index lowercase~~ Elasticsearch index must be lowercase Feb 4, 2020

urso mentioned this pull request Feb 4, 2020

Fix issue 6342: Automatically lowercase the index name #6640

Closed

urso added the needs_backport PR is waiting to be backported to other branches. label Feb 4, 2020

urso requested a review from a team February 4, 2020 18:46

kvch reviewed Feb 4, 2020

View reviewed changes

CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved

urso force-pushed the es-index-lowercase branch from 1192071 to 9d11d49 Compare February 4, 2020 18:56

kvch approved these changes Feb 4, 2020

View reviewed changes

urso requested a review from kvch February 4, 2020 19:03

kvch reviewed Feb 4, 2020

View reviewed changes

kvch approved these changes Feb 5, 2020

View reviewed changes

andresrc added [zube]: Inbox [zube]: In Review and removed [zube]: Inbox labels Feb 5, 2020

urso added 7 commits February 5, 2020 18:07

update kafka/redis/LS output to guarantee lowercase index

749b73a

add changelog entry

7ab4af3

unexport symbols

6773e45

cleanup changelog

e77dfc3

More logic in fmtSelectorExpr

288e07f

re-export symbols and add godoc

29eed25

urso force-pushed the es-index-lowercase branch from 5828f4e to 29eed25 Compare February 5, 2020 18:10

fix logstash integration test build

c9aa41a

urso merged commit 7ddcb1e into elastic:master Feb 6, 2020

urso deleted the es-index-lowercase branch February 6, 2020 17:55

zube bot added [zube]: Done and removed [zube]: In Review labels Feb 6, 2020

urso mentioned this pull request Feb 6, 2020

Cherry-pick #16081 to 7.x: Elasticsearch index must be lowercase #16142

Merged

3 tasks

urso removed the needs_backport PR is waiting to be backported to other branches. label Feb 6, 2020

andresrc removed the [zube]: Done label Feb 11, 2020

andresrc added the Team:Integrations Label for the Integrations team label Mar 6, 2020

andresrc added the test-plan-added This PR has been added to the test plan label Mar 21, 2020

This was referenced May 28, 2020

[Winlogbeat] Output/Kafka Can’t use topics with uppercase letters anymore #18640

Closed

Make selector string casing configurable #18854

Merged

urso mentioned this pull request Jun 10, 2020

Cherry-pick #18854 to 7.x: Make selector string casing configurable #19118

Merged

4 tasks

urso mentioned this pull request Jun 12, 2020

Cherry-pick #18854 to 7.7: Make selector string casing configurable #19175

Merged

4 tasks

urso mentioned this pull request Jun 25, 2020

Cherry-pick #18854 to 7.8: Make selector string casing configurable #19409

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Elasticsearch index must be lowercase #16081

Elasticsearch index must be lowercase #16081

urso commented Feb 4, 2020 •

edited by andresrc

Loading

kvch Feb 4, 2020

urso Feb 4, 2020

urso commented Feb 6, 2020

Elasticsearch index must be lowercase #16081

Elasticsearch index must be lowercase #16081

Conversation

urso commented Feb 4, 2020 • edited by andresrc Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

kvch Feb 4, 2020

Choose a reason for hiding this comment

urso Feb 4, 2020

Choose a reason for hiding this comment

urso commented Feb 6, 2020

urso commented Feb 4, 2020 •

edited by andresrc

Loading