Introduce a Hashing Processor #31087

talevy · 2018-06-05T03:34:23Z

It is useful to have a processor similar to
https://www.elastic.co/guide/en/logstash/6.0/plugins-filters-fingerprint.html
in Elasticsearch. A processor that leverages a variety of hashing algorithms
to create cryptographically-secure one-way hashes of values in documents.

supersedes #30790

TODO:

rest tests maybe

follow-up PR: add documentation #31694

elasticmachine · 2018-06-05T03:34:24Z

Pinging @elastic/es-core-infra

joshbressers · 2018-06-05T18:57:00Z

I think replicating the logstash functionality would be a good first step. I see a lot of value in having a consistent experience for the users. I think having similar functionality out of the gate then evolving both processors would be a big win. My current understanding is the logstash functionality is acceptable for GDPR requirements.

gingerwizard · 2018-06-09T11:54:24Z

@talevy i assume in the first iteration the salt/hmac key will just be held as part of the ingest processor defn itself?

talevy · 2018-06-13T17:34:38Z

@gingerwizard any private keys will probably need to be stored in the keystore.

It makes sense to introduce new Security ingest processors (example: elastic#31087), and this change would give them a good place to be written.

It makes sense to introduce new Security ingest processors (example: #31087), and this change would give them a good place to be written.

It is useful to have a processor similar to https://www.elastic.co/guide/en/logstash/6.0/plugins-filters-fingerprint.html in Elasticsearch. A processor that leverages a variety of hashing algorithms to create cryptographically-secure one-way hashes of values in documents.

martijnvg

From the ingest side it looks good. I left two minor comments. If docs and an integ test is added then it LGTM.

martijnvg · 2018-06-27T08:59:52Z

x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/ingest/HashProcessor.java

+ * A processor that hashes the contents of a field (or fields) using various hashing algorithms
+ */
+public final class HashProcessor extends AbstractProcessor {
+    enum Method {


nit: maybe move this enum below the factory class?

martijnvg · 2018-06-27T09:01:35Z

x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/ingest/HashProcessor.java

+    public void execute(IngestDocument document) {
+        Map<String, String> hashedFieldValues = fields.stream().map(f -> {
+            try {
+                String value = document.getFieldValue(f, String.class);


Should there be missing field support?

talevy · 2018-06-27T15:12:04Z

thanks for the review @martijnvg, I'll follow up soon!

talevy · 2018-06-28T05:36:21Z

hi @martijnvg thank you for the review. I've added the ignore_missing flag, as well as the salt param back. The salt param can both be used by users, and also leveraged by our test infrastructure to verify hash values.

I could not find a great place in the security documentation to add docs for this and I do not want to block the PR because of this (I am going on vacation next week). For this reason, I will open an issue to add documentation for when I come back, what do you think?

martijnvg

LGTM, assuming the PR build is green. Currently it fails because of a checkstyle violation.

I could not find a great place in the security documentation to add docs for this and I do not want to block the PR because of this (I am going on vacation next week). For this reason, I will open an issue to add documentation for when I come back, what do you think?

I'm ok with this.

jkakavas · 2018-06-28T14:18:57Z

...ck/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authc/support/Hasher.java

@@ -277,7 +277,7 @@ public static Hasher resolve(String name) {

    public abstract boolean verify(SecureString data, char[] hash);

-    static final class SaltProvider {
+    public static final class SaltProvider {


We removed SaltProvider in soon to be merged #31234 and opted for generating a random byte array from SecureRandom and then Base64 encoding that to a string, so probably you need to do something similar here too. https://github.com/elastic/elasticsearch/pull/31234/files#diff-ebc23bc2cb194fa926b2cdafaedef9d4R565

ah, it is also private! I see, thanks for the heads up. How soon will that be merged? One of us will have to change their code pre-pushing 😄

It is useful to have a processor similar to logstash-filter-fingerprint in Elasticsearch. A processor that leverages a variety of hashing algorithms to create cryptographically-secure one-way hashes of values in documents. This processor introduces a pbkdf2hmac hashing scheme to fields in documents for indexing

* master: Mute 'Test typed keys parameter for suggesters' as we await a fix. Build test: Thread linger Fix gradle4.8 deprecation warnings (#31654) Mute FileRealmTests#testAuthenticateCaching with an @AwaitsFix. Mute TransportChangePasswordActionTests#testIncorrectPasswordHashingAlgorithm with an @AwaitsFix. Build: Fix naming conventions task (#31681) Introduce a Hashing Processor (#31087)

* elastic/6.x: Enable setting client path prefix to / (#30119) [DOCS] Secure settings specified per node (#31621) Build test: Thread linger Build: Fix naming conventions task (#31681) Introduce a Hashing Processor (#31087)

* elastic/ccr: (30 commits) Enable setting client path prefix to / (elastic#30119) [DOCS] Secure settings specified per node (elastic#31621) has_parent builder: exception message/param fix (elastic#31182) TEST: Randomize soft-deletes settings (elastic#31585) Mute 'Test typed keys parameter for suggesters' as we await a fix. Build test: Thread linger Fix gradle4.8 deprecation warnings (elastic#31654) Mute FileRealmTests#testAuthenticateCaching with an @AwaitsFix. Mute TransportChangePasswordActionTests#testIncorrectPasswordHashingAlgorithm with an @AwaitsFix. Build: Fix naming conventions task (elastic#31681) Introduce a Hashing Processor (elastic#31087) Do not check for object existence when deleting repository index files (elastic#31680) Remove extra check for object existence in repository-gcs read object (elastic#31661) Support multiple system store types (elastic#31650) [Test] Clean up some repository-s3 tests (elastic#31601) [Docs] Use capital letters in section headings (elastic#31678) muted tests that will be replaced by the shard follow task refactoring: elastic#31581 [DOCS] Add PQL language Plugin (elastic#31237) Merge AzureStorageService and AzureStorageServiceImpl and clean up tests (elastic#31607) TEST: Fix test task invocation (elastic#31657) ...

This reverts commit 8c78fe7.

This reverts commit b296936.

talevy · 2018-07-18T20:49:01Z

There are concerns that this implementation as was merged can easily lead to incorrect usage without clear feedback. For example, secret keys may be inconsistent between ESKeyStores which will lead to inconsistent hashing across ingests depending on which node operated on the documents.

For this reason, and others, This change is to be reverted.
PR for 6.x(6.4): #32179
PR for master: #32178

This reverts commit b296936.

This reverts commit 8c78fe7.

* 6.x: Fix rollup on date fields that don't support epoch_millis (#31890) Revert "Introduce a Hashing Processor (#31087)" (#32179) [test] use randomized runner in packaging tests (#32109) Painless: Fix caching bug and clean up addPainlessClass. (#32142) Fix BwC Tests looking for UUID Pre 6.4 (#32158) (#32169) Call setReferences() on custom referring tokenfilters in _analyze (#32157) Add more contexts to painless execute api (#30511) Add EC2 credential test for repository-s3 (#31918) Fix CP for namingConventions when gradle home has spaces (#31914) Convert Version to Java - clusterformation part1 (#32009) Fix Java 11 javadoc compile problem Improve docs for search preferences (#32098) Configurable password hashing algorithm/cost(#31234) (#32092) [DOCS] Update TLS on Docker for 6.3 ESIndexLevelReplicationTestCase doesn't support replicated failures but it's good to know what they are Switch distribution to new style Requests (#30595) Build: Skip jar tests if jar disabled Build: Move shadow customizations into common code (#32014) Painless: Add PainlessClassBuilder (#32141) Fix accidental duplication of bwc test for script behavior Handle missing values in painless (#30975) (#31903) Build: Make additional test deps of check (#32015) Painless: Fix Bug with Duplicate PainlessClasses (#32110) Adjust translog after versionType removed in 7.0 (#32020) Disable C2 from using AVX-512 on JDK 10 (#32138) [Rollup] Add new capabilities endpoint for concrete rollup indices (#32111) Mute :qa:mixed-cluster indices.stats/10_index/Index - all’ [ML] Wait for aliases in multi-node tests (#32086) Ensure to release translog snapshot in primary-replica resync (#32045) Docs: Fix missing example script quote (#32010) Add Index UUID to `/_stats` Response (#31871) (#32113) [ML] Move analyzer dependencies out of categorization config (#32123) [ML][DOCS] Add missing 6.3.0 release notes (#32099) Updates the build to gradle 4.9 (#32087) Update monitoring template version to 6040099 (#32088) Fix put mappings java API documentation (#31955) Add exclusion option to `keep_types` token filter (#32012)

@return

* master: Painless: Simplify Naming in Lookup Package (#32177) Handle missing values in painless (#32207) add support for write index resolution when creating/updating documents (#31520) ECS Task IAM profile credentials ignored in repository-s3 plugin (#31864) Remove indication of future multi-homing support (#32187) Rest test - allow for snapshots to take 0 milliseconds Make x-pack-core generate a pom file Rest HL client: Add put watch action (#32026) Build: Remove pom generation for plugin zip files (#32180) Fix comments causing errors with Java 11 Fix rollup on date fields that don't support epoch_millis (#31890) Detect and prevent configuration that triggers a Gradle bug (#31912) [test] port linux package packaging tests (#31943) Revert "Introduce a Hashing Processor (#31087)" (#32178) Remove empty @return from JavaDoc Adjust SSLDriver behavior for JDK11 changes (#32145) [test] use randomized runner in packaging tests (#32109) Add support for field aliases. (#32172) Painless: Fix caching bug and clean up addPainlessClass. (#32142) Call setReferences() on custom referring tokenfilters in _analyze (#32157) Fix BwC Tests looking for UUID Pre 6.4 (#32158) Improve docs for search preferences (#32159) use before instead of onOrBefore Add more contexts to painless execute api (#30511) Add EC2 credential test for repository-s3 (#31918) A replica can be promoted and started in one cluster state update (#32042) Fix Java 11 javadoc compile problem Fix CP for namingConventions when gradle home has spaces (#31914) Fix `range` queries on `_type` field for singe type indices (#31756) [DOCS] Update TLS on Docker for 6.3 (#32114) ESIndexLevelReplicationTestCase doesn't support replicated failures but it's good to know what they are Remove versionType from translog (#31945) Switch distribution to new style Requests (#30595) Build: Skip jar tests if jar disabled Painless: Add PainlessClassBuilder (#32141) Build: Make additional test deps of check (#32015) Disable C2 from using AVX-512 on JDK 10 (#32138) Build: Move shadow customizations into common code (#32014) Painless: Fix Bug with Duplicate PainlessClasses (#32110) Remove empty @param from Javadoc Re-disable packaging tests on suse boxes Docs: Fix missing example script quote (#32010) [ML] Wait for aliases in multi-node tests (#32086) [ML] Move analyzer dependencies out of categorization config (#32123) Ensure to release translog snapshot in primary-replica resync (#32045) Handle TokenizerFactory TODOs (#32063) Relax TermVectors API to work with textual fields other than TextFieldType (#31915) Updates the build to gradle 4.9 (#32087) Mute :qa:mixed-cluster indices.stats/10_index/Index - all’ Check that client methods match API defined in the REST spec (#31825) Enable testing in FIPS140 JVM (#31666) Fix put mappings java API documentation (#31955) Add exclusion option to `keep_types` token filter (#32012) [Test] Modify assert statement for ssl handshake (#32072)

cdahlqvist · 2018-09-21T13:47:58Z

A hashing function that like Logstash fingerprint filter supports MURMUR3, MD5, SHA1 and SHA256 would be very useful when creating pipelines that can avoid duplicates without requiring Logstash in the mix. For this, the secrecy of the key is probably less important than when hashing from a security perspective.

It is useful to have a processor similar to logstash-filter-fingerprint in Elasticsearch. A processor that leverages a variety of hashing algorithms to create cryptographically-secure one-way hashes of values in documents. This processor introduces a pbkdf2hmac hashing scheme to fields in documents for indexing

AydinChavez · 2020-11-16T10:28:31Z

There are concerns that this implementation as was merged can easily lead to incorrect usage without clear feedback. For example, secret keys may be inconsistent between ESKeyStores which will lead to inconsistent hashing across ingests depending on which node operated on the documents.

For this reason, and others, This change is to be reverted.
PR for 6.x(6.4): #32179
PR for master: #32178

Is there any progress on this topic? Incorrect usage is quite a broad topic and configuration issues can arise anywhere. I don't think it should block the progress of this one. I mean as long as the functional part is working it should be fine..

danhermann · 2021-02-09T11:38:24Z

Note that a processor suitable for content fingerprinting was added in #68415 though it is not designed for the content anonymization use case.

talevy added WIP :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP labels Jun 5, 2018

talevy added the >feature label Jun 5, 2018

talevy force-pushed the hash-processor branch from 6213419 to 51698c6 Compare June 5, 2018 15:13

talevy added a commit to talevy/elasticsearch that referenced this pull request Jun 13, 2018

move security plugins to a sub ingest directory

04f0c9a

It makes sense to introduce new Security ingest processors (example: elastic#31087), and this change would give them a good place to be written.

talevy mentioned this pull request Jun 13, 2018

move security ingest processors to a sub ingest directory #31306

Merged

talevy added a commit that referenced this pull request Jun 13, 2018

move security ingest processors to a sub ingest directory (#31306)

bbb7889

It makes sense to introduce new Security ingest processors (example: #31087), and this change would give them a good place to be written.

talevy added a commit that referenced this pull request Jun 13, 2018

move security ingest processors to a sub ingest directory (#31306)

d72a6bd

It makes sense to introduce new Security ingest processors (example: #31087), and this change would give them a good place to be written.

talevy force-pushed the hash-processor branch 3 times, most recently from 2494c30 to b7e6898 Compare June 18, 2018 23:23

talevy force-pushed the hash-processor branch from b7e6898 to 0e77233 Compare June 26, 2018 00:16

talevy requested review from jkakavas and martijnvg June 26, 2018 00:36

talevy added review and removed WIP labels Jun 26, 2018

martijnvg reviewed Jun 27, 2018

View reviewed changes

talevy added 2 commits June 27, 2018 21:03

Merge branch 'master' into hash-processor

c6f9ff6

add rest test, and add two params for salt & ignore_missing

80ca1ee

martijnvg approved these changes Jun 28, 2018

View reviewed changes

talevy added 2 commits June 28, 2018 06:25

Merge branch 'master' into hash-processor

514664e

fix key resolution

1676aa9

jkakavas reviewed Jun 28, 2018

View reviewed changes

talevy deleted the hash-processor branch June 29, 2018 16:30

talevy added the v7.0.0 label Jun 29, 2018

talevy mentioned this pull request Jun 29, 2018

Write documentation for Security's HashProcessor #31694

Closed

talevy added v6.4.0 and removed review labels Jun 29, 2018

talevy mentioned this pull request Jul 17, 2018

add documentation for Security's Hash Processor #32112

Closed

talevy added a commit to talevy/elasticsearch that referenced this pull request Jul 18, 2018

Revert "Introduce a Hashing Processor (elastic#31087)"

62c659b

This reverts commit 8c78fe7.

talevy added a commit to talevy/elasticsearch that referenced this pull request Jul 18, 2018

Revert "Introduce a Hashing Processor (elastic#31087)"

55c4bc2

This reverts commit b296936.

talevy removed v6.4.0 v7.0.0 labels Jul 18, 2018

talevy added a commit that referenced this pull request Jul 18, 2018

Revert "Introduce a Hashing Processor (#31087)" (#32179)

161f5ae

This reverts commit b296936.

talevy added a commit that referenced this pull request Jul 18, 2018

Revert "Introduce a Hashing Processor (#31087)" (#32178)

27d663b

This reverts commit 8c78fe7.

talevy mentioned this pull request Aug 9, 2018

Create new secure setting type that is stored (encrypted) in cluster state #32727

Closed

jakelandis mentioned this pull request Sep 26, 2018

[ingest] Re-introduce the Hash Processor with consistent keys across all nodes #34085

Closed

danhermann mentioned this pull request Sep 24, 2019

[WIP] Re-introduce hash processor #47047

Closed

danhermann mentioned this pull request Mar 16, 2020

Add "fingerprint" ingest processor #53578

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce a Hashing Processor #31087

Introduce a Hashing Processor #31087

talevy commented Jun 5, 2018 •

edited

Loading

elasticmachine commented Jun 5, 2018

joshbressers commented Jun 5, 2018

gingerwizard commented Jun 9, 2018

talevy commented Jun 13, 2018 •

edited by jasontedor

Loading

martijnvg left a comment

martijnvg Jun 27, 2018

martijnvg Jun 27, 2018

talevy commented Jun 27, 2018

talevy commented Jun 28, 2018 •

edited

Loading

martijnvg left a comment

jkakavas Jun 28, 2018

talevy Jun 28, 2018

talevy commented Jul 18, 2018

cdahlqvist commented Sep 21, 2018 •

edited

Loading

AydinChavez commented Nov 16, 2020

danhermann commented Feb 9, 2021

Introduce a Hashing Processor #31087

Introduce a Hashing Processor #31087

Conversation

talevy commented Jun 5, 2018 • edited Loading

elasticmachine commented Jun 5, 2018

joshbressers commented Jun 5, 2018

gingerwizard commented Jun 9, 2018

talevy commented Jun 13, 2018 • edited by jasontedor Loading

martijnvg left a comment

Choose a reason for hiding this comment

martijnvg Jun 27, 2018

Choose a reason for hiding this comment

martijnvg Jun 27, 2018

Choose a reason for hiding this comment

talevy commented Jun 27, 2018

talevy commented Jun 28, 2018 • edited Loading

martijnvg left a comment

Choose a reason for hiding this comment

jkakavas Jun 28, 2018

Choose a reason for hiding this comment

talevy Jun 28, 2018

Choose a reason for hiding this comment

talevy commented Jul 18, 2018

cdahlqvist commented Sep 21, 2018 • edited Loading

AydinChavez commented Nov 16, 2020

danhermann commented Feb 9, 2021

talevy commented Jun 5, 2018 •

edited

Loading

talevy commented Jun 13, 2018 •

edited by jasontedor

Loading

talevy commented Jun 28, 2018 •

edited

Loading

cdahlqvist commented Sep 21, 2018 •

edited

Loading