A new ottl function/converter to convert telemetry data to its hash value or digest

[rnishtala-sumo]

Component(s)

pkg/ottl

Is your feature request related to a problem? Please describe.

A new ottl function/converter when invoked will transform the underlying telemetry data to a hash value or digest.

Hashing is commonly used for

• data integrity checks
• password storage (where the password itself is not stored but its hash value is)
• verifying data authenticity.

Describe the solution you'd like

The hashing converter/function could be used with the replace_all_matches editor like below

• replace_all_matches(target, pattern, sha256)
• replace_all_matches(target, pattern, sha1)

The replace_all_matches function replaces any matching string value with its hash value.
target is a path expression to a pdata.Map type field. pattern is a string following filepath.Match syntax.sha1 is the hashing function.
Each string value in target that matches pattern will get replaced with its hash value. Non-string values are ignored.
We could define two hash functions for the following two algorithms

• SHA-256
• SHA-1 (the attributes processor uses this)

or with the set editor like below

• set(target, hash(value))

Describe alternatives you've considered

An alternative to using this could be to use the attributes processor

• https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/attributesprocessor

With the hash action

# Key specifies the attribute to act upon.
- key: <key>
  action: hash
  # Rule specifies the regex pattern for attribute names to act upon.
  pattern: <regular pattern>

Additional context

No response

[github-actions]

Pinging code owners:

• pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[TylerHelmuth]

This is functionality we should provide an a necessary step towards eventually replacing the attributes processor with OTTL (#18643).

I think to start providing a Hash Converter function is the most like-for-like replacement of attributesprocessor's functionality. If there is a need to convert all matching keys in a map then I think we should add a new function, like hash_all_patterns, that is specific to that use case.

The converter function should take a StringGetter that represents the value to be hashed. Unless there is a need I'd stick with the attributesprocessor solution and only hash with SHA-1.

[TylerHelmuth]

With @bogdandrutu suggestion we should name the function based on its algorithm like SHA1() or SHA256 as an example.

If we want to hash only parts of the string we can create a function similar to replace_all_patterns that is specific to hashing.

[rnishtala-sumo]

@TylerHelmuth @jpkrohling wanted to confirm, the suggestion was to use to consider using fnv hashing (non-cryptographic) instead of sha1 or sh256?
as suggested here: https://github.com/open-telemetry/opentelemetry-collector/blob/main/CONTRIBUTING.md#recommended-libraries--defaults

[TylerHelmuth]

I think we could provide functions for all the needed hash functions, but based on the link you provided starting with fnv is in line with collector guidelines, but we should also supply SHA265 options to be compatible with the attributes processor.

[rnishtala-sumo]

@TylerHelmuth, a new issue has been created #22787 for addressing this comment above #22725 (comment)

If we want to hash only parts of the string we can create a function similar to replace_all_patterns that is specific to hashing.

I can start working on this issue, i.e work on the following hash functions/converters

• fnv
• sha1
• sha256

and then the new issue created could use these functions. Please let me know your thoughts on this.

[TylerHelmuth]

@rnishtala-sumo with the functions merged into OTTL they are ready to be added to the function lists of the dependent components.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[evan-bradley]

Closing as resolved by #22968. These functions are now provided in OTTL's standard set of Converters.

@TylerHelmuth I see you removed the stale label a few months ago, please reopen this if you don't feel it has been resolved.