-
Notifications
You must be signed in to change notification settings - Fork 195
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Data Prepper support for dynamic renaming of keys #4849
Comments
Thank you @soghoyanaws for raising this issue. If I understand the problem, the key itself has no well-defined name that we can use. Correct? If so, we need to support mutating the key name more dynamically. The number of possible key names could be just as varied as the possible values in them. In this case it is a key that starts with a value. In other situations it may be a JSON string. |
Hi @dlvenable , correct, the key name is dynamic since it depends on UUID. |
Thanks for opening this issue for us @soghoyanaws (this is a problem from my company 😬). I personally would have liked to rename these keys but unfortunately that would be prohibitively expensive due to the size of the DynamoDB table. Using regexp matching for target keynames in processors across the board would make all processors much more powerful. This something I was wishing for in many cases. |
Here is a possible solution using the existing
Another option we may consider is the ability to rename using the captured group. This could allow us to support more dynamic solutions. For example, perhaps the key can start with either
These two solutions do not need to be in conflict either. We could allow both. Either providing We are also considering using a different processor instead. |
@dlvenable that sounds like a great solution |
Is your feature request related to a problem? Please describe.
In our DynamoDB (DDB) table, we have documents that have fields like this:
Where the string behind the
|
is a random string (UUID) and the value is a float (representing a timestamp). We'd like to extract the value from this field in DDB and index it in OpenSearch as simplydelivered_at: 1722622017.993797
Describe the solution you'd like
Is it possible to plugin custom code, that will
Describe alternatives you've considered (Optional)
There is a
rename_keys
processor (https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/processors/rename-keys/). But it currently can only handle static key names. So with this config, pipeline can rename the key, but if the uuid changes, rename won't work.processor: - rename_keys: entries: - from_key: "delivered_at|d87e56e8-f52f-474f-ad18-155b2a08f680" to_key: "delivered_at"
> -Tried a number of different configurations with the grok pipeline processor and none of them have worked. The challenge is that some of the pipeline processors support Pipeline Expressions, while others do not, and it's not well-documented which ones do and which ones don't.
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered: