Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enrichment transformer tracking #224

Open
KristianLyng opened this issue Apr 5, 2022 · 1 comment
Open

Enrichment transformer tracking #224

KristianLyng opened this issue Apr 5, 2022 · 1 comment
Labels
help wanted Extra attention is needed Transformer

Comments

@KristianLyng
Copy link
Collaborator

KristianLyng commented Apr 5, 2022

I've added an experimental enrichment transformer, which allows an operator to use skogul to add fields to metrics based on metadata. E.g.: Add description to interface names from a database.

This is the simplest possible implementation I could make, and will definitely change as the idea matures. As such, the documentation is somewhat deliberately spotty. Instead, use this issue until things mature.

It currently works by parsing a JSON document consisting of an array of metrics, though timestamp is ignored. See docs/examples/payload/enrich.json for an example, and docs/examples/tester_to_stdout_enrich.json for config example.

Each metric will require metadata, which is what will be matched, and data, which is what will be added to the metadata struct. I appreciate that this is somewhat non-intuitive, and it was mainly done for convenience, and I am guessing it wont actually be a Metric data structure in the future.

Additionally, only metadatafields listed in the "keys" is actually used, which is an obvious short coming, but necessary.

I see numerous places where this needs to be improved, but I wanted to get something out fast, to test with production data/traffic before I start optimizing and improving things.

Please use this thread for general feedback. For obvious bugs, separate issues are fine too of course.

@KristianLyng KristianLyng added help wanted Extra attention is needed Transformer labels Apr 5, 2022
@KristianLyng
Copy link
Collaborator Author

I've added some updates:

  1. The enricher transformer no longer has a source field or any local method of updating itself
  2. The enricherUpdater sender is added which updates the enricher transformer (ok, slightly confusing, but not THAT bad)
  3. Examples are updated
  4. The WholeData receiver can be used to replace the Source-field. The example provided demonstrates this.

One annoyance that's left is the need for timestamps that are never used.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed Transformer
Projects
None yet
Development

No branches or pull requests

1 participant