Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add Secret Filtering component #1593

Merged
merged 22 commits into from
Sep 24, 2024
Merged

feat: Add Secret Filtering component #1593

merged 22 commits into from
Sep 24, 2024

Conversation

romain-gaillard
Copy link
Contributor

@romain-gaillard romain-gaillard commented Aug 30, 2024

PR Description

This PR adds a secret filtering component for Loki into Alloy. It is based on the work we have done in the August 2024 hackathon.

Which issue(s) this PR fixes

Notes to the Reviewer

PR Checklist

  • CHANGELOG.md updated
  • Documentation added
  • Tests updated
  • Config converters updated

@CLAassistant
Copy link

CLAassistant commented Aug 30, 2024

CLA assistant check
All committers have signed the CLA.

@romain-gaillard romain-gaillard self-assigned this Sep 3, 2024
@kelnage kelnage self-requested a review September 5, 2024 07:40
@romain-gaillard romain-gaillard changed the title Secret Filtering component for Alloy feat: Add Secret Filtering component Sep 5, 2024
@romain-gaillard romain-gaillard marked this pull request as ready for review September 6, 2024 16:39
//
// For the first case, we can replace the entire match with the redaction string.
// For the second case, we can replace the first submatch with the redaction string (to avoid redacting something else than the secret such as delimiters).
for _, occ := range r.regex.FindAllStringSubmatch(entry.Line, -1) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think we can split this block into a separate func, lots of deep nesting going on.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed in d8fa6e5.

Copy link
Contributor

@clayton-cornell clayton-cornell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some initial (minor) doc suggestions

Copy link
Contributor

@clayton-cornell clayton-cornell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some initial (minor) doc suggestions

@clayton-cornell clayton-cornell added the type/docs Docs Squad label across all Grafana Labs repos label Sep 6, 2024
@romain-gaillard
Copy link
Contributor Author

Some initial (minor) doc suggestions

Thanks a lot! I addressed your suggestions in 500891f 😄

Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com>
Copy link
Contributor

@wildum wildum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the changes, only two small nits left.
Did you try to run a pipeline with this component at scale to see what the performance impact is?

internal/component/loki/secretfilter/secretfilter.go Outdated Show resolved Hide resolved
internal/component/loki/secretfilter/secretfilter.go Outdated Show resolved Hide resolved
@romain-gaillard
Copy link
Contributor Author

Thanks for making the changes, only two small nits left. Did you try to run a pipeline with this component at scale to see what the performance impact is?

Done, thanks.

We've done basic benchmarks and found that the overhead for redaction is less than a ms per log entry on a standard laptop, with all secrets enabled.
We have subsequent tickets to more closely analyse and improve the performance of the component in the future when it's used in anger and we have more accurate data and profiling enabled.

Right now, our focus is to get it merged so that we can start to use it to redact secrets internally for security purposes.

@wildum wildum merged commit 21bbe5d into main Sep 24, 2024
18 checks passed
@wildum wildum deleted the secret-filtering branch September 24, 2024 11:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/docs Docs Squad label across all Grafana Labs repos
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants