Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

libbeat/processors/add_process_metadata: implement a process cache eviction policy #29717

Merged
merged 1 commit into from
Jan 11, 2022
Merged

libbeat/processors/add_process_metadata: implement a process cache eviction policy #29717

merged 1 commit into from
Jan 11, 2022

Conversation

efd6
Copy link
Contributor

@efd6 efd6 commented Jan 6, 2022

What does this PR do?

The eviction policy implemented here guarantees a hard limit to the number of entries
in the cache and provides a downward force on the number of entries over time with an
exponential decay on the fraction of entries that are older than the cache expiration
duration.

Why is it important?

This fixes a memory leak in auditbeat on systems with high kernel.pid_max.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

N/A

How to test this PR locally

Run go test in libbeat/processors/add_process_metadata.

Related issues

Use cases

N/A

Screenshots

N/A

Logs

N/A

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jan 6, 2022
@mergify
Copy link
Contributor

mergify bot commented Jan 6, 2022

This pull request does not have a backport label. Could you fix it @efd6? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

@mergify mergify bot added the backport-skip Skip notification from the automated backport with mergify label Jan 6, 2022
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jan 6, 2022
@mergify mergify bot removed the backport-skip Skip notification from the automated backport with mergify label Jan 6, 2022
@efd6 efd6 added 8.1-candidate backport-skip Skip notification from the automated backport with mergify and removed 8.0-candidate backport-skip Skip notification from the automated backport with mergify labels Jan 6, 2022
@efd6 efd6 marked this pull request as ready for review January 6, 2022 03:36
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@elasticmachine
Copy link
Collaborator

elasticmachine commented Jan 6, 2022 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-01-09T21:12:30.240+0000

  • Duration: 128 min 34 sec

  • Commit: 44c5c32

Test stats 🧪

Test Results
Failed 0
Passed 47839
Skipped 4277
Total 52116

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@mergify
Copy link
Contributor

mergify bot commented Jan 8, 2022

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b cacheeviction upstream/cacheeviction
git merge upstream/master
git push upstream cacheeviction

@efd6
libbeat/processors/add_process_metadata: implement a process cache ev…
44c5c32 
…iction policy

The eviction policy implemented here guarantees a hard limit to the number of entries
in the cache and provides a downward force on the number of entries over time with an
exponential decay on the fraction of entries that are older than the cache expiration
duration.
@efd6 efd6 requested a review from a team January 11, 2022 09:54
andrewkroh
andrewkroh approved these changes Jan 11, 2022
View reviewed changes
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@efd6 efd6 merged commit 9bd989b into elastic:master Jan 11, 2022
@efd6 efd6 deleted the cacheeviction branch January 11, 2022 20:14
mergify bot pushed a commit that referenced this pull request Jan 11, 2022
@efd6
libbeat/processors/add_process_metadata: implement a process cache ev…
516ee6e 
…iction policy (#29717)

The eviction policy implemented here guarantees a hard limit to the number of entries
in the cache and provides a downward force on the number of entries over time with an
exponential decay on the fraction of entries that are older than the cache expiration
duration.

(cherry picked from commit 9bd989b)
@mergify mergify bot mentioned this pull request Jan 11, 2022
Merged
efd6 added a commit that referenced this pull request Jan 11, 2022
@mergify @efd6
libbeat/processors/add_process_metadata: implement a process cache ev…
bd1ffa5 
…iction policy (#29717) (#29797)

The eviction policy implemented here guarantees a hard limit to the number of entries
in the cache and provides a downward force on the number of entries over time with an
exponential decay on the fraction of entries that are older than the cache expiration
duration.

(cherry picked from commit 9bd989b)

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>
v1v added a commit to v1v/beats that referenced this pull request Jan 12, 2022
@v1v
Merge remote-tracking branch 'upstream/master' into feature/add-githu…
0bab1d5 
…b-for-macos

* upstream/master: (172 commits)
  [Elastic Agent] Fix issue with ensureServiceToken. (elastic#29800)
  [Winlogbeat] Add provider name to Security routing pipeline check (elastic#29781)
  Add summary to journeys which don't emit journey:end (early node subprocess exits) (elastic#29606)
  Prepare 8.0.0-rc1 changelog (elastic#29795) (elastic#29806)
  Change docker image from CentOS 7 to Ubuntu 20.04 (elastic#29681)
  libbeat/processors/add_process_metadata: implement a process cache eviction policy (elastic#29717)
  [Automation] Update elastic stack version to 8.1.0-7004acda for testing (elastic#29783)
  Missing changelog entry for elastic#29773 (elastic#29791)
  Add a readme for k8s autodiscover provider (elastic#28213)
  Remove overriding of index pattern on the Kubernetes overview dashboard (elastic#29676)
  jjbb: remove obsoleted branches (<7.16) (elastic#29707)
  Add k8s metadata in state_cronjob metricset (elastic#29572)
  ibmmq: Fix timestamp parsing (elastic#29773)
  Do not add date to index if `@meta.index` is set (elastic#29775)
  ci: uses aliases for the branches (elastic#29706)
  Filebeat tests: Restore `@timestamp` field validation (elastic#29772)
  Forward port 7.16.3 changelog to master (elastic#29777)
  auditd: Store program arguments in process.args array (elastic#29601)
  System/socket: Support kernel_clone() replacement for _do_fork() (elastic#29744)
  Do not mention removal if version is not specified in `cfgwarn` messages (elastic#29727)
  ...
@adriansr adriansr added the backport-7.17 Automated backport to the 7.17 branch with mergify label Aug 2, 2022
@mergify mergify bot mentioned this pull request Aug 2, 2022
Merged
mergify bot pushed a commit that referenced this pull request Aug 2, 2022
@efd6 @mergify
libbeat/processors/add_process_metadata: implement a process cache ev…
7ea23f8 
…iction policy (#29717)

The eviction policy implemented here guarantees a hard limit to the number of entries
in the cache and provides a downward force on the number of entries over time with an
exponential decay on the fraction of entries that are older than the cache expiration
duration.

(cherry picked from commit 9bd989b)
efd6 added a commit that referenced this pull request Aug 3, 2022
@mergify @efd6
[7.17](backport #29717) libbeat/processors/add_process_metadata: impl…
bd114fc 
…ement a process cache eviction policy (#32579)

* libbeat/processors/add_process_metadata: implement a process cache eviction policy (#29717)

The eviction policy implemented here guarantees a hard limit to the number of entries
in the cache and provides a downward force on the number of entries over time with an
exponential decay on the fraction of entries that are older than the cache expiration
duration.

(cherry picked from commit 9bd989b)

* remove irrelevant changelog lines

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>
Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewkroh andrewkroh andrewkroh approved these changes

Assignees
No one assigned
Labels
8.1-candidate backport-7.17 Automated backport to the 7.17 branch with mergify backport-v8.0.0 Automated backport with mergify bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Memory leak with add_process_metadata and k8s manifest for Auditbeat
4 participants
@efd6 @elasticmachine @andrewkroh @adriansr