Optimize audit logging #14596
Merged
Optimize audit logging #14596
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
k8s-ci-robot
added
cncf-cla: yes
k8s-ci-robot
added
the
approved
|
/ok-to-test |
k8s-ci-robot
added
the
ok-to-test
medyagh
reviewed
Jul 15, 2022
There was a problem hiding this comment.
I wonder if there is a way we could test this ?
like could we make a fast minikube command run 10000 times in a for loop? and then expect the audit file to be not more than X ?
for that test we could use a separate MINIKUBE_HOME folder to not mess up other test audit
|
These are the flake rates of all failed tests.
Too many tests failed - See test logs for more details. To see the flake rates of all tests by environment, click here. |
|
/retest-this-please |
|
kvm2 driver with docker runtime Times for minikube start: 58.4s 56.5s 55.6s 56.8s 56.7s Times for minikube (PR 14596) ingress: 29.1s 25.0s 30.0s 30.1s 25.1s docker driver with docker runtime Times for minikube start: 27.9s 27.1s 27.7s 28.9s 28.2s Times for minikube ingress: 22.4s 22.4s 22.9s 22.9s 22.9s docker driver with containerd runtime Times for minikube ingress: 27.4s 27.4s 26.9s 26.9s 27.0s Times for minikube start: 23.7s 23.7s 23.7s 23.5s 34.4s |
|
These are the flake rates of all failed tests.
Too many tests failed - See test logs for more details. To see the flake rates of all tests by environment, click here. |
This PR only optimizes the audit logging so it's faster, this does not limit the size of the audit log, that can be a follow up PR. |
|
As a part of the approval process, I worked with @spowelljr to test this PR locally.
cat ~/.minikube/logs/audit.json Before - running without update: After - running with update: From 11 minutes (660 seconds) to about 3 seconds. This is an efficient change with a marked 99.5455% decrease in time. |
klaases
approved these changes
Jul 30, 2022
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: klaases, spowelljr The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related #14543
Base Case (empty audit log):
Before (with 5.5 MB audit log):
After (with 5.5 MB audit log):
92.8% reduction
Cause of problem:
Appending the contents to a string variable is what was taking up all the time.
Solution:
Writing to the file every time instead of concatenating to a string resolved the bottleneck.
Also made small changes to the tests, discovered the tests were using the main
audit.jsonfile instead of the tmp file the tests were creating.