-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
usdt: Add helpers to set semaphore values #2953
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
While debugging a high memory consumption issue in bpftrace, I noticed that a USDT::Context object can take ~10M per instance [0]. Along with the new --usdt-file-activation feature in bpftrace ( bpftrace/bpftrace#1317 ), bpftrace can potentially hold onto many dozens of USDT:Context instances, causing memory issues. While reducing the amount of memory USDT::Context uses is one option, we can potentially side step it by allowing the usdt semaphore count to be set independently. Before, the only way to increment the count (by 1) is to call bcc_usdt_enable*(). bcc_usdt_enable*() has checks that limit it to a single increment per context. The only way to decrement the count is by calling bcc_usdt_close() which naturally only allows for one decrement. With independent semaphore helpers, we can avoid holding onto a USDT::Context instance for the lifetime of the tracing session. We can simply: 1. create a USDT::Context 2. increment the semaphore count for the probe we care about 3. destroy the USDT::Context 4. repeat 1-3 for all probes we want to attach to 5. do our tracing 6. create a USDT::Context for the probe we care about 7. decrement the semaphore count 8. destroy the USDT::Context 9. repeat 6-8 for all the probes we're attached to This approach also has the benefit of 1 USDT::Context instance being alive at a time which can help keep memory high watermark low. [0]: Through gdb single stepping and /proc/pid/status. Exact process is not described here b/c memory usage probably varies based on tracee binary.
[buildbot, test this please] |
[buildbot, ok to test] |
LGTM. Thanks! |
danobi
added a commit
to danobi/bpftrace
that referenced
this pull request
Jun 5, 2020
While debugging a high memory consumption issue, I noticed that a USDT::Context object can take ~10M per instance [0]. Along with the new --usdt-file-activation, bpftrace can potentially hold onto many dozens of USDT:Context instances, causing memory issues. This patch makes use of the new bcc usdt semaphore helpers ( iovisor/bcc#2953 ) to avoid holding onto USDT::Context instances for the lifetime of the tracing session. The new algorithm is: 1. create a USDT::Context 2. increment the semaphore count for the probe we care about 3. destroy the USDT::Context 4. repeat 1-3 for all probes we want to attach to 5. do our tracing 6. create a USDT::Context for the probe we care about 7. decrement the semaphore count 8. destroy the USDT::Context 9. repeat 6-8 for all the probes we're attached to This approach also has the benefit of 1 USDT::Context instance being alive at a time which can help keep memory high watermark low. [0]: Through gdb single stepping and /proc/pid/status. Exact process is not described here b/c memory usage probably varies based on tracee binary.
2 tasks
danobi
added a commit
to danobi/bpftrace
that referenced
this pull request
Jun 5, 2020
While debugging a high memory consumption issue, I noticed that a USDT::Context object can take ~10M per instance [0]. Along with the new --usdt-file-activation, bpftrace can potentially hold onto many dozens of USDT:Context instances, causing memory issues. This patch makes use of the new bcc usdt semaphore helpers ( iovisor/bcc#2953 ) to avoid holding onto USDT::Context instances for the lifetime of the tracing session. The new algorithm is: 1. create a USDT::Context 2. increment the semaphore count for the probe we care about 3. destroy the USDT::Context 4. repeat 1-3 for all probes we want to attach to 5. do our tracing 6. create a USDT::Context for the probe we care about 7. decrement the semaphore count 8. destroy the USDT::Context 9. repeat 6-8 for all the probes we're attached to This approach also has the benefit of 1 USDT::Context instance being alive at a time which can help keep memory high watermark low. [0]: Through gdb single stepping and /proc/pid/status. Exact process is not described here b/c memory usage probably varies based on tracee binary.
danobi
added a commit
to danobi/bpftrace
that referenced
this pull request
Jun 15, 2020
While debugging a high memory consumption issue, I noticed that a USDT::Context object can take ~10M per instance [0]. Along with the new --usdt-file-activation, bpftrace can potentially hold onto many dozens of USDT:Context instances, causing memory issues. This patch makes use of the new bcc usdt semaphore helpers ( iovisor/bcc#2953 ) to avoid holding onto USDT::Context instances for the lifetime of the tracing session. The new algorithm is: 1. create a USDT::Context 2. increment the semaphore count for the probe we care about 3. destroy the USDT::Context 4. repeat 1-3 for all probes we want to attach to 5. do our tracing 6. create a USDT::Context for the probe we care about 7. decrement the semaphore count 8. destroy the USDT::Context 9. repeat 6-8 for all the probes we're attached to This approach also has the benefit of 1 USDT::Context instance being alive at a time which can help keep memory high watermark low. [0]: Through gdb single stepping and /proc/pid/status. Exact process is not described here b/c memory usage probably varies based on tracee binary.
danobi
added a commit
to bpftrace/bpftrace
that referenced
this pull request
Jun 18, 2020
While debugging a high memory consumption issue, I noticed that a USDT::Context object can take ~10M per instance [0]. Along with the new --usdt-file-activation, bpftrace can potentially hold onto many dozens of USDT:Context instances, causing memory issues. This patch makes use of the new bcc usdt semaphore helpers ( iovisor/bcc#2953 ) to avoid holding onto USDT::Context instances for the lifetime of the tracing session. The new algorithm is: 1. create a USDT::Context 2. increment the semaphore count for the probe we care about 3. destroy the USDT::Context 4. repeat 1-3 for all probes we want to attach to 5. do our tracing 6. create a USDT::Context for the probe we care about 7. decrement the semaphore count 8. destroy the USDT::Context 9. repeat 6-8 for all the probes we're attached to This approach also has the benefit of 1 USDT::Context instance being alive at a time which can help keep memory high watermark low. [0]: Through gdb single stepping and /proc/pid/status. Exact process is not described here b/c memory usage probably varies based on tracee binary.
sravyamks
pushed a commit
to sravyamks/bpftrace
that referenced
this pull request
Aug 7, 2020
While debugging a high memory consumption issue, I noticed that a USDT::Context object can take ~10M per instance [0]. Along with the new --usdt-file-activation, bpftrace can potentially hold onto many dozens of USDT:Context instances, causing memory issues. This patch makes use of the new bcc usdt semaphore helpers ( iovisor/bcc#2953 ) to avoid holding onto USDT::Context instances for the lifetime of the tracing session. The new algorithm is: 1. create a USDT::Context 2. increment the semaphore count for the probe we care about 3. destroy the USDT::Context 4. repeat 1-3 for all probes we want to attach to 5. do our tracing 6. create a USDT::Context for the probe we care about 7. decrement the semaphore count 8. destroy the USDT::Context 9. repeat 6-8 for all the probes we're attached to This approach also has the benefit of 1 USDT::Context instance being alive at a time which can help keep memory high watermark low. [0]: Through gdb single stepping and /proc/pid/status. Exact process is not described here b/c memory usage probably varies based on tracee binary.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While debugging a high memory consumption issue in bpftrace, I noticed
that a USDT::Context object can take ~10M per instance [0]. Along with
the new --usdt-file-activation feature in bpftrace
( bpftrace/bpftrace#1317 ), bpftrace can
potentially hold onto many dozens of USDT:Context instances, causing
memory issues.
While reducing the amount of memory USDT::Context uses is one option,
we can potentially side step it by allowing the usdt semaphore count to
be set independently. Before, the only way to increment the count (by 1)
is to call bcc_usdt_enable*(). bcc_usdt_enable*() has checks that limit
it to a single increment per context. The only way to decrement the
count is by calling bcc_usdt_close() which naturally only allows for
one decrement.
With independent semaphore helpers, we can avoid holding onto a
USDT::Context instance for the lifetime of the tracing session. We can
simply:
This approach also has the benefit of 1 USDT::Context instance being
alive at a time which can help keep memory high watermark low.
[0]: Through gdb single stepping and /proc/pid/status. Exact process is
not described here b/c memory usage probably varies based on tracee
binary.