Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

usdt: Add helpers to set semaphore values #2953

Merged
merged 1 commit into from
Jun 5, 2020

Conversation

danobi
Copy link
Member

@danobi danobi commented Jun 5, 2020

While debugging a high memory consumption issue in bpftrace, I noticed
that a USDT::Context object can take ~10M per instance [0]. Along with
the new --usdt-file-activation feature in bpftrace
( bpftrace/bpftrace#1317 ), bpftrace can
potentially hold onto many dozens of USDT:Context instances, causing
memory issues.

While reducing the amount of memory USDT::Context uses is one option,
we can potentially side step it by allowing the usdt semaphore count to
be set independently. Before, the only way to increment the count (by 1)
is to call bcc_usdt_enable*(). bcc_usdt_enable*() has checks that limit
it to a single increment per context. The only way to decrement the
count is by calling bcc_usdt_close() which naturally only allows for
one decrement.

With independent semaphore helpers, we can avoid holding onto a
USDT::Context instance for the lifetime of the tracing session. We can
simply:

  1. create a USDT::Context
  2. increment the semaphore count for the probe we care about
  3. destroy the USDT::Context
  4. repeat 1-3 for all probes we want to attach to
  5. do our tracing
  6. create a USDT::Context for the probe we care about
  7. decrement the semaphore count
  8. destroy the USDT::Context
  9. repeat 6-8 for all the probes we're attached to

This approach also has the benefit of 1 USDT::Context instance being
alive at a time which can help keep memory high watermark low.

[0]: Through gdb single stepping and /proc/pid/status. Exact process is
not described here b/c memory usage probably varies based on tracee
binary.

While debugging a high memory consumption issue in bpftrace, I noticed
that a USDT::Context object can take ~10M per instance [0]. Along with
the new --usdt-file-activation feature in bpftrace
( bpftrace/bpftrace#1317 ), bpftrace can
potentially hold onto many dozens of USDT:Context instances, causing
memory issues.

While reducing the amount of memory USDT::Context uses is one option,
we can potentially side step it by allowing the usdt semaphore count to
be set independently. Before, the only way to increment the count (by 1)
is to call bcc_usdt_enable*(). bcc_usdt_enable*() has checks that limit
it to a single increment per context. The only way to decrement the
count is by calling bcc_usdt_close() which naturally only allows for
one decrement.

With independent semaphore helpers, we can avoid holding onto a
USDT::Context instance for the lifetime of the tracing session. We can
simply:

1. create a USDT::Context
2. increment the semaphore count for the probe we care about
3. destroy the USDT::Context
4. repeat 1-3 for all probes we want to attach to
5. do our tracing
6. create a USDT::Context for the probe we care about
7. decrement the semaphore count
8. destroy the USDT::Context
9. repeat 6-8 for all the probes we're attached to

This approach also has the benefit of 1 USDT::Context instance being
alive at a time which can help keep memory high watermark low.

[0]: Through gdb single stepping and /proc/pid/status. Exact process is
not described here b/c memory usage probably varies based on tracee
binary.
@yonghong-song
Copy link
Collaborator

[buildbot, test this please]

@yonghong-song
Copy link
Collaborator

[buildbot, ok to test]

@yonghong-song
Copy link
Collaborator

LGTM. Thanks!

@yonghong-song yonghong-song merged commit 126054e into iovisor:master Jun 5, 2020
danobi added a commit to danobi/bpftrace that referenced this pull request Jun 5, 2020
While debugging a high memory consumption issue, I noticed that a
USDT::Context object can take ~10M per instance [0]. Along with the new
--usdt-file-activation, bpftrace can potentially hold onto many dozens
of USDT:Context instances, causing memory issues.

This patch makes use of the new bcc usdt semaphore helpers (
iovisor/bcc#2953 ) to avoid holding onto
USDT::Context instances for the lifetime of the tracing session.

The new algorithm is:

1. create a USDT::Context
2. increment the semaphore count for the probe we care about
3. destroy the USDT::Context
4. repeat 1-3 for all probes we want to attach to
5. do our tracing
6. create a USDT::Context for the probe we care about
7. decrement the semaphore count
8. destroy the USDT::Context
9. repeat 6-8 for all the probes we're attached to

This approach also has the benefit of 1 USDT::Context instance being
alive at a time which can help keep memory high watermark low.

[0]: Through gdb single stepping and /proc/pid/status. Exact process is
not described here b/c memory usage probably varies based on tracee
binary.
danobi added a commit to danobi/bpftrace that referenced this pull request Jun 5, 2020
While debugging a high memory consumption issue, I noticed that a
USDT::Context object can take ~10M per instance [0]. Along with the new
--usdt-file-activation, bpftrace can potentially hold onto many dozens
of USDT:Context instances, causing memory issues.

This patch makes use of the new bcc usdt semaphore helpers (
iovisor/bcc#2953 ) to avoid holding onto
USDT::Context instances for the lifetime of the tracing session.

The new algorithm is:

1. create a USDT::Context
2. increment the semaphore count for the probe we care about
3. destroy the USDT::Context
4. repeat 1-3 for all probes we want to attach to
5. do our tracing
6. create a USDT::Context for the probe we care about
7. decrement the semaphore count
8. destroy the USDT::Context
9. repeat 6-8 for all the probes we're attached to

This approach also has the benefit of 1 USDT::Context instance being
alive at a time which can help keep memory high watermark low.

[0]: Through gdb single stepping and /proc/pid/status. Exact process is
not described here b/c memory usage probably varies based on tracee
binary.
danobi added a commit to danobi/bpftrace that referenced this pull request Jun 15, 2020
While debugging a high memory consumption issue, I noticed that a
USDT::Context object can take ~10M per instance [0]. Along with the new
--usdt-file-activation, bpftrace can potentially hold onto many dozens
of USDT:Context instances, causing memory issues.

This patch makes use of the new bcc usdt semaphore helpers (
iovisor/bcc#2953 ) to avoid holding onto
USDT::Context instances for the lifetime of the tracing session.

The new algorithm is:

1. create a USDT::Context
2. increment the semaphore count for the probe we care about
3. destroy the USDT::Context
4. repeat 1-3 for all probes we want to attach to
5. do our tracing
6. create a USDT::Context for the probe we care about
7. decrement the semaphore count
8. destroy the USDT::Context
9. repeat 6-8 for all the probes we're attached to

This approach also has the benefit of 1 USDT::Context instance being
alive at a time which can help keep memory high watermark low.

[0]: Through gdb single stepping and /proc/pid/status. Exact process is
not described here b/c memory usage probably varies based on tracee
binary.
danobi added a commit to bpftrace/bpftrace that referenced this pull request Jun 18, 2020
While debugging a high memory consumption issue, I noticed that a
USDT::Context object can take ~10M per instance [0]. Along with the new
--usdt-file-activation, bpftrace can potentially hold onto many dozens
of USDT:Context instances, causing memory issues.

This patch makes use of the new bcc usdt semaphore helpers (
iovisor/bcc#2953 ) to avoid holding onto
USDT::Context instances for the lifetime of the tracing session.

The new algorithm is:

1. create a USDT::Context
2. increment the semaphore count for the probe we care about
3. destroy the USDT::Context
4. repeat 1-3 for all probes we want to attach to
5. do our tracing
6. create a USDT::Context for the probe we care about
7. decrement the semaphore count
8. destroy the USDT::Context
9. repeat 6-8 for all the probes we're attached to

This approach also has the benefit of 1 USDT::Context instance being
alive at a time which can help keep memory high watermark low.

[0]: Through gdb single stepping and /proc/pid/status. Exact process is
not described here b/c memory usage probably varies based on tracee
binary.
sravyamks pushed a commit to sravyamks/bpftrace that referenced this pull request Aug 7, 2020
While debugging a high memory consumption issue, I noticed that a
USDT::Context object can take ~10M per instance [0]. Along with the new
--usdt-file-activation, bpftrace can potentially hold onto many dozens
of USDT:Context instances, causing memory issues.

This patch makes use of the new bcc usdt semaphore helpers (
iovisor/bcc#2953 ) to avoid holding onto
USDT::Context instances for the lifetime of the tracing session.

The new algorithm is:

1. create a USDT::Context
2. increment the semaphore count for the probe we care about
3. destroy the USDT::Context
4. repeat 1-3 for all probes we want to attach to
5. do our tracing
6. create a USDT::Context for the probe we care about
7. decrement the semaphore count
8. destroy the USDT::Context
9. repeat 6-8 for all the probes we're attached to

This approach also has the benefit of 1 USDT::Context instance being
alive at a time which can help keep memory high watermark low.

[0]: Through gdb single stepping and /proc/pid/status. Exact process is
not described here b/c memory usage probably varies based on tracee
binary.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants