Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NH-52515 Remove cached custom name at calculate_custom #187

Merged
merged 15 commits into from
Aug 3, 2023

Conversation

tammy-baylis-swi
Copy link
Contributor

@tammy-baylis-swi tammy-baylis-swi commented Aug 2, 2023

EDIT: Also adds several unit tests for inbound metrics processor.

Adds removal of any stored custom transaction name at the end of calculate_custom_transaction_name. It will only be re-added to the manager if span is sampled so that exporter will have access to the name, before removing name again after reporting. This is to stop leaking in the case where span is not sampled but custom transaction name is set (used for inbound metrics reporting). Big thanks to @swi-jared for seeing this!

Step-wise:

  1. Customer code calls set_transaction_name("foo"). Call is successful and manager stores "<trace_id>-<span_id>": "foo".
  2. Otel SDK ends span with that <span_id> and on_end of processors called for all recorded spans, sampled or not (already in place!)
  3. calculate_transaction_names for inbound metrics is called. Helper calculate_custom_transaction_name gets "<trace_id>-<span_id>": "foo" from manager.
  4. New: calculate_custom_transaction_name deletes "<trace_id>-<span_id>": "foo" from manager.
  5. Inbound metrics are generated next in on_end.
  6. If ended span is sampled, cache "<trace_id>-<span_id>": "foo" in manager. This can be a re-cache of custom name or new cache of 'regular' transaction name.
    a. Exporter gets "<trace_id>-<span_id>": "foo" from manager.
    b. Exporter deletes "<trace_id>-<span_id>": "foo" from manager.
  7. If ended span not sampled, there is no (re-)caching.

Test trace from Django A testbed app's "reentry" route that requests its own route and also uses API set_transaction_name -- sw.transaction is custom-name-django-a-root at the root span, and is custom-name-reentry at the SERVER:GET another/ span. Just a quick smoke test to make sure custom txn naming still works: https://my.na-01.cloud.solarwinds.com/140638900734749696/traces/E51FD08B882DA4677B26254C08F5701A/E5C99F94B8B0D44A/details/breakdown?perspective=requests&serviceId=e-1540126275982954496

I made a second request a few minutes later with APM Python hardcoded to always decide RECORD_ONLY. This is to make sure we get metrics blips for the same custom names even if no trace exported:

metrics_naming

@tammy-baylis-swi tammy-baylis-swi marked this pull request as ready for review August 2, 2023 20:53
@tammy-baylis-swi tammy-baylis-swi requested a review from a team August 2, 2023 20:53
Copy link
Contributor

@swi-jared swi-jared left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a test to confirm this behavior?

@tammy-baylis-swi
Copy link
Contributor Author

Can you add a test to confirm this behavior?

Added in 1d7445f. Going to keep going on this PR to add tests for other processor functions!

@tammy-baylis-swi
Copy link
Contributor Author

Thanks @swi-jared ! Lots more unit tests now. Please could you have a look when you have a moment.

mocker.Mock(),
mocker.Mock(),
)
result = processor.calculate_transaction_names(mock_span)

Check notice

Code scanning / CodeQL

Unused local variable

Variable result is not used.
mocker.Mock(),
mocker.Mock(),
)
result = processor.calculate_transaction_names(mock_span)

Check notice

Code scanning / CodeQL

Unused local variable

Variable result is not used.
mocker.Mock(),
mocker.Mock(),
)
result = processor.calculate_transaction_names(mock_span)

Check notice

Code scanning / CodeQL

Unused local variable

Variable result is not used.
Copy link
Contributor

@swi-jared swi-jared left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't originally mean for you to do all this extra work! Well done! 🥇

@tammy-baylis-swi tammy-baylis-swi merged commit 4ddf7e0 into main Aug 3, 2023
@tammy-baylis-swi tammy-baylis-swi deleted the NH-52515-fix-record-only-custom-name-leak branch August 3, 2023 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants