Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhancement(observability): Convert BytesReceived to a registered event #13934

Merged
merged 11 commits into from
Aug 26, 2022

Conversation

bruceg
Copy link
Member

@bruceg bruceg commented Aug 10, 2022

This change moves the common BytesReceived into vector-common and converts
it and all uses of it to a registered event.

Ref #13691

@bruceg bruceg added type: enhancement A value-adding code change that enhances its existing functionality. domain: observability Anything related to monitoring/observing Vector domain: sources Anything related to the Vector's sources domain: performance Anything related to Vector's performance source: internal_metrics Anything `internal_metrics` source related labels Aug 10, 2022
@bruceg bruceg self-assigned this Aug 10, 2022
@netlify
Copy link

netlify bot commented Aug 10, 2022

Deploy Preview for vector-project ready!

Name Link
🔨 Latest commit 2530bf8
🔍 Latest deploy log https://app.netlify.com/sites/vector-project/deploys/6307e7f64417c50009d846fc
😎 Deploy Preview https://deploy-preview-13934--vector-project.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@github-actions github-actions bot added domain: ci Anything related to Vector's CI environment domain: sinks Anything related to the Vector's sinks labels Aug 10, 2022
@bruceg bruceg changed the title enhancement(observability): Add BytesReceived to a registered event enhancement(observability): Convert BytesReceived to a registered event Aug 10, 2022
Cargo.toml Outdated Show resolved Hide resolved
@github-actions
Copy link

Soak Test Results

Baseline: c5a94ff
Comparison: 863b1ca
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
syslog_loki 223.58KiB 1.54 100.00% 14.17MiB 523.5KiB 10.71KiB 0 0.0360656 14.39MiB 820.81KiB 16.69KiB 0 0.05569 False False
syslog_log2metric_splunk_hec_metrics 189.99KiB 1.13 100.00% 16.41MiB 940.81KiB 19.17KiB 0 0.0559919 16.59MiB 1.03MiB 21.55KiB 0 0.0623347 False False
syslog_humio_logs 150.22KiB 0.89 100.00% 16.43MiB 142.18KiB 2.9KiB 0 0.00845162 16.57MiB 151.61KiB 3.1KiB 0 0.00893264 False False
syslog_splunk_hec_logs 142.44KiB 0.88 100.00% 15.82MiB 962.26KiB 19.56KiB 0 0.0593964 15.96MiB 870.26KiB 17.73KiB 0 0.0532494 False False
splunk_hec_route_s3 154.43KiB 0.8 98.30% 18.95MiB 2.23MiB 46.38KiB 0 0.117575 19.1MiB 2.15MiB 45.06KiB 0 0.112682 False False
syslog_regex_logs2metric_ddmetrics 33.48KiB 0.27 95.32% 11.97MiB 557.77KiB 11.37KiB 0 0.0454883 12.0MiB 609.28KiB 12.42KiB 0 0.0495536 False False
datadog_agent_remap_blackhole 96.16KiB 0.15 63.31% 60.61MiB 4.01MiB 83.7KiB 0 0.0662134 60.71MiB 3.16MiB 65.96KiB 0 0.0520578 False False
splunk_hec_to_splunk_hec_logs_acks 21.18KiB 0.09 61.72% 23.74MiB 880.9KiB 17.91KiB 0 0.0362249 23.76MiB 804.42KiB 16.37KiB 0 0.0330511 False False
splunk_hec_to_splunk_hec_logs_noack 5.28KiB 0.02 39.00% 23.83MiB 384.42KiB 7.85KiB 0 0.0157496 23.84MiB 329.84KiB 6.74KiB 0 0.0135102 False False
http_to_http_acks -1.59KiB -0.01 0.53% 17.38MiB 8.13MiB 170.01KiB 0 0.467647 17.38MiB 8.17MiB 170.78KiB 0 0.469917 True True
splunk_hec_indexer_ack_blackhole -5.15KiB -0.02 15.23% 23.75MiB 917.63KiB 18.67KiB 0 0.0377231 23.75MiB 945.23KiB 19.23KiB 0 0.0388659 False False
socket_to_socket_blackhole -4.53KiB -0.02 12.45% 22.41MiB 1001.72KiB 20.45KiB 0 0.0436405 22.41MiB 1000.57KiB 20.44KiB 0 0.0435988 False False
enterprise_http_to_http -3.82KiB -0.02 38.93% 23.85MiB 260.69KiB 5.32KiB 0 0.0106732 23.84MiB 258.84KiB 5.3KiB 0 0.0105992 False False
file_to_blackhole -45.35KiB -0.05 31.83% 95.34MiB 3.46MiB 71.82KiB 0 0.036329 95.29MiB 4.04MiB 84.04KiB 0 0.0423847 False False
syslog_log2metric_humio_metrics -13.63KiB -0.1 68.69% 12.79MiB 408.08KiB 8.33KiB 0 0.0311419 12.78MiB 521.71KiB 10.63KiB 0 0.0398543 False False
http_to_http_json -25.08KiB -0.1 95.63% 23.84MiB 360.24KiB 7.35KiB 0 0.0147519 23.82MiB 490.51KiB 10.02KiB 0 0.020107 False False
http_pipelines_blackhole_acks -2.09KiB -0.17 64.28% 1.19MiB 90.18KiB 1.84KiB 0 0.0741811 1.18MiB 65.64KiB 1.34KiB 0 0.0540886 False False
http_to_http_noack -70.92KiB -0.29 99.90% 23.84MiB 402.59KiB 8.23KiB 0 0.0164905 23.77MiB 978.79KiB 19.94KiB 0 0.0402087 False False
fluent_elasticsearch -695.49KiB -0.85 100.00% 79.47MiB 54.55KiB 1.1KiB 0 0.000670194 78.79MiB 6.1MiB 125.13KiB 0 0.0774466 False False
datadog_agent_remap_blackhole_acks -1010.14KiB -1.59 100.00% 61.88MiB 5.4MiB 112.4KiB 0 0.0872106 60.89MiB 4.72MiB 98.75KiB 0 0.0775227 False False
datadog_agent_remap_datadog_logs_acks -1.17MiB -1.9 100.00% 61.64MiB 3.5MiB 73.16KiB 0 0.0567242 60.46MiB 4.69MiB 97.54KiB 0 0.0774832 False False
http_pipelines_no_grok_blackhole -246.65KiB -2.2 100.00% 10.93MiB 597.34KiB 12.2KiB 0 0.0533372 10.69MiB 1.23MiB 25.55KiB 0 0.114765 False False
datadog_agent_remap_datadog_logs -1.93MiB -3.17 100.00% 60.92MiB 3.48MiB 72.93KiB 0 0.0570842 58.99MiB 4.89MiB 101.87KiB 0 0.0828926 False False
http_pipelines_blackhole -59.53KiB -3.72 100.00% 1.56MiB 110.95KiB 2.27KiB 0 0.0693132 1.5MiB 156.37KiB 3.19KiB 0 0.101466 False False
http_text_to_http_json -1.5MiB -3.82 100.00% 39.32MiB 748.58KiB 15.28KiB 0 0.0185869 37.82MiB 895.79KiB 18.29KiB 0 0.0231257 False False

Copy link
Contributor

@tobz tobz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me overall, well done!

I left one comment about not understanding why we still need the handle trait in scope, but it's non-blocking.

src/sinks/azure_common/service.rs Outdated Show resolved Hide resolved
Cargo.toml Outdated Show resolved Hide resolved
@jszwedko jszwedko added the ci-condition: integration tests enable Run integration tests on this PR label Aug 11, 2022
@jszwedko
Copy link
Member

Added label to run integration tests here.

This change moves the common `BytesReceived` into `vector-common` and converts
it and all uses of it to a registered event.
@github-actions github-actions bot removed the domain: sinks Anything related to the Vector's sinks label Aug 12, 2022
@bruceg
Copy link
Member Author

bruceg commented Aug 12, 2022

Rebased to remove the broken BytesSent bits.

Copy link
Member

@jszwedko jszwedko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly approving based on previous approval since I had a hard time spotting any changes.

@github-actions
Copy link

Soak Test Results

Baseline: d40d102
Comparison: 276407a
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
socket_to_socket_blackhole 346.47KiB 1.46 100.00% 23.17MiB 265.45KiB 5.42KiB 0 0.0111847 23.51MiB 163.35KiB 3.34KiB 0 0.00678376 False False
http_text_to_http_json 327.28KiB 0.83 100.00% 38.42MiB 1.01MiB 21.1KiB 0 0.0262695 38.74MiB 999.18KiB 20.4KiB 0 0.025183 False False
http_pipelines_blackhole_acks 7.4KiB 0.64 98.88% 1.14MiB 115.32KiB 2.35KiB 0 0.0990068 1.14MiB 85.03KiB 1.73KiB 0 0.0725454 False False
syslog_regex_logs2metric_ddmetrics 65.39KiB 0.52 100.00% 12.36MiB 602.84KiB 12.28KiB 0 0.0476062 12.43MiB 488.75KiB 9.97KiB 0 0.0383978 False False
datadog_agent_remap_blackhole 267.04KiB 0.41 98.27% 62.9MiB 4.31MiB 89.75KiB 0 0.0684917 63.16MiB 3.22MiB 67.26KiB 0 0.0510435 False False
splunk_hec_to_splunk_hec_logs_noack 20.94KiB 0.09 91.28% 23.82MiB 501.89KiB 10.24KiB 0 0.0205756 23.84MiB 328.32KiB 6.7KiB 0 0.0134483 False False
splunk_hec_indexer_ack_blackhole 17.83KiB 0.07 51.13% 23.74MiB 918.47KiB 18.68KiB 0 0.037771 23.76MiB 871.16KiB 17.73KiB 0 0.0357993 False False
http_pipelines_blackhole 859.69B 0.05 29.26% 1.59MiB 45.9KiB 960.65B 0 0.0282744 1.59MiB 99.57KiB 2.03KiB 0 0.0613019 False False
datadog_agent_remap_blackhole_acks 16.05KiB 0.03 9.87% 59.9MiB 4.76MiB 99.01KiB 0 0.0793968 59.91MiB 3.99MiB 83.39KiB 0 0.0666331 False False
splunk_hec_to_splunk_hec_logs_acks -491.26B -0 1.62% 23.76MiB 817.72KiB 16.64KiB 0 0.0336048 23.76MiB 828.29KiB 16.85KiB 0 0.0340398 False False
enterprise_http_to_http -1.57KiB -0.01 17.32% 23.85MiB 246.7KiB 5.04KiB 0 0.0101005 23.85MiB 249.62KiB 5.11KiB 0 0.0102208 False False
datadog_agent_remap_datadog_logs_acks -9.95KiB -0.02 7.17% 63.04MiB 3.02MiB 63.12KiB 0 0.047874 63.03MiB 4.36MiB 90.81KiB 0 0.0691838 False False
file_to_blackhole -36.54KiB -0.04 28.84% 95.34MiB 3.23MiB 67.0KiB 0 0.0338939 95.3MiB 3.49MiB 72.64KiB 0 0.0366452 False False
http_to_http_json -38.0KiB -0.16 99.55% 23.84MiB 357.91KiB 7.31KiB 0 0.0146557 23.81MiB 549.31KiB 11.21KiB 0 0.0225285 False False
syslog_log2metric_humio_metrics -23.35KiB -0.18 93.60% 12.47MiB 240.74KiB 4.92KiB 0 0.0188511 12.45MiB 570.23KiB 11.6KiB 0 0.0447328 False False
http_to_http_noack -78.23KiB -0.32 99.95% 23.84MiB 412.07KiB 8.43KiB 0 0.0168781 23.76MiB 1.0MiB 20.96KiB 0 0.0422798 False False
http_to_http_acks -58.7KiB -0.33 19.41% 17.44MiB 7.93MiB 165.67KiB 0 0.454461 17.38MiB 8.25MiB 172.06KiB 0 0.474321 True True
fluent_elasticsearch -279.19KiB -0.34 99.97% 79.47MiB 55.71KiB 1.13KiB 0 0.000684376 79.2MiB 3.74MiB 76.95KiB 0 0.0472103 False False
datadog_agent_remap_datadog_logs -311.5KiB -0.48 99.98% 63.46MiB 273.25KiB 5.6KiB 0 0.00420422 63.15MiB 4.02MiB 83.7KiB 0 0.0636003 False False
splunk_hec_route_s3 -147.56KiB -0.78 96.66% 18.37MiB 2.4MiB 49.96KiB 0 0.130563 18.22MiB 2.3MiB 48.09KiB 0 0.126043 False False
http_pipelines_no_grok_blackhole -111.11KiB -0.97 100.00% 11.22MiB 67.83KiB 1.39KiB 0 0.00590136 11.11MiB 971.8KiB 19.78KiB 0 0.0853714 False False
syslog_log2metric_splunk_hec_metrics -172.38KiB -0.98 100.00% 17.15MiB 1023.99KiB 20.86KiB 0 0.0582993 16.98MiB 999.76KiB 20.37KiB 0 0.0574843 False False
syslog_splunk_hec_logs -182.62KiB -1.13 100.00% 15.74MiB 817.64KiB 16.64KiB 0 0.0507333 15.56MiB 732.1KiB 14.92KiB 0 0.0459466 False False
syslog_humio_logs -193.96KiB -1.14 100.00% 16.6MiB 229.3KiB 4.68KiB 0 0.01349 16.41MiB 130.18KiB 2.67KiB 0 0.00774718 False False
syslog_loki -404.15KiB -2.67 100.00% 14.79MiB 363.88KiB 7.45KiB 0 0.0240163 14.4MiB 740.27KiB 15.05KiB 0 0.0501979 False False

Copy link
Contributor

@tobz tobz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👌🏻

src/sources/exec/#mod.rs# Outdated Show resolved Hide resolved
src/sources/opentelemetry/http.rs Outdated Show resolved Hide resolved
@github-actions
Copy link

Soak Test Results

Baseline: 6d712b1
Comparison: a477200
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
http_text_to_http_json 1016.94KiB 2.61 100.00% 38.08MiB 914.5KiB 18.67KiB 0 0.0234445 39.08MiB 884.36KiB 18.06KiB 0 0.0220955 False False
socket_to_socket_blackhole 539.11KiB 2.32 100.00% 22.65MiB 257.02KiB 5.25KiB 0 0.0110792 23.18MiB 202.95KiB 4.14KiB 0 0.00854987 False False
datadog_agent_remap_blackhole_acks 1.31MiB 2.12 100.00% 61.57MiB 4.83MiB 100.63KiB 0 0.0784798 62.88MiB 3.19MiB 66.67KiB 0 0.0506433 False False
syslog_regex_logs2metric_ddmetrics 175.83KiB 1.44 100.00% 11.96MiB 647.9KiB 13.2KiB 0 0.052906 12.13MiB 579.49KiB 11.81KiB 0 0.0466498 False False
splunk_hec_route_s3 210.5KiB 1.08 99.86% 19.0MiB 2.31MiB 48.05KiB 0 0.121376 19.2MiB 2.16MiB 45.08KiB 0 0.112278 False False
datadog_agent_remap_blackhole 573.54KiB 0.94 100.00% 59.63MiB 4.08MiB 85.09KiB 0 0.0684502 60.19MiB 3.11MiB 64.99KiB 0 0.0517172 False False
syslog_log2metric_splunk_hec_metrics 79.08KiB 0.46 97.66% 16.85MiB 1.15MiB 24.05KiB 0 0.0684088 16.93MiB 1.21MiB 25.26KiB 0 0.0715637 False False
http_pipelines_blackhole_acks 5.46KiB 0.44 94.23% 1.22MiB 117.66KiB 2.39KiB 0 0.0940125 1.23MiB 77.98KiB 1.59KiB 0 0.062034 False False
syslog_humio_logs 40.06KiB 0.24 100.00% 16.28MiB 173.87KiB 3.55KiB 0 0.0104254 16.32MiB 195.47KiB 4.0KiB 0 0.0116926 False False
syslog_loki 28.06KiB 0.19 90.25% 14.62MiB 349.02KiB 7.15KiB 0 0.0233046 14.65MiB 754.84KiB 15.34KiB 0 0.0503072 False False
datadog_agent_remap_datadog_logs 56.46KiB 0.09 44.56% 63.02MiB 1.23MiB 25.82KiB 0 0.0195478 63.07MiB 4.42MiB 91.93KiB 0 0.0699942 False False
datadog_agent_remap_datadog_logs_acks 35.09KiB 0.06 22.67% 60.69MiB 3.35MiB 70.02KiB 0 0.0551217 60.72MiB 4.79MiB 99.65KiB 0 0.0788259 False False
splunk_hec_to_splunk_hec_logs_noack 11.41KiB 0.05 69.26% 23.83MiB 435.46KiB 8.89KiB 0 0.0178431 23.84MiB 332.66KiB 6.79KiB 0 0.0136245 False False
splunk_hec_to_splunk_hec_logs_acks 10.39KiB 0.04 32.43% 23.74MiB 878.49KiB 17.87KiB 0 0.0361225 23.76MiB 847.94KiB 17.25KiB 0 0.0348514 False False
enterprise_http_to_http -1.97KiB -0.01 20.67% 23.85MiB 258.62KiB 5.28KiB 0 0.0105884 23.85MiB 260.9KiB 5.34KiB 0 0.0106824 False False
splunk_hec_indexer_ack_blackhole -6.89KiB -0.03 20.02% 23.75MiB 927.56KiB 18.87KiB 0 0.0381378 23.74MiB 961.26KiB 19.55KiB 0 0.0395348 False False
file_to_blackhole -51.82KiB -0.05 46.82% 95.36MiB 2.7MiB 56.03KiB 0 0.0283379 95.31MiB 2.93MiB 61.05KiB 0 0.0307847 False False
http_to_http_json -38.65KiB -0.16 99.74% 23.85MiB 331.73KiB 6.77KiB 0 0.0135818 23.81MiB 533.03KiB 10.89KiB 0 0.0218582 False False
fluent_elasticsearch -187.69KiB -0.23 100.00% 79.47MiB 53.21KiB 1.08KiB 0 0.000653663 79.29MiB 1.67MiB 34.31KiB 0 0.0210144 False False
http_to_http_noack -104.47KiB -0.43 100.00% 23.85MiB 251.53KiB 5.14KiB 0 0.0102992 23.74MiB 1.11MiB 23.13KiB 0 0.0467048 False False
http_to_http_acks -153.15KiB -0.86 48.46% 17.4MiB 8.03MiB 167.76KiB 0 0.461102 17.25MiB 7.91MiB 165.15KiB 0 0.458633 True True
http_pipelines_blackhole -15.47KiB -0.9 100.00% 1.68MiB 52.52KiB 1.07KiB 0 0.0304872 1.67MiB 123.54KiB 2.52KiB 0 0.0723608 False False
syslog_log2metric_humio_metrics -144.78KiB -1.13 100.00% 12.47MiB 519.5KiB 10.6KiB 0 0.0406637 12.33MiB 748.78KiB 15.26KiB 0 0.059282 False False
http_pipelines_no_grok_blackhole -139.34KiB -1.23 100.00% 11.03MiB 45.79KiB 957.13B 0 0.00405157 10.9MiB 1.02MiB 21.33KiB 0 0.0939358 False False
syslog_splunk_hec_logs -238.41KiB -1.44 100.00% 16.13MiB 922.04KiB 18.78KiB 0 0.0558117 15.9MiB 949.23KiB 19.34KiB 0 0.0582994 False False

Copy link
Contributor

@tobz tobz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This all works for me. 👍🏻

@bruceg bruceg enabled auto-merge (squash) August 26, 2022 16:27
@bruceg bruceg dismissed spencergilbert’s stale review August 26, 2022 17:24

Changes were made to address the concerns.

@bruceg bruceg merged commit 573b309 into master Aug 26, 2022
@bruceg bruceg deleted the registered-bytes-received branch August 26, 2022 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci-condition: integration tests enable Run integration tests on this PR domain: ci Anything related to Vector's CI environment domain: observability Anything related to monitoring/observing Vector domain: performance Anything related to Vector's performance domain: sources Anything related to the Vector's sources source: internal_metrics Anything `internal_metrics` source related type: enhancement A value-adding code change that enhances its existing functionality.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants