Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(ci): Re-update k8s-e2e to use new helm chart #10567

Conversation

jszwedko
Copy link
Member

Reverts #10566

Re-introduces for debugging.

@jszwedko jszwedko added ci-condition: k8s e2e all targets Run Kubernetes E2E test suite for all targets (instead of just the essential subset) ci-condition: k8s e2e tests enable Run Kubernetes E2E test suite for this PR labels Dec 22, 2021
@netlify
Copy link

netlify bot commented Dec 22, 2021

✔️ Deploy Preview for vector-project canceled.

🔨 Explore the source changes: da7888f

🔍 Inspect the deploy log: https://app.netlify.com/sites/vector-project/deploys/61ca15462b04210008649609

@github-actions github-actions bot added the domain: ci Anything related to Vector's CI environment label Dec 22, 2021
@spencergilbert spencergilbert self-assigned this Dec 22, 2021
@github-actions
Copy link

Soak Test Results

Baseline: 738e6c2
Comparison: 0b682ad
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Test units below are bytes/second/CPU, except for "skewness". The further "skewness" is from 0.0 the more indication that vector lacks consistency in behavior, making predictions of fitness in the field challenging.

The abbreviated table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. The abbreviated table will be omitted if no statistically interesting changes are observed.

No statistically interesting changes with confidence 90.00%.

Fine details of change detection per experiment.
experiment Δ mean Δ mean % baseline mean baseline stdev baseline outlier percentage comparison mean comparison stdev comparison outlier percentage t-statistic p-value erratic
fluent_elasticsearch 1.98MiB 2.95 67.34MiB 2.19MiB 0 69.32MiB 820.76KiB 0 -16.0918 5.64386e-47 False
http_to_http_noack 1.06MiB 4.69 22.54MiB 1.72MiB 1.37741 23.59MiB 1.44MiB 0.316456 -8.72425 2.07506e-17 False
datadog_agent_remap_datadog_logs 221.6KiB 0.59 36.7MiB 352.19KiB 1.38122 36.92MiB 510.27KiB 0 -6.85069 1.69361e-11 False
syslog_regex_logs2metric_ddmetrics 114.46KiB 1.48 7.58MiB 658.5KiB 0 7.69MiB 588.04KiB 0 -2.46683 0.0138657 False
syslog_loki 92.52KiB 1.24 7.26MiB 237.05KiB 0 7.35MiB 163.8KiB 0 -6.10935 1.7348e-09 True
syslog_log2metric_humio_metrics 72.59KiB 0.84 8.39MiB 269.54KiB 0 8.46MiB 71.02KiB 0.826446 -4.50045 9.43692e-06 False
splunk_hec_indexer_ack_blackhole 60.19KiB 0.26 23.02MiB 1.22MiB 0.315457 23.07MiB 1.3MiB 0 -0.605291 0.54519 False
http_pipelines_blackhole_acks 40.04KiB 4.88 820.12KiB 698.99KiB 1.67131 860.15KiB 704.34KiB 1.26582 -0.739551 0.459835 False
datadog_agent_remap_datadog_logs_acks 38.72KiB 0.1 37.73MiB 1.05MiB 0 37.77MiB 566.86KiB 0 -0.604947 0.54546 False
http_datadog_filter_blackhole 29.57KiB 1.33 2.17MiB 911.3KiB 0.552486 2.2MiB 986.24KiB 0.276243 -0.419043 0.67531 False
syslog_log2metric_splunk_hec_metrics 7.67KiB 0.08 9.09MiB 148.12KiB 9.30233 9.1MiB 75.1KiB 0 -0.848459 0.396574 False
http_pipelines_no_grok_blackhole 657.85B 0.03 1.95MiB 981.75KiB 0 1.95MiB 965.67KiB 3.59116 -0.00888227 0.992916 False
http_pipelines_blackhole -19.87KiB -2.26 879.24KiB 729.27KiB 1.94444 859.37KiB 736.24KiB 2.20126 0.352214 0.724789 False
splunk_transforms_splunk3 -92.15KiB -2.03 4.42MiB 1.58MiB 1.9337 4.34MiB 1.72MiB 0.57971 0.725182 0.468585 False
splunk_hec_to_splunk_hec_logs_acks -93.33KiB -0.49 18.46MiB 1.65MiB 0.293255 18.37MiB 1.54MiB 0.826446 0.755938 0.449944 False
splunk_hec_route_s3 -110.84KiB -1.28 8.47MiB 2.06MiB 0.275482 8.36MiB 2.09MiB 0.285714 0.69673 0.4862 False
datadog_agent_remap_blackhole_acks -302.16KiB -0.94 31.25MiB 907.26KiB 0 30.95MiB 581.03KiB 0 5.21839 2.55655e-07 False
syslog_humio_logs -333.49KiB -3.42 9.51MiB 305.78KiB 0 9.19MiB 72.24KiB 2.20386 20.1957 4.42563e-63 False
http_to_http_acks -387.7KiB -6.21 6.09MiB 2.42MiB 0 5.72MiB 2.8MiB 0 1.95185 0.0513477 True
syslog_splunk_hec_logs -437.63KiB -4.5 9.49MiB 210.44KiB 0 9.06MiB 136.74KiB 3.75335 33.3566 2.15412e-140 False
fluent_remap_aws_firehose -637.59KiB -0.92 67.69MiB 915.43KiB 0 67.07MiB 991.42KiB 1.53374 8.74371 1.84004e-17 False
datadog_agent_remap_blackhole -795.63KiB -2.57 30.24MiB 1.29MiB 0 29.46MiB 1.02MiB 0 8.8681 7.54496e-18 False
splunk_hec_to_splunk_hec_logs_noack -850.48KiB -4.33 19.2MiB 1.33MiB 0 18.37MiB 1.16MiB 0 8.84695 7.28636e-18 False
Fine details of each soak run.
(experiment, variant) total samples mean std min average p90 p95 p99 max skewness
('fluent_elasticsearch', 'comparison') 297 69.32MiB 820.76KiB 67.35MiB 69.44MiB 70.3MiB 70.54MiB 70.73MiB 71.15MiB -0.287918
('fluent_remap_aws_firehose', 'baseline') 364 67.69MiB 915.43KiB 65.29MiB 67.64MiB 68.93MiB 69.22MiB 69.52MiB 69.87MiB 0.00496613
('fluent_elasticsearch', 'baseline') 369 67.34MiB 2.19MiB 63.63MiB 66.62MiB 69.87MiB 70.17MiB 70.44MiB 70.79MiB -0.00669712
('fluent_remap_aws_firehose', 'comparison') 326 67.07MiB 991.42KiB 63.23MiB 66.9MiB 68.59MiB 68.9MiB 69.22MiB 69.77MiB 0.246866
('datadog_agent_remap_datadog_logs_acks', 'comparison') 341 37.77MiB 566.86KiB 36.61MiB 37.73MiB 38.59MiB 38.73MiB 39.04MiB 39.23MiB 0.284777
('datadog_agent_remap_datadog_logs_acks', 'baseline') 364 37.73MiB 1.05MiB 35.51MiB 37.7MiB 39.08MiB 39.24MiB 39.54MiB 39.79MiB 0.015633
('datadog_agent_remap_datadog_logs', 'comparison') 370 36.92MiB 510.27KiB 35.68MiB 36.91MiB 37.58MiB 37.71MiB 37.94MiB 38.18MiB 0.0141638
('datadog_agent_remap_datadog_logs', 'baseline') 362 36.7MiB 352.19KiB 35.57MiB 36.71MiB 37.14MiB 37.24MiB 37.52MiB 37.85MiB -0.105013
('datadog_agent_remap_blackhole_acks', 'baseline') 334 31.25MiB 907.26KiB 29.05MiB 31.07MiB 32.37MiB 32.61MiB 32.84MiB 32.91MiB 0.0651382
('datadog_agent_remap_blackhole_acks', 'comparison') 380 30.95MiB 581.03KiB 29.67MiB 30.96MiB 31.66MiB 31.88MiB 32.15MiB 32.49MiB 0.00402637
('datadog_agent_remap_blackhole', 'baseline') 336 30.24MiB 1.29MiB 27.51MiB 30.85MiB 31.61MiB 31.81MiB 32.08MiB 32.57MiB -0.364657
('datadog_agent_remap_blackhole', 'comparison') 384 29.46MiB 1.02MiB 26.47MiB 29.57MiB 30.71MiB 30.81MiB 31.08MiB 31.32MiB -0.251312
('http_to_http_noack', 'comparison') 316 23.59MiB 1.44MiB 19.87MiB 23.55MiB 25.56MiB 26.03MiB 26.69MiB 28.18MiB 0.121999
('splunk_hec_indexer_ack_blackhole', 'comparison') 360 23.07MiB 1.3MiB 19.69MiB 23.15MiB 24.68MiB 25.18MiB 26.36MiB 26.73MiB 0.0819103
('splunk_hec_indexer_ack_blackhole', 'baseline') 317 23.02MiB 1.22MiB 19.93MiB 22.91MiB 24.7MiB 25.07MiB 25.64MiB 26.61MiB 0.117001
('http_to_http_noack', 'baseline') 363 22.54MiB 1.72MiB 17.99MiB 22.46MiB 24.84MiB 25.35MiB 26.38MiB 27.25MiB -0.0199242
('splunk_hec_to_splunk_hec_logs_noack', 'baseline') 363 19.2MiB 1.33MiB 15.73MiB 19.19MiB 20.89MiB 21.28MiB 22.02MiB 22.41MiB -0.079183
('splunk_hec_to_splunk_hec_logs_acks', 'baseline') 341 18.46MiB 1.65MiB 13.14MiB 18.56MiB 20.59MiB 21.01MiB 21.94MiB 22.41MiB -0.236275
('splunk_hec_to_splunk_hec_logs_noack', 'comparison') 342 18.37MiB 1.16MiB 15.49MiB 18.41MiB 19.85MiB 20.2MiB 20.86MiB 21.57MiB -0.0183872
('splunk_hec_to_splunk_hec_logs_acks', 'comparison') 363 18.37MiB 1.54MiB 12.9MiB 18.44MiB 20.28MiB 20.74MiB 21.13MiB 22.05MiB -0.381281
('syslog_humio_logs', 'baseline') 362 9.51MiB 305.78KiB 9.05MiB 9.35MiB 9.86MiB 9.88MiB 9.92MiB 10.31MiB 0.0602725
('syslog_splunk_hec_logs', 'baseline') 363 9.49MiB 210.44KiB 9.13MiB 9.52MiB 9.73MiB 9.76MiB 9.79MiB 9.84MiB -0.032178
('syslog_humio_logs', 'comparison') 363 9.19MiB 72.24KiB 8.92MiB 9.19MiB 9.28MiB 9.29MiB 9.33MiB 9.34MiB -0.568611
('syslog_log2metric_splunk_hec_metrics', 'comparison') 316 9.1MiB 75.1KiB 8.93MiB 9.1MiB 9.19MiB 9.22MiB 9.29MiB 9.3MiB 0.159397
('syslog_log2metric_splunk_hec_metrics', 'baseline') 344 9.09MiB 148.12KiB 8.48MiB 9.12MiB 9.22MiB 9.24MiB 9.27MiB 9.29MiB -1.91119
('syslog_splunk_hec_logs', 'comparison') 373 9.06MiB 136.74KiB 8.33MiB 9.08MiB 9.17MiB 9.19MiB 9.23MiB 9.24MiB -3.50207
('splunk_hec_route_s3', 'baseline') 363 8.47MiB 2.06MiB 3.96MiB 8.31MiB 11.26MiB 12.01MiB 13.31MiB 14.87MiB 0.278029
('syslog_log2metric_humio_metrics', 'comparison') 363 8.46MiB 71.02KiB 8.25MiB 8.46MiB 8.55MiB 8.57MiB 8.62MiB 8.63MiB -0.210314
('syslog_log2metric_humio_metrics', 'baseline') 295 8.39MiB 269.54KiB 8.03MiB 8.24MiB 8.7MiB 8.73MiB 8.77MiB 8.78MiB 0.109877
('splunk_hec_route_s3', 'comparison') 350 8.36MiB 2.09MiB 3.28MiB 8.21MiB 11.12MiB 12.05MiB 13.19MiB 14.78MiB 0.194415
('syslog_regex_logs2metric_ddmetrics', 'comparison') 362 7.69MiB 588.04KiB 6.42MiB 7.8MiB 8.42MiB 8.45MiB 8.49MiB 8.54MiB -0.381505
('syslog_regex_logs2metric_ddmetrics', 'baseline') 362 7.58MiB 658.5KiB 6.05MiB 7.55MiB 8.45MiB 8.49MiB 8.55MiB 8.66MiB -0.250719
('syslog_loki', 'comparison') 362 7.35MiB 163.8KiB 7.0MiB 7.34MiB 7.55MiB 7.57MiB 7.59MiB 7.62MiB -0.110005
('syslog_loki', 'baseline') 362 7.26MiB 237.05KiB 6.67MiB 7.2MiB 7.56MiB 7.58MiB 7.63MiB 7.69MiB -0.161147
('http_to_http_acks', 'baseline') 362 6.09MiB 2.42MiB 731.41KiB 6.66MiB 9.47MiB 9.71MiB 10.14MiB 12.06MiB -0.14999
('http_to_http_acks', 'comparison') 366 5.72MiB 2.8MiB 0B 6.37MiB 9.46MiB 9.86MiB 10.99MiB 12.01MiB 0.0200111
('splunk_transforms_splunk3', 'baseline') 362 4.42MiB 1.58MiB 945.64KiB 4.32MiB 6.34MiB 7.21MiB 9.16MiB 10.37MiB 0.692869
('splunk_transforms_splunk3', 'comparison') 345 4.34MiB 1.72MiB 396.37KiB 4.26MiB 6.51MiB 7.43MiB 8.48MiB 9.62MiB 0.297353
('http_datadog_filter_blackhole', 'comparison') 362 2.2MiB 986.24KiB 31.98KiB 2.2MiB 3.47MiB 3.72MiB 4.38MiB 5.44MiB 0.189195
('http_datadog_filter_blackhole', 'baseline') 362 2.17MiB 911.3KiB 312.24KiB 2.1MiB 3.4MiB 3.77MiB 4.28MiB 5.67MiB 0.395602
('http_pipelines_no_grok_blackhole', 'comparison') 362 1.95MiB 965.67KiB 0B 1.9MiB 3.14MiB 3.76MiB 4.47MiB 5.08MiB 0.577024
('http_pipelines_no_grok_blackhole', 'baseline') 363 1.95MiB 981.75KiB 124.71KiB 1.89MiB 3.14MiB 3.65MiB 4.27MiB 4.51MiB 0.317523
('http_pipelines_blackhole', 'baseline') 360 879.24KiB 729.27KiB 0B 768.02KiB 1.77MiB 2.33MiB 2.99MiB 3.16MiB 0.906613
('http_pipelines_blackhole_acks', 'comparison') 316 860.15KiB 704.34KiB 0B 709.51KiB 1.73MiB 1.95MiB 2.83MiB 4.12MiB 1.0349
('http_pipelines_blackhole', 'comparison') 318 859.37KiB 736.24KiB 0B 679.65KiB 1.81MiB 2.19MiB 3.26MiB 3.7MiB 1.15872
('http_pipelines_blackhole_acks', 'baseline') 359 820.12KiB 698.99KiB 0B 646.69KiB 1.8MiB 2.04MiB 2.68MiB 4.06MiB 1.10694

Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>
@github-actions
Copy link

Soak Test Results

Baseline: 8ab99d1
Comparison: c60c7a6
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Test units below are bytes/second/CPU, except for "skewness". The further "skewness" is from 0.0 the more indication that vector lacks consistency in behavior, making predictions of fitness in the field challenging.

The abbreviated table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. The abbreviated table will be omitted if no statistically interesting changes are observed.

No statistically interesting changes with confidence 90.00%.

Fine details of change detection per experiment.
experiment Δ mean Δ mean % baseline mean baseline stdev baseline outlier percentage comparison mean comparison stdev comparison outlier percentage t-statistic p-value erratic
fluent_remap_aws_firehose 3.74MiB 5.65 66.17MiB 928.04KiB 0 69.91MiB 3.06MiB 0 -22.3269 1.24956e-73 False
datadog_agent_remap_datadog_logs_acks 2.48MiB 6.95 35.66MiB 5.98MiB 4.3956 38.14MiB 1.17MiB 0 -7.75668 7.65757e-14 False
fluent_elasticsearch 1.58MiB 2.45 64.42MiB 1.19MiB 0 65.99MiB 711.79KiB 0 -21.7936 1.843e-77 False
datadog_agent_remap_blackhole_acks 1.26MiB 4.03 31.21MiB 571.37KiB 1.49701 32.47MiB 438.99KiB 2.09424 -33.4768 3.48923e-141 False
datadog_agent_remap_datadog_logs 991.69KiB 2.56 37.9MiB 1.36MiB 0 38.87MiB 817.67KiB 7.7135 -11.677 1.84267e-28 False
splunk_hec_to_splunk_hec_logs_acks 940.99KiB 5.1 18.01MiB 1.4MiB 3.82514 18.93MiB 1.72MiB 1.10497 -7.91335 9.90504e-15 False
splunk_hec_indexer_ack_blackhole 818.45KiB 3.54 22.55MiB 1.21MiB 0.824176 23.35MiB 1.19MiB 0.826446 -8.97709 2.36036e-18 False
splunk_hec_to_splunk_hec_logs_noack 366.53KiB 1.94 18.49MiB 1.18MiB 0.58309 18.85MiB 1.37MiB 0.828729 -3.72822 0.000208488 False
syslog_log2metric_splunk_hec_metrics 307.0KiB 3.29 9.1MiB 159.44KiB 0 9.4MiB 281.14KiB 0.3125 -17.1205 5.19128e-52 False
syslog_loki 196.77KiB 2.63 7.31MiB 257.31KiB 15.1934 7.5MiB 76.56KiB 1.07527 -13.9605 1.07545e-36 True
http_to_http_noack 164.71KiB 0.72 22.26MiB 1.57MiB 0 22.42MiB 1.63MiB 0 -1.35962 0.174369 False
splunk_transforms_splunk3 108.91KiB 2.33 4.56MiB 1.58MiB 0.552486 4.67MiB 1.64MiB 0.276243 -0.887706 0.374995 False
http_to_http_acks 90.04KiB 1.63 5.4MiB 2.37MiB 0 5.49MiB 2.43MiB 0 -0.477893 0.632881 True
syslog_regex_logs2metric_ddmetrics 47.57KiB 0.6 7.76MiB 569.6KiB 0 7.81MiB 593.4KiB 0 -1.1033 0.270265 False
splunk_hec_route_s3 40.1KiB 0.45 8.71MiB 2.01MiB 0.630915 8.75MiB 2.08MiB 0.293255 -0.245099 0.806457 False
syslog_log2metric_humio_metrics 14.52KiB 0.17 8.19MiB 130.17KiB 2.80374 8.21MiB 69.76KiB 0.554017 -1.78425 0.0750185 False
http_pipelines_blackhole_acks 5.69KiB 0.66 858.22KiB 714.96KiB 1.91781 863.91KiB 757.98KiB 2.21519 -0.100291 0.920144 False
http_pipelines_blackhole -7.66KiB -0.87 884.15KiB 705.14KiB 1.65746 876.49KiB 696.73KiB 0.31348 0.142329 0.886863 False
syslog_humio_logs -25.06KiB -0.27 9.2MiB 72.85KiB 0.550964 9.18MiB 123.76KiB 2.73973 3.33092 0.000919676 False
http_pipelines_no_grok_blackhole -60.37KiB -2.86 2.06MiB 1.13MiB 4.6832 2.0MiB 1.01MiB 3.85675 0.741954 0.458359 False
http_datadog_filter_blackhole -63.65KiB -2.69 2.31MiB 895.03KiB 0.276243 2.25MiB 1.01MiB 0.828729 0.882969 0.377553 False
syslog_splunk_hec_logs -261.73KiB -2.64 9.68MiB 89.2KiB 2.75482 9.42MiB 345.16KiB 0 14.0059 1.01638e-36 False
datadog_agent_remap_blackhole -1.29MiB -4.11 31.42MiB 498.6KiB 0.26178 30.13MiB 1.54MiB 0 14.8013 7.6562e-40 False
Fine details of each soak run.
(experiment, variant) total samples mean std min average p90 p95 p99 max skewness
('fluent_remap_aws_firehose', 'comparison') 363 69.91MiB 3.06MiB 65.29MiB 71.67MiB 73.32MiB 73.46MiB 73.77MiB 73.99MiB -0.0108464
('fluent_remap_aws_firehose', 'baseline') 363 66.17MiB 928.04KiB 64.35MiB 65.96MiB 67.38MiB 67.64MiB 67.96MiB 68.07MiB 0.165619
('fluent_elasticsearch', 'comparison') 362 65.99MiB 711.79KiB 64.23MiB 65.98MiB 66.91MiB 67.07MiB 67.41MiB 67.63MiB -0.120258
('fluent_elasticsearch', 'baseline') 362 64.42MiB 1.19MiB 61.81MiB 64.3MiB 66.12MiB 66.37MiB 66.83MiB 67.12MiB 0.183477
('datadog_agent_remap_datadog_logs', 'comparison') 363 38.87MiB 817.67KiB 35.24MiB 39.0MiB 39.67MiB 39.82MiB 40.13MiB 40.42MiB -1.46435
('datadog_agent_remap_datadog_logs_acks', 'comparison') 362 38.14MiB 1.17MiB 35.72MiB 38.36MiB 39.53MiB 39.69MiB 39.94MiB 40.02MiB -0.0844816
('datadog_agent_remap_datadog_logs', 'baseline') 363 37.9MiB 1.36MiB 35.43MiB 38.05MiB 39.49MiB 39.64MiB 39.9MiB 40.17MiB -0.0581134
('datadog_agent_remap_datadog_logs_acks', 'baseline') 364 35.66MiB 5.98MiB 0B 36.69MiB 37.19MiB 37.35MiB 37.64MiB 38.28MiB -5.75601
('datadog_agent_remap_blackhole_acks', 'comparison') 382 32.47MiB 438.99KiB 31.17MiB 32.45MiB 33.02MiB 33.19MiB 33.53MiB 33.62MiB 0.115111
('datadog_agent_remap_blackhole', 'baseline') 382 31.42MiB 498.6KiB 29.86MiB 31.46MiB 32.03MiB 32.11MiB 32.42MiB 32.61MiB -0.266603
('datadog_agent_remap_blackhole_acks', 'baseline') 334 31.21MiB 571.37KiB 29.27MiB 31.23MiB 31.91MiB 32.03MiB 32.34MiB 32.77MiB -0.444951
('datadog_agent_remap_blackhole', 'comparison') 341 30.13MiB 1.54MiB 27.22MiB 30.8MiB 31.93MiB 32.06MiB 32.22MiB 32.46MiB -0.223216
('splunk_hec_indexer_ack_blackhole', 'comparison') 363 23.35MiB 1.19MiB 19.55MiB 23.48MiB 24.79MiB 25.14MiB 25.7MiB 26.58MiB -0.339637
('splunk_hec_indexer_ack_blackhole', 'baseline') 364 22.55MiB 1.21MiB 19.01MiB 22.53MiB 24.16MiB 24.52MiB 24.99MiB 26.4MiB -0.0420041
('http_to_http_noack', 'comparison') 369 22.42MiB 1.63MiB 18.32MiB 22.46MiB 24.63MiB 25.1MiB 25.96MiB 26.18MiB 0.0569689
('http_to_http_noack', 'baseline') 362 22.26MiB 1.57MiB 18.5MiB 22.19MiB 24.36MiB 24.87MiB 25.92MiB 26.6MiB 0.13352
('splunk_hec_to_splunk_hec_logs_acks', 'comparison') 362 18.93MiB 1.72MiB 13.53MiB 18.98MiB 21.01MiB 21.63MiB 22.32MiB 23.0MiB -0.304284
('splunk_hec_to_splunk_hec_logs_noack', 'comparison') 362 18.85MiB 1.37MiB 14.45MiB 18.92MiB 20.53MiB 20.89MiB 21.73MiB 22.29MiB -0.280451
('splunk_hec_to_splunk_hec_logs_noack', 'baseline') 343 18.49MiB 1.18MiB 15.29MiB 18.5MiB 20.05MiB 20.44MiB 21.07MiB 21.63MiB -0.0202212
('splunk_hec_to_splunk_hec_logs_acks', 'baseline') 366 18.01MiB 1.4MiB 13.68MiB 18.04MiB 19.67MiB 20.17MiB 21.26MiB 21.75MiB -0.343656
('syslog_splunk_hec_logs', 'baseline') 363 9.68MiB 89.2KiB 9.01MiB 9.69MiB 9.76MiB 9.77MiB 9.82MiB 9.87MiB -3.45784
('syslog_splunk_hec_logs', 'comparison') 364 9.42MiB 345.16KiB 8.93MiB 9.25MiB 9.79MiB 9.81MiB 9.86MiB 9.88MiB -0.0117328
('syslog_log2metric_splunk_hec_metrics', 'comparison') 320 9.4MiB 281.14KiB 8.28MiB 9.56MiB 9.71MiB 9.72MiB 9.75MiB 9.94MiB -0.203231
('syslog_humio_logs', 'baseline') 363 9.2MiB 72.85KiB 9.0MiB 9.21MiB 9.29MiB 9.31MiB 9.35MiB 9.41MiB -0.314613
('syslog_humio_logs', 'comparison') 365 9.18MiB 123.76KiB 8.51MiB 9.18MiB 9.32MiB 9.36MiB 9.41MiB 9.45MiB -1.26922
('syslog_log2metric_splunk_hec_metrics', 'baseline') 341 9.1MiB 159.44KiB 8.85MiB 9.06MiB 9.3MiB 9.32MiB 9.36MiB 9.4MiB 0.135895
('splunk_hec_route_s3', 'comparison') 341 8.75MiB 2.08MiB 4.14MiB 8.64MiB 11.38MiB 12.22MiB 13.52MiB 15.85MiB 0.255817
('splunk_hec_route_s3', 'baseline') 317 8.71MiB 2.01MiB 3.31MiB 8.7MiB 11.33MiB 12.01MiB 13.66MiB 14.75MiB 0.110734
('syslog_log2metric_humio_metrics', 'comparison') 361 8.21MiB 69.76KiB 8.02MiB 8.21MiB 8.3MiB 8.32MiB 8.36MiB 8.43MiB 0.105094
('syslog_log2metric_humio_metrics', 'baseline') 321 8.19MiB 130.17KiB 7.67MiB 8.21MiB 8.33MiB 8.36MiB 8.41MiB 8.42MiB -1.03553
('syslog_regex_logs2metric_ddmetrics', 'comparison') 364 7.81MiB 593.4KiB 6.49MiB 7.95MiB 8.5MiB 8.54MiB 8.59MiB 8.67MiB -0.490819
('syslog_regex_logs2metric_ddmetrics', 'baseline') 364 7.76MiB 569.6KiB 6.65MiB 8.01MiB 8.38MiB 8.43MiB 8.46MiB 8.49MiB -0.559612
('syslog_loki', 'comparison') 372 7.5MiB 76.56KiB 7.24MiB 7.51MiB 7.59MiB 7.61MiB 7.65MiB 7.67MiB -0.537078
('syslog_loki', 'baseline') 362 7.31MiB 257.31KiB 6.55MiB 7.38MiB 7.53MiB 7.55MiB 7.58MiB 7.61MiB -1.54149
('http_to_http_acks', 'comparison') 363 5.49MiB 2.43MiB 1.43MiB 5.15MiB 9.03MiB 9.41MiB 10.54MiB 11.29MiB 0.199716
('http_to_http_acks', 'baseline') 320 5.4MiB 2.37MiB 1005.66KiB 5.33MiB 8.6MiB 9.18MiB 10.14MiB 10.52MiB 0.0250691
('splunk_transforms_splunk3', 'comparison') 362 4.67MiB 1.64MiB 915.31KiB 4.48MiB 6.91MiB 7.74MiB 8.56MiB 9.74MiB 0.421789
('splunk_transforms_splunk3', 'baseline') 362 4.56MiB 1.58MiB 1.28MiB 4.47MiB 6.73MiB 7.26MiB 8.44MiB 9.0MiB 0.344644
('http_datadog_filter_blackhole', 'baseline') 362 2.31MiB 895.03KiB 311.2KiB 2.25MiB 3.47MiB 3.77MiB 4.45MiB 4.87MiB 0.262537
('http_datadog_filter_blackhole', 'comparison') 362 2.25MiB 1.01MiB 248.28KiB 2.1MiB 3.7MiB 4.23MiB 4.71MiB 5.53MiB 0.532427
('http_pipelines_no_grok_blackhole', 'baseline') 363 2.06MiB 1.13MiB 123.57KiB 1.77MiB 3.47MiB 4.21MiB 5.76MiB 6.21MiB 1.12925
('http_pipelines_no_grok_blackhole', 'comparison') 363 2.0MiB 1.01MiB 186.49KiB 1.86MiB 3.37MiB 3.9MiB 4.78MiB 5.26MiB 0.770526
('http_pipelines_blackhole', 'baseline') 362 884.15KiB 705.14KiB 0B 736.04KiB 1.9MiB 2.06MiB 3.06MiB 3.38MiB 1.0065
('http_pipelines_blackhole', 'comparison') 319 876.49KiB 696.73KiB 0B 736.04KiB 1.71MiB 2.13MiB 2.6MiB 3.34MiB 0.718672
('http_pipelines_blackhole_acks', 'comparison') 316 863.91KiB 757.98KiB 0B 664.32KiB 1.93MiB 2.18MiB 3.28MiB 3.46MiB 1.0496
('http_pipelines_blackhole_acks', 'baseline') 365 858.22KiB 714.96KiB 0B 736.04KiB 1.81MiB 2.14MiB 3.22MiB 3.37MiB 1.0425

@spencergilbert spencergilbert force-pushed the revert-10566-revert-10521-spencer/10467-update-k8s-e2e-to-new-helm branch from 2ac0bc1 to c325b36 Compare December 23, 2021 22:33
Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>
@spencergilbert spencergilbert force-pushed the revert-10566-revert-10521-spencer/10467-update-k8s-e2e-to-new-helm branch from c325b36 to 1dacc57 Compare December 23, 2021 22:37
@github-actions
Copy link

Soak Test Results

Baseline: 2400b48
Comparison: 1dacc57
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Test units below are bytes/second/CPU, except for "skewness". The further "skewness" is from 0.0 the more indication that vector lacks consistency in behavior, making predictions of fitness in the field challenging.

The abbreviated table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. The abbreviated table will be omitted if no statistically interesting changes are observed.

No statistically interesting changes with confidence 90.00%.

Fine details of change detection per experiment.
experiment Δ mean Δ mean % baseline mean baseline stdev baseline outlier percentage comparison mean comparison stdev comparison outlier percentage t-statistic p-value erratic
fluent_remap_aws_firehose 2.48MiB 3.54 70.22MiB 2.85MiB 0 72.7MiB 1.44MiB 3.30579 -14.8294 5.54694e-42 False
datadog_agent_remap_datadog_logs_acks 1.83MiB 4.9 37.34MiB 1.28MiB 0 39.17MiB 441.82KiB 2.20994 -24.4026 6.1187e-80 False
splunk_hec_to_splunk_hec_logs_acks 1.18MiB 6.66 17.75MiB 1.47MiB 1.65289 18.93MiB 1.4MiB 1.37363 -11.0744 1.94367e-26 False
splunk_hec_to_splunk_hec_logs_noack 923.12KiB 4.96 18.16MiB 1.28MiB 0.275482 19.06MiB 1.37MiB 0.544959 -9.16654 4.94279e-19 False
syslog_loki 478.77KiB 6.69 6.99MiB 96.48KiB 0.879765 7.45MiB 114.7KiB 0 -60.0154 1.47936e-276 True
datadog_agent_remap_datadog_logs 471.82KiB 1.21 38.03MiB 1.61MiB 0 38.49MiB 1.07MiB 0 -4.4612 9.75844e-06 False
http_to_http_acks 448.95KiB 9.04 4.85MiB 2.56MiB 0 5.29MiB 2.55MiB 0 -2.2773 0.0230695 True
http_to_http_noack 373.49KiB 1.58 23.12MiB 1.48MiB 0.623053 23.49MiB 1.38MiB 0 -3.32425 0.000935762 False
syslog_log2metric_splunk_hec_metrics 259.86KiB 2.71 9.38MiB 517.86KiB 0 9.63MiB 159.54KiB 17.3554 -9.1942 1.55211e-18 False
http_datadog_filter_blackhole 127.25KiB 5.78 2.15MiB 904.25KiB 0.828729 2.27MiB 948.26KiB 0 -1.84769 0.0650576 False
http_pipelines_no_grok_blackhole 23.49KiB 1.15 2.0MiB 978.58KiB 0 2.02MiB 1.04MiB 1.85185 -0.28382 0.776655 False
http_pipelines_blackhole 22.79KiB 2.58 884.3KiB 680.32KiB 1.38889 907.08KiB 761.44KiB 2.50696 -0.423079 0.672366 False
syslog_splunk_hec_logs 8.88KiB 0.09 9.53MiB 244.3KiB 0 9.54MiB 221.13KiB 0 -0.497842 0.618758 False
splunk_transforms_splunk3 -9.28KiB -0.21 4.36MiB 1.59MiB 1.37741 4.35MiB 1.56MiB 0.824176 0.0777433 0.938054 False
http_pipelines_blackhole_acks -35.15KiB -4.17 843.24KiB 758.99KiB 0.828729 808.09KiB 675.16KiB 1.36986 0.659466 0.509809 False
splunk_hec_route_s3 -48.54KiB -0.57 8.36MiB 1.89MiB 0.552486 8.31MiB 2.29MiB 1.92837 0.30401 0.761211 False
syslog_humio_logs -67.02KiB -0.69 9.52MiB 319.78KiB 0 9.45MiB 469.93KiB 0 2.24345 0.0252108 False
syslog_log2metric_humio_metrics -219.5KiB -2.52 8.49MiB 105.93KiB 0 8.28MiB 187.61KiB 0 19.3834 1.14159e-64 False
syslog_regex_logs2metric_ddmetrics -259.38KiB -3.26 7.76MiB 586.54KiB 0 7.51MiB 614.38KiB 0 5.81803 8.94663e-09 False
splunk_hec_indexer_ack_blackhole -585.23KiB -2.39 23.9MiB 1.21MiB 0 23.33MiB 1.14MiB 1.25786 6.13946 1.461e-09 False
datadog_agent_remap_blackhole -700.54KiB -2.31 29.63MiB 967.14KiB 0 28.94MiB 616.67KiB 0.520833 11.3529 5.7748e-27 False
datadog_agent_remap_blackhole_acks -871.27KiB -2.74 31.07MiB 561.85KiB 0.263158 30.22MiB 530.57KiB 2.09581 21.2977 3.59069e-78 False
fluent_elasticsearch -1.53MiB -2.2 69.38MiB 1.22MiB 0 67.86MiB 1.67MiB 0 14.0114 3.06163e-39 False
Fine details of each soak run.
(experiment, variant) total samples mean std min average p90 p95 p99 max skewness
('fluent_remap_aws_firehose', 'comparison') 363 72.7MiB 1.44MiB 65.38MiB 72.79MiB 74.15MiB 74.46MiB 74.8MiB 75.02MiB -2.32815
('fluent_remap_aws_firehose', 'baseline') 365 70.22MiB 2.85MiB 65.67MiB 68.29MiB 73.47MiB 73.73MiB 74.0MiB 74.17MiB 0.022648
('fluent_elasticsearch', 'baseline') 362 69.38MiB 1.22MiB 66.72MiB 69.4MiB 70.93MiB 71.13MiB 71.61MiB 71.92MiB -0.0809844
('fluent_elasticsearch', 'comparison') 362 67.86MiB 1.67MiB 64.83MiB 67.9MiB 69.92MiB 70.12MiB 70.48MiB 70.96MiB 0.0214916
('datadog_agent_remap_datadog_logs_acks', 'comparison') 362 39.17MiB 441.82KiB 37.54MiB 39.18MiB 39.72MiB 39.83MiB 40.13MiB 40.41MiB -0.206902
('datadog_agent_remap_datadog_logs', 'comparison') 365 38.49MiB 1.07MiB 36.49MiB 38.11MiB 39.82MiB 39.99MiB 40.2MiB 40.45MiB 0.104188
('datadog_agent_remap_datadog_logs', 'baseline') 343 38.03MiB 1.61MiB 35.29MiB 38.86MiB 39.75MiB 39.91MiB 40.26MiB 40.38MiB -0.153098
('datadog_agent_remap_datadog_logs_acks', 'baseline') 319 37.34MiB 1.28MiB 34.53MiB 36.82MiB 38.95MiB 39.12MiB 39.36MiB 39.52MiB 0.105795
('datadog_agent_remap_blackhole_acks', 'baseline') 380 31.07MiB 561.85KiB 29.4MiB 31.07MiB 31.75MiB 31.91MiB 32.09MiB 32.44MiB -0.259133
('datadog_agent_remap_blackhole_acks', 'comparison') 334 30.22MiB 530.57KiB 28.74MiB 30.27MiB 30.84MiB 31.02MiB 31.32MiB 31.72MiB -0.245458
('datadog_agent_remap_blackhole', 'baseline') 332 29.63MiB 967.14KiB 27.34MiB 29.9MiB 30.73MiB 30.89MiB 31.25MiB 31.3MiB -0.284826
('datadog_agent_remap_blackhole', 'comparison') 384 28.94MiB 616.67KiB 26.72MiB 28.94MiB 29.72MiB 29.91MiB 30.26MiB 30.56MiB -0.129505
('splunk_hec_indexer_ack_blackhole', 'baseline') 320 23.9MiB 1.21MiB 20.89MiB 23.82MiB 25.5MiB 26.01MiB 26.37MiB 26.94MiB 0.041879
('http_to_http_noack', 'comparison') 363 23.49MiB 1.38MiB 19.87MiB 23.47MiB 25.29MiB 25.67MiB 26.46MiB 27.11MiB -0.0156918
('splunk_hec_indexer_ack_blackhole', 'comparison') 318 23.33MiB 1.14MiB 19.73MiB 23.47MiB 24.67MiB 24.97MiB 25.54MiB 25.89MiB -0.474349
('http_to_http_noack', 'baseline') 321 23.12MiB 1.48MiB 19.54MiB 23.17MiB 25.05MiB 25.68MiB 26.2MiB 27.61MiB 0.0535349
('splunk_hec_to_splunk_hec_logs_noack', 'comparison') 367 19.06MiB 1.37MiB 14.36MiB 19.19MiB 20.68MiB 21.14MiB 22.21MiB 23.27MiB -0.0955387
('splunk_hec_to_splunk_hec_logs_acks', 'comparison') 364 18.93MiB 1.4MiB 14.39MiB 19.01MiB 20.64MiB 21.15MiB 22.02MiB 22.44MiB -0.312859
('splunk_hec_to_splunk_hec_logs_noack', 'baseline') 363 18.16MiB 1.28MiB 14.48MiB 18.27MiB 19.79MiB 20.19MiB 20.89MiB 21.19MiB -0.153075
('splunk_hec_to_splunk_hec_logs_acks', 'baseline') 363 17.75MiB 1.47MiB 12.22MiB 17.91MiB 19.4MiB 19.87MiB 20.59MiB 21.01MiB -0.668563
('syslog_log2metric_splunk_hec_metrics', 'comparison') 363 9.63MiB 159.54KiB 9.2MiB 9.67MiB 9.78MiB 9.79MiB 9.81MiB 9.85MiB -1.27946
('syslog_splunk_hec_logs', 'comparison') 342 9.54MiB 221.13KiB 9.17MiB 9.44MiB 9.89MiB 9.91MiB 9.95MiB 9.98MiB 0.793906
('syslog_splunk_hec_logs', 'baseline') 341 9.53MiB 244.3KiB 9.04MiB 9.39MiB 9.82MiB 9.84MiB 9.86MiB 9.91MiB 0.0928591
('syslog_humio_logs', 'baseline') 362 9.52MiB 319.78KiB 8.91MiB 9.64MiB 9.86MiB 9.9MiB 9.93MiB 9.94MiB -0.172091
('syslog_humio_logs', 'comparison') 362 9.45MiB 469.93KiB 8.83MiB 9.39MiB 9.96MiB 10.0MiB 10.03MiB 10.06MiB 0.00292229
('syslog_log2metric_splunk_hec_metrics', 'baseline') 368 9.38MiB 517.86KiB 8.72MiB 9.76MiB 9.91MiB 9.94MiB 9.97MiB 9.99MiB -0.0326507
('syslog_log2metric_humio_metrics', 'baseline') 362 8.49MiB 105.93KiB 8.28MiB 8.5MiB 8.63MiB 8.65MiB 8.69MiB 8.75MiB 0.0352649
('splunk_hec_route_s3', 'baseline') 362 8.36MiB 1.89MiB 3.81MiB 8.28MiB 10.72MiB 11.44MiB 12.85MiB 13.77MiB 0.157108
('splunk_hec_route_s3', 'comparison') 363 8.31MiB 2.29MiB 3.43MiB 8.02MiB 11.64MiB 12.36MiB 14.58MiB 15.5MiB 0.530567
('syslog_log2metric_humio_metrics', 'comparison') 362 8.28MiB 187.61KiB 7.78MiB 8.29MiB 8.51MiB 8.54MiB 8.6MiB 8.63MiB -0.247951
('syslog_regex_logs2metric_ddmetrics', 'baseline') 363 7.76MiB 586.54KiB 6.52MiB 7.93MiB 8.45MiB 8.48MiB 8.53MiB 8.55MiB -0.493096
('syslog_regex_logs2metric_ddmetrics', 'comparison') 363 7.51MiB 614.38KiB 6.07MiB 7.48MiB 8.32MiB 8.43MiB 8.48MiB 8.55MiB -0.291074
('syslog_loki', 'comparison') 362 7.45MiB 114.7KiB 7.13MiB 7.46MiB 7.59MiB 7.61MiB 7.65MiB 7.67MiB -0.285418
('syslog_loki', 'baseline') 341 6.99MiB 96.48KiB 6.67MiB 7.0MiB 7.1MiB 7.12MiB 7.14MiB 7.16MiB -0.649816
('http_to_http_acks', 'comparison') 341 5.29MiB 2.55MiB 0B 4.83MiB 9.09MiB 9.56MiB 10.24MiB 11.32MiB 0.189938
('http_to_http_acks', 'baseline') 362 4.85MiB 2.56MiB 0B 4.65MiB 8.27MiB 9.41MiB 9.99MiB 11.02MiB 0.269011
('splunk_transforms_splunk3', 'baseline') 363 4.36MiB 1.59MiB 915.26KiB 4.23MiB 6.55MiB 7.2MiB 8.6MiB 9.39MiB 0.487363
('splunk_transforms_splunk3', 'comparison') 364 4.35MiB 1.56MiB 1.28MiB 4.2MiB 6.34MiB 6.94MiB 8.46MiB 10.64MiB 0.610119
('http_datadog_filter_blackhole', 'comparison') 362 2.27MiB 948.26KiB 186.49KiB 2.17MiB 3.5MiB 3.77MiB 4.34MiB 4.7MiB 0.22559
('http_datadog_filter_blackhole', 'baseline') 362 2.15MiB 904.25KiB 401.58KiB 2.09MiB 3.34MiB 3.64MiB 4.32MiB 5.5MiB 0.476596
('http_pipelines_no_grok_blackhole', 'comparison') 270 2.02MiB 1.04MiB 0B 1.86MiB 3.44MiB 4.23MiB 4.79MiB 5.29MiB 0.692254
('http_pipelines_no_grok_blackhole', 'baseline') 361 2.0MiB 978.58KiB 307.82KiB 1.8MiB 3.35MiB 3.89MiB 4.28MiB 4.48MiB 0.560542
('http_pipelines_blackhole', 'comparison') 359 907.08KiB 761.44KiB 0B 736.04KiB 1.94MiB 2.31MiB 3.16MiB 3.85MiB 1.10914
('http_pipelines_blackhole', 'baseline') 360 884.3KiB 680.32KiB 0B 768.02KiB 1.7MiB 2.06MiB 2.6MiB 3.82MiB 1.09216
('http_pipelines_blackhole_acks', 'baseline') 362 843.24KiB 758.99KiB 0B 617.89KiB 1.9MiB 2.28MiB 2.53MiB 3.37MiB 0.867749
('http_pipelines_blackhole_acks', 'comparison') 365 808.09KiB 675.16KiB 0B 616.77KiB 1.79MiB 1.94MiB 2.64MiB 3.16MiB 0.834905

@spencergilbert
Copy link
Contributor

Upgrading minikube causes all crio jobs to fail because the cluster isn't brought up cleanly:

runner@fv-az201-838:~/work/vector/vector$ kubectl get all -A
NAMESPACE     NAME                                   READY   STATUS                 RESTARTS   AGE
kube-system   pod/coredns-f9fd979d6-4r8wf            0/1     ContainerCreating      0          7m55s
...
kube-system   pod/kube-proxy-7ms4g                   0/1     CreateContainerError   0          7m55s
...
kube-system   pod/storage-provisioner                0/1     CrashLoopBackOff       5          8m8s

Seemingly related: kubernetes/minikube#12684 kubernetes/minikube#12705 kubernetes-sigs/kind#2058

However it does seem like the underlying versions are new enough that they should be patched, unless I'm missing something:

docker@minikube:~$ sudo runc --version
runc version 1.0.1
commit: v1.0.1-0-g4144b63
spec: 1.0.2-dev
go: go1.15.14
libseccomp: 2.5.1
docker@minikube:~$ sudo crictl version
Version:  0.1.0
RuntimeName:  cri-o
RuntimeVersion:  1.22.0
RuntimeApiVersion:  v1alpha2

Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>
@github-actions
Copy link

Soak Test Results

Baseline: 2400b48
Comparison: 5c50a53
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Test units below are bytes/second/CPU, except for "skewness". The further "skewness" is from 0.0 the more indication that vector lacks consistency in behavior, making predictions of fitness in the field challenging.

The abbreviated table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. The abbreviated table will be omitted if no statistically interesting changes are observed.

experiment Δ mean Δ mean % confidence
datadog_agent_remap_datadog_logs_acks 4.75MiB 13.84 100.00%
Fine details of change detection per experiment.
experiment Δ mean Δ mean % baseline mean baseline stdev baseline outlier percentage comparison mean comparison stdev comparison outlier percentage t-statistic p-value erratic
datadog_agent_remap_datadog_logs_acks 4.75MiB 13.84 34.33MiB 10.39MiB 10.221 39.08MiB 499.33KiB 0.58651 -8.68701 1.29248e-16 False
fluent_remap_aws_firehose 1.56MiB 2.2 70.84MiB 2.45MiB 0 72.4MiB 572.03KiB 1.38122 -11.1457 7.02255e-25 False
datadog_agent_remap_blackhole_acks 628.31KiB 2 30.62MiB 517.79KiB 0.26178 31.23MiB 1.11MiB 0 -9.82706 4.67845e-21 False
splunk_hec_to_splunk_hec_logs_noack 627.43KiB 3.27 18.76MiB 1.05MiB 0.552486 19.38MiB 1.14MiB 0.276243 -7.54684 1.3562e-13 False
splunk_hec_indexer_ack_blackhole 587.28KiB 2.52 22.77MiB 1.2MiB 0.958466 23.34MiB 1.3MiB 0.879765 -5.86375 7.19313e-09 False
syslog_log2metric_humio_metrics 394.35KiB 4.72 8.15MiB 256.42KiB 0 8.54MiB 95.08KiB 0 -27.4355 9.18109e-99 False
splunk_hec_route_s3 289.12KiB 3.44 8.2MiB 1.88MiB 1.23457 8.48MiB 2.43MiB 3.02198 -1.71539 0.0867332 False
http_to_http_acks 281.65KiB 5.04 5.45MiB 2.46MiB 0 5.73MiB 2.51MiB 0 -1.46624 0.143031 True
syslog_log2metric_splunk_hec_metrics 173.26KiB 1.82 9.28MiB 83.87KiB 1.65746 9.45MiB 346.77KiB 0 -9.25186 1.29611e-18 False
syslog_regex_logs2metric_ddmetrics 168.61KiB 2.14 7.68MiB 589.81KiB 0 7.84MiB 612.58KiB 0 -3.77527 0.000172971 False
http_datadog_filter_blackhole 137.85KiB 6.27 2.15MiB 1.01MiB 1.65289 2.28MiB 1.01MiB 0.31746 -1.73728 0.082803 False
datadog_agent_remap_blackhole 105.68KiB 0.35 29.23MiB 630.89KiB 0.298507 29.33MiB 686.51KiB 0 -2.15018 0.0318758 False
http_pipelines_no_grok_blackhole 95.58KiB 4.63 2.02MiB 857.1KiB 0 2.11MiB 953.19KiB 0 -1.4175 0.156773 False
splunk_hec_to_splunk_hec_logs_acks 94.72KiB 0.51 18.25MiB 1.52MiB 2.48619 18.34MiB 1.47MiB 1.10193 -0.833754 0.404695 False
syslog_splunk_hec_logs 72.48KiB 0.72 9.78MiB 73.72KiB 1.10193 9.85MiB 181.17KiB 0 -7.05246 6.20862e-12 False
splunk_transforms_splunk3 -25.66KiB -0.55 4.58MiB 1.58MiB 0.58309 4.55MiB 1.48MiB 0.874636 0.214291 0.830384 False
http_pipelines_blackhole -32.84KiB -3.6 911.93KiB 768.26KiB 1.94444 879.09KiB 672.6KiB 0.316456 0.592515 0.553704 False
http_pipelines_blackhole_acks -44.71KiB -5.01 892.58KiB 680.28KiB 0.821918 847.86KiB 703.37KiB 1.38122 0.871144 0.383965 False
syslog_loki -64.12KiB -0.9 6.98MiB 63.21KiB 1.64835 6.91MiB 277.15KiB 0 4.29235 2.22274e-05 True
http_to_http_noack -224.11KiB -0.93 23.54MiB 1.32MiB 0.552486 23.32MiB 1.38MiB 1.09589 2.18793 0.0289939 False
syslog_humio_logs -235.51KiB -2.45 9.38MiB 119.93KiB 0 9.15MiB 121.7KiB 0 26.2794 1.9339e-107 False
datadog_agent_remap_datadog_logs -280.48KiB -0.72 38.16MiB 1.6MiB 0 37.89MiB 769.96KiB 2.33918 2.93622 0.00346874 False
fluent_elasticsearch -1.85MiB -2.66 69.65MiB 1.17MiB 0.276243 67.8MiB 2.1MiB 0.276243 14.6374 2.31394e-41 False
Fine details of each soak run.
(experiment, variant) total samples mean std min average p90 p95 p99 max skewness
('fluent_remap_aws_firehose', 'comparison') 362 72.4MiB 572.03KiB 70.77MiB 72.41MiB 73.1MiB 73.27MiB 73.52MiB 73.87MiB -0.292005
('fluent_remap_aws_firehose', 'baseline') 320 70.84MiB 2.45MiB 66.66MiB 72.5MiB 73.26MiB 73.43MiB 73.77MiB 73.95MiB -0.286381
('fluent_elasticsearch', 'baseline') 362 69.65MiB 1.17MiB 66.2MiB 69.73MiB 71.06MiB 71.33MiB 72.06MiB 74.85MiB 0.0394344
('fluent_elasticsearch', 'comparison') 362 67.8MiB 2.1MiB 51.79MiB 67.92MiB 70.07MiB 70.23MiB 70.54MiB 71.02MiB -1.21347
('datadog_agent_remap_datadog_logs_acks', 'comparison') 341 39.08MiB 499.33KiB 37.51MiB 39.11MiB 39.68MiB 39.87MiB 40.22MiB 40.52MiB 0.0768011
('datadog_agent_remap_datadog_logs', 'baseline') 364 38.16MiB 1.6MiB 35.75MiB 38.81MiB 40.05MiB 40.28MiB 40.5MiB 40.8MiB -0.000980958
('datadog_agent_remap_datadog_logs', 'comparison') 342 37.89MiB 769.96KiB 36.63MiB 37.76MiB 38.87MiB 39.42MiB 39.94MiB 40.51MiB 0.857923
('datadog_agent_remap_datadog_logs_acks', 'baseline') 362 34.33MiB 10.39MiB 0B 36.99MiB 39.37MiB 39.48MiB 39.88MiB 40.16MiB -2.82805
('datadog_agent_remap_blackhole_acks', 'comparison') 382 31.23MiB 1.11MiB 28.84MiB 31.28MiB 32.55MiB 32.68MiB 33.01MiB 33.69MiB -0.139041
('datadog_agent_remap_blackhole_acks', 'baseline') 382 30.62MiB 517.79KiB 29.0MiB 30.69MiB 31.22MiB 31.33MiB 31.53MiB 31.77MiB -0.472771
('datadog_agent_remap_blackhole', 'comparison') 384 29.33MiB 686.51KiB 27.38MiB 29.28MiB 30.21MiB 30.39MiB 30.71MiB 31.26MiB 0.0164032
('datadog_agent_remap_blackhole', 'baseline') 335 29.23MiB 630.89KiB 27.53MiB 29.28MiB 30.01MiB 30.18MiB 30.53MiB 30.97MiB -0.102881
('http_to_http_noack', 'baseline') 362 23.54MiB 1.32MiB 19.21MiB 23.53MiB 25.14MiB 25.56MiB 26.57MiB 27.64MiB 0.0478616
('splunk_hec_indexer_ack_blackhole', 'comparison') 341 23.34MiB 1.3MiB 19.34MiB 23.51MiB 24.91MiB 25.3MiB 25.84MiB 26.37MiB -0.419797
('http_to_http_noack', 'comparison') 365 23.32MiB 1.38MiB 18.59MiB 23.41MiB 25.02MiB 25.38MiB 26.44MiB 27.82MiB -0.128563
('splunk_hec_indexer_ack_blackhole', 'baseline') 313 22.77MiB 1.2MiB 19.22MiB 22.74MiB 24.46MiB 24.92MiB 25.47MiB 26.31MiB 0.0694422
('splunk_hec_to_splunk_hec_logs_noack', 'comparison') 362 19.38MiB 1.14MiB 16.54MiB 19.38MiB 20.82MiB 21.22MiB 22.0MiB 22.82MiB 0.141074
('splunk_hec_to_splunk_hec_logs_noack', 'baseline') 362 18.76MiB 1.05MiB 15.58MiB 18.73MiB 20.05MiB 20.38MiB 21.01MiB 21.31MiB -0.143348
('splunk_hec_to_splunk_hec_logs_acks', 'comparison') 363 18.34MiB 1.47MiB 14.0MiB 18.18MiB 20.31MiB 20.91MiB 22.03MiB 22.82MiB 0.183646
('splunk_hec_to_splunk_hec_logs_acks', 'baseline') 362 18.25MiB 1.52MiB 13.32MiB 18.43MiB 20.05MiB 20.41MiB 21.0MiB 21.3MiB -0.783339
('syslog_splunk_hec_logs', 'comparison') 362 9.85MiB 181.17KiB 9.44MiB 9.89MiB 10.05MiB 10.07MiB 10.12MiB 10.16MiB -0.158188
('syslog_splunk_hec_logs', 'baseline') 363 9.78MiB 73.72KiB 9.59MiB 9.76MiB 9.88MiB 9.9MiB 9.96MiB 10.01MiB 0.469665
('syslog_log2metric_splunk_hec_metrics', 'comparison') 363 9.45MiB 346.77KiB 8.91MiB 9.39MiB 9.84MiB 9.86MiB 9.92MiB 9.95MiB 0.0064123
('syslog_humio_logs', 'baseline') 363 9.38MiB 119.93KiB 9.13MiB 9.36MiB 9.55MiB 9.58MiB 9.62MiB 9.68MiB 0.245978
('syslog_log2metric_splunk_hec_metrics', 'baseline') 362 9.28MiB 83.87KiB 8.48MiB 9.29MiB 9.36MiB 9.38MiB 9.42MiB 9.44MiB -4.16444
('syslog_humio_logs', 'comparison') 364 9.15MiB 121.7KiB 8.84MiB 9.16MiB 9.3MiB 9.32MiB 9.34MiB 9.37MiB -0.167248
('syslog_log2metric_humio_metrics', 'comparison') 362 8.54MiB 95.08KiB 8.3MiB 8.54MiB 8.65MiB 8.68MiB 8.73MiB 8.74MiB -0.0900241
('splunk_hec_route_s3', 'comparison') 364 8.48MiB 2.43MiB 2.32MiB 8.21MiB 11.59MiB 13.34MiB 15.41MiB 16.66MiB 0.672243
('splunk_hec_route_s3', 'baseline') 324 8.2MiB 1.88MiB 2.71MiB 8.22MiB 10.44MiB 11.34MiB 12.9MiB 16.39MiB 0.369435
('syslog_log2metric_humio_metrics', 'baseline') 362 8.15MiB 256.42KiB 7.7MiB 8.24MiB 8.44MiB 8.47MiB 8.5MiB 8.54MiB -0.0213588
('syslog_regex_logs2metric_ddmetrics', 'comparison') 363 7.84MiB 612.58KiB 6.47MiB 8.01MiB 8.57MiB 8.6MiB 8.67MiB 8.7MiB -0.504092
('syslog_regex_logs2metric_ddmetrics', 'baseline') 362 7.68MiB 589.81KiB 6.35MiB 7.97MiB 8.25MiB 8.28MiB 8.35MiB 8.39MiB -0.687093
('syslog_loki', 'baseline') 364 6.98MiB 63.21KiB 6.77MiB 6.98MiB 7.05MiB 7.07MiB 7.1MiB 7.16MiB -0.436901
('syslog_loki', 'comparison') 362 6.91MiB 277.15KiB 6.29MiB 7.03MiB 7.15MiB 7.18MiB 7.22MiB 7.3MiB -1.02787
('http_to_http_acks', 'comparison') 362 5.73MiB 2.51MiB 0B 6.42MiB 9.32MiB 9.53MiB 10.44MiB 11.38MiB -0.0844089
('http_to_http_acks', 'baseline') 341 5.45MiB 2.46MiB 0B 5.0MiB 8.91MiB 9.5MiB 9.81MiB 10.22MiB 0.0453211
('splunk_transforms_splunk3', 'baseline') 343 4.58MiB 1.58MiB 976.02KiB 4.44MiB 6.56MiB 7.35MiB 8.18MiB 9.36MiB 0.226204
('splunk_transforms_splunk3', 'comparison') 343 4.55MiB 1.48MiB 793.07KiB 4.5MiB 6.47MiB 7.06MiB 8.04MiB 9.45MiB 0.216412
('http_datadog_filter_blackhole', 'comparison') 315 2.28MiB 1.01MiB 245.3KiB 2.18MiB 3.52MiB 4.1MiB 4.77MiB 5.49MiB 0.344015
('http_datadog_filter_blackhole', 'baseline') 363 2.15MiB 1.01MiB 0B 2.04MiB 3.35MiB 3.77MiB 4.95MiB 5.83MiB 0.626674
('http_pipelines_no_grok_blackhole', 'comparison') 361 2.11MiB 953.19KiB 31.98KiB 2.0MiB 3.32MiB 3.58MiB 4.57MiB 4.63MiB 0.31791
('http_pipelines_no_grok_blackhole', 'baseline') 362 2.02MiB 857.1KiB 490.01KiB 2.01MiB 3.14MiB 3.41MiB 3.92MiB 4.31MiB 0.2243
('http_pipelines_blackhole', 'baseline') 360 911.93KiB 768.26KiB 0B 739.42KiB 1.91MiB 2.27MiB 3.42MiB 4.09MiB 1.15849
('http_pipelines_blackhole_acks', 'baseline') 365 892.58KiB 680.28KiB 0B 736.04KiB 1.78MiB 2.15MiB 2.67MiB 3.34MiB 0.759634
('http_pipelines_blackhole', 'comparison') 316 879.09KiB 672.6KiB 0B 813.81KiB 1.73MiB 2.09MiB 2.42MiB 3.05MiB 0.601451
('http_pipelines_blackhole_acks', 'comparison') 362 847.86KiB 703.37KiB 0B 676.5KiB 1.76MiB 2.21MiB 2.8MiB 3.37MiB 1.00547

jszwedko added a commit that referenced this pull request Dec 27, 2021
Isolating from #10567

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>
Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>
@github-actions
Copy link

Soak Test Results

Baseline: e3dd70e
Comparison: 4c86a20
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Test units below are bytes/second/CPU, except for "skewness". The further "skewness" is from 0.0 the more indication that vector lacks consistency in behavior, making predictions of fitness in the field challenging.

The abbreviated table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. The abbreviated table will be omitted if no statistically interesting changes are observed.

No statistically interesting changes with confidence 90.00%.

Fine details of change detection per experiment.
experiment Δ mean Δ mean % baseline mean baseline stdev baseline outlier percentage comparison mean comparison stdev comparison outlier percentage t-statistic p-value erratic
datadog_agent_remap_datadog_logs 1.38MiB 3.74 36.94MiB 368.61KiB 1.16959 38.33MiB 1.26MiB 0 -19.6192 1.54047e-60 False
datadog_agent_remap_datadog_logs_acks 1.36MiB 3.56 38.09MiB 1.18MiB 0 39.44MiB 541.98KiB 0.552486 -19.8823 3.42586e-65 False
datadog_agent_remap_blackhole_acks 924.5KiB 2.95 30.64MiB 1.45MiB 0 31.54MiB 778.18KiB 0 -10.166 3.96831e-22 False
syslog_splunk_hec_logs 606.44KiB 6.42 9.22MiB 549.35KiB 0 9.82MiB 146.67KiB 0 -20.1954 7.74228e-64 False
splunk_hec_route_s3 530.41KiB 6.27 8.26MiB 2.27MiB 0.581395 8.77MiB 2.55MiB 0.628931 -2.74947 0.00613877 False
splunk_hec_to_splunk_hec_logs_acks 455.32KiB 2.46 18.07MiB 1.41MiB 1.38122 18.52MiB 1.35MiB 4.12088 -4.33539 1.66158e-05 False
syslog_loki 380.7KiB 5.35 6.94MiB 268.02KiB 17.5342 7.32MiB 236.77KiB 0 -20.4219 1.7371e-73 True
syslog_log2metric_splunk_hec_metrics 367.95KiB 3.94 9.13MiB 61.19KiB 0.57971 9.49MiB 331.99KiB 0 -20.7768 4.85311e-65 False
splunk_hec_to_splunk_hec_logs_noack 366.18KiB 1.86 19.27MiB 1.28MiB 0 19.62MiB 1.23MiB 0 -3.65789 0.000275105 False
splunk_transforms_splunk3 203.38KiB 4.39 4.53MiB 1.75MiB 0.58309 4.73MiB 1.76MiB 1.38122 -1.50556 0.13263 False
http_to_http_acks 167.4KiB 3.05 5.36MiB 2.53MiB 0 5.52MiB 2.64MiB 0 -0.831331 0.406076 True
http_datadog_filter_blackhole 93.14KiB 4.11 2.22MiB 1.05MiB 0.555556 2.31MiB 1.15MiB 0.828729 -1.11219 0.266431 False
datadog_agent_remap_blackhole 32.1KiB 0.11 29.21MiB 595.37KiB 1.37931 29.24MiB 625.04KiB 0 -0.67749 0.498341 False
http_pipelines_blackhole 12.36KiB 1.4 883.67KiB 637.29KiB 2.20386 896.02KiB 718.71KiB 2.54777 -0.235014 0.814274 False
http_pipelines_no_grok_blackhole -7.06KiB -0.33 2.11MiB 1.0MiB 2.76243 2.11MiB 1014.76KiB 0.824176 0.0932096 0.925763 False
syslog_humio_logs -53.73KiB -0.55 9.48MiB 377.99KiB 0 9.43MiB 315.51KiB 0 2.05786 0.0399748 False
http_pipelines_blackhole_acks -55.66KiB -6.06 918.42KiB 755.7KiB 2.19436 862.76KiB 793.45KiB 3.31492 0.936865 0.349163 False
splunk_hec_indexer_ack_blackhole -69.33KiB -0.29 23.04MiB 1.23MiB 0.550964 22.97MiB 1.14MiB 0.555556 0.769616 0.441781 False
syslog_log2metric_humio_metrics -188.66KiB -2.18 8.44MiB 381.32KiB 0 8.25MiB 283.6KiB 0 7.47828 2.37764e-13 False
syslog_regex_logs2metric_ddmetrics -277.96KiB -3.52 7.71MiB 610.36KiB 0 7.44MiB 549.53KiB 0 6.33638 4.26974e-10 False
fluent_elasticsearch -419.87KiB -0.59 68.97MiB 795.14KiB 1.64835 68.56MiB 2.17MiB 0.276243 3.39008 0.000760038 False
http_to_http_noack -547.86KiB -2.34 22.85MiB 1.55MiB 0.291545 22.31MiB 1.69MiB 0 4.3148 1.83576e-05 False
fluent_remap_aws_firehose -774.01KiB -1.04 72.77MiB 763.67KiB 2.20994 72.01MiB 1.02MiB 0.31348 10.9218 2.28526e-25 False
Fine details of each soak run.
(experiment, variant) total samples mean std min average p90 p95 p99 max skewness
('fluent_remap_aws_firehose', 'baseline') 362 72.77MiB 763.67KiB 69.0MiB 72.86MiB 73.59MiB 73.74MiB 74.12MiB 74.34MiB -1.30046
('fluent_remap_aws_firehose', 'comparison') 319 72.01MiB 1.02MiB 68.12MiB 71.81MiB 73.43MiB 73.63MiB 73.89MiB 74.06MiB -0.0479036
('fluent_elasticsearch', 'baseline') 364 68.97MiB 795.14KiB 66.32MiB 68.98MiB 69.95MiB 70.13MiB 70.6MiB 71.24MiB -0.207035
('fluent_elasticsearch', 'comparison') 362 68.56MiB 2.17MiB 50.99MiB 68.83MiB 70.89MiB 71.06MiB 71.41MiB 72.15MiB -1.51432
('datadog_agent_remap_datadog_logs_acks', 'comparison') 362 39.44MiB 541.98KiB 37.94MiB 39.51MiB 40.06MiB 40.18MiB 40.6MiB 40.69MiB -0.386283
('datadog_agent_remap_datadog_logs', 'comparison') 345 38.33MiB 1.26MiB 36.58MiB 37.71MiB 39.89MiB 40.11MiB 40.41MiB 40.88MiB 0.159949
('datadog_agent_remap_datadog_logs_acks', 'baseline') 361 38.09MiB 1.18MiB 36.02MiB 37.86MiB 39.5MiB 39.66MiB 39.93MiB 39.99MiB -0.0626397
('datadog_agent_remap_datadog_logs', 'baseline') 342 36.94MiB 368.61KiB 35.55MiB 36.95MiB 37.36MiB 37.51MiB 37.67MiB 37.75MiB -0.458661
('datadog_agent_remap_blackhole_acks', 'comparison') 380 31.54MiB 778.18KiB 29.71MiB 31.6MiB 32.46MiB 32.63MiB 33.15MiB 33.35MiB -0.104325
('datadog_agent_remap_blackhole_acks', 'baseline') 331 30.64MiB 1.45MiB 26.95MiB 31.34MiB 32.12MiB 32.28MiB 32.5MiB 32.73MiB -0.369604
('datadog_agent_remap_blackhole', 'comparison') 382 29.24MiB 625.04KiB 27.84MiB 29.21MiB 30.0MiB 30.31MiB 30.61MiB 30.81MiB 0.175576
('datadog_agent_remap_blackhole', 'baseline') 290 29.21MiB 595.37KiB 27.57MiB 29.23MiB 29.92MiB 30.19MiB 30.47MiB 30.63MiB -0.163048
('splunk_hec_indexer_ack_blackhole', 'baseline') 363 23.04MiB 1.23MiB 20.05MiB 23.06MiB 24.58MiB 25.09MiB 25.98MiB 27.26MiB 0.14298
('splunk_hec_indexer_ack_blackhole', 'comparison') 360 22.97MiB 1.14MiB 20.26MiB 22.94MiB 24.5MiB 24.88MiB 25.72MiB 26.4MiB 0.177699
('http_to_http_noack', 'baseline') 343 22.85MiB 1.55MiB 18.23MiB 22.93MiB 24.69MiB 25.16MiB 26.12MiB 26.63MiB -0.181827
('http_to_http_noack', 'comparison') 341 22.31MiB 1.69MiB 18.29MiB 22.25MiB 24.43MiB 25.17MiB 26.41MiB 26.87MiB 0.166105
('splunk_hec_to_splunk_hec_logs_noack', 'comparison') 298 19.62MiB 1.23MiB 16.6MiB 19.67MiB 21.22MiB 21.76MiB 22.29MiB 22.68MiB 0.0715532
('splunk_hec_to_splunk_hec_logs_noack', 'baseline') 367 19.27MiB 1.28MiB 15.94MiB 19.28MiB 20.85MiB 21.25MiB 22.18MiB 22.76MiB -0.120633
('splunk_hec_to_splunk_hec_logs_acks', 'comparison') 364 18.52MiB 1.35MiB 14.12MiB 18.62MiB 20.06MiB 20.47MiB 21.21MiB 22.08MiB -0.696372
('splunk_hec_to_splunk_hec_logs_acks', 'baseline') 362 18.07MiB 1.41MiB 13.23MiB 18.03MiB 19.93MiB 20.41MiB 21.18MiB 21.6MiB -0.178895
('syslog_splunk_hec_logs', 'comparison') 316 9.82MiB 146.67KiB 9.43MiB 9.87MiB 9.95MiB 9.97MiB 10.0MiB 10.03MiB -0.975079
('syslog_log2metric_splunk_hec_metrics', 'comparison') 364 9.49MiB 331.99KiB 9.03MiB 9.53MiB 9.85MiB 9.87MiB 9.93MiB 9.99MiB 0.0103196
('syslog_humio_logs', 'baseline') 365 9.48MiB 377.99KiB 8.92MiB 9.26MiB 9.9MiB 9.92MiB 9.97MiB 9.99MiB 0.00273785
('syslog_humio_logs', 'comparison') 343 9.43MiB 315.51KiB 8.94MiB 9.25MiB 9.83MiB 9.85MiB 9.88MiB 9.93MiB 0.310181
('syslog_splunk_hec_logs', 'baseline') 362 9.22MiB 549.35KiB 8.48MiB 9.21MiB 9.8MiB 9.82MiB 9.85MiB 9.92MiB -0.0153691
('syslog_log2metric_splunk_hec_metrics', 'baseline') 345 9.13MiB 61.19KiB 8.96MiB 9.13MiB 9.21MiB 9.23MiB 9.26MiB 9.29MiB 0.039377
('splunk_hec_route_s3', 'comparison') 318 8.77MiB 2.55MiB 2.41MiB 8.79MiB 12.06MiB 13.0MiB 14.67MiB 18.18MiB 0.255146
('syslog_log2metric_humio_metrics', 'baseline') 363 8.44MiB 381.32KiB 7.38MiB 8.64MiB 8.82MiB 8.84MiB 8.89MiB 8.93MiB -0.424384
('splunk_hec_route_s3', 'baseline') 344 8.26MiB 2.27MiB 3.04MiB 8.36MiB 11.31MiB 12.12MiB 13.42MiB 15.23MiB 0.124509
('syslog_log2metric_humio_metrics', 'comparison') 341 8.25MiB 283.6KiB 7.56MiB 8.37MiB 8.54MiB 8.58MiB 8.61MiB 8.65MiB -0.51137
('syslog_regex_logs2metric_ddmetrics', 'baseline') 341 7.71MiB 610.36KiB 6.4MiB 8.02MiB 8.32MiB 8.35MiB 8.41MiB 8.45MiB -0.66295
('syslog_regex_logs2metric_ddmetrics', 'comparison') 363 7.44MiB 549.53KiB 6.21MiB 7.65MiB 7.99MiB 8.03MiB 8.06MiB 8.32MiB -0.795826
('syslog_loki', 'comparison') 372 7.32MiB 236.77KiB 6.94MiB 7.37MiB 7.6MiB 7.62MiB 7.65MiB 7.69MiB -0.00410229
('syslog_loki', 'baseline') 365 6.94MiB 268.02KiB 5.66MiB 7.04MiB 7.15MiB 7.17MiB 7.22MiB 7.23MiB -1.67255
('http_to_http_acks', 'comparison') 365 5.52MiB 2.64MiB 1.19MiB 5.12MiB 9.52MiB 9.8MiB 10.59MiB 11.98MiB 0.20556
('http_to_http_acks', 'baseline') 326 5.36MiB 2.53MiB 1.31MiB 4.85MiB 9.26MiB 9.56MiB 10.03MiB 12.36MiB 0.236825
('splunk_transforms_splunk3', 'comparison') 362 4.73MiB 1.76MiB 1.16MiB 4.65MiB 7.06MiB 7.98MiB 9.53MiB 12.1MiB 0.697928
('splunk_transforms_splunk3', 'baseline') 343 4.53MiB 1.75MiB 884.77KiB 4.38MiB 6.85MiB 7.54MiB 8.64MiB 9.95MiB 0.300067
('http_datadog_filter_blackhole', 'comparison') 362 2.31MiB 1.15MiB 246.03KiB 2.13MiB 3.91MiB 4.27MiB 5.1MiB 6.37MiB 0.580669
('http_datadog_filter_blackhole', 'baseline') 360 2.22MiB 1.05MiB 278.01KiB 2.04MiB 3.76MiB 4.25MiB 4.72MiB 5.44MiB 0.563734
('http_pipelines_no_grok_blackhole', 'baseline') 362 2.11MiB 1.0MiB 124.71KiB 2.02MiB 3.53MiB 3.94MiB 4.63MiB 5.05MiB 0.537686
('http_pipelines_no_grok_blackhole', 'comparison') 364 2.11MiB 1014.76KiB 0B 1.99MiB 3.49MiB 3.88MiB 4.42MiB 5.02MiB 0.408136
('http_pipelines_blackhole_acks', 'baseline') 319 918.42KiB 755.7KiB 0B 768.02KiB 1.91MiB 2.31MiB 2.96MiB 3.68MiB 0.986432
('http_pipelines_blackhole', 'comparison') 314 896.02KiB 718.71KiB 0B 738.76KiB 1.91MiB 2.21MiB 2.92MiB 3.34MiB 0.988125
('http_pipelines_blackhole', 'baseline') 363 883.67KiB 637.29KiB 0B 797.82KiB 1.73MiB 2.03MiB 2.71MiB 2.98MiB 0.829091
('http_pipelines_blackhole_acks', 'comparison') 362 862.76KiB 793.45KiB 0B 695.21KiB 1.92MiB 2.21MiB 3.46MiB 4.96MiB 1.50812

Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>
@github-actions
Copy link

Soak Test Results

Baseline: f228891
Comparison: da7888f
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Test units below are bytes/second/CPU, except for "skewness". The further "skewness" is from 0.0 the more indication that vector lacks consistency in behavior, making predictions of fitness in the field challenging.

The abbreviated table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. The abbreviated table will be omitted if no statistically interesting changes are observed.

No statistically interesting changes with confidence 90.00%.

Fine details of change detection per experiment.
experiment Δ mean Δ mean % baseline mean baseline stdev baseline outlier percentage comparison mean comparison stdev comparison outlier percentage t-statistic p-value erratic
splunk_hec_indexer_ack_blackhole 1.21MiB 5.31 22.69MiB 1.2MiB 0.647249 23.89MiB 986.72KiB 0.275482 -14.1922 1.5734e-39 False
fluent_remap_aws_firehose 1.2MiB 1.69 71.02MiB 2.15MiB 0 72.22MiB 1.05MiB 0.552486 -9.59339 3.40566e-20 False
datadog_agent_remap_datadog_logs_acks 1.05MiB 2.79 37.53MiB 864.37KiB 0 38.58MiB 1.09MiB 0 -14.4917 1.12053e-41 False
http_to_http_acks 753.79KiB 14.53 5.07MiB 1.87MiB 0 5.8MiB 2.38MiB 0 -4.35931 1.53556e-05 True
http_to_http_noack 644.8KiB 2.81 22.43MiB 1.64MiB 0.274725 23.06MiB 1.61MiB 0.285714 -5.17944 2.90026e-07 False
datadog_agent_remap_datadog_logs 564.71KiB 1.45 38.09MiB 1.18MiB 0 38.64MiB 1.22MiB 0 -6.18668 1.02594e-09 False
splunk_transforms_splunk3 213.47KiB 4.67 4.46MiB 1.62MiB 1.66205 4.67MiB 1.54MiB 0.940439 -1.71974 0.0859391 False
syslog_log2metric_humio_metrics 165.49KiB 1.9 8.5MiB 74.65KiB 0.277008 8.66MiB 74.46KiB 3.03867 -29.8442 4.61614e-128 False
syslog_loki 159.43KiB 2.25 6.91MiB 83.51KiB 0.831025 7.07MiB 185.72KiB 9.11602 -14.8934 8.36934e-42 True
syslog_splunk_hec_logs 117.11KiB 1.18 9.73MiB 93.14KiB 2.33918 9.84MiB 92.25KiB 0 -16.7958 1.81321e-53 False
http_datadog_filter_blackhole 115.68KiB 5.25 2.15MiB 1.18MiB 1.37741 2.26MiB 1017.84KiB 0 -1.39537 0.163345 False
datadog_agent_remap_blackhole 58.92KiB 0.2 28.75MiB 497.26KiB 0.787402 28.81MiB 609.28KiB 0.52356 -1.46351 0.143757 False
syslog_log2metric_splunk_hec_metrics 55.74KiB 0.6 9.08MiB 89.52KiB 0 9.13MiB 138.13KiB 9.31507 -6.46682 2.02123e-10 False
splunk_hec_to_splunk_hec_logs_acks 35.04KiB 0.18 18.73MiB 1.34MiB 1.17302 18.77MiB 1.37MiB 3.02198 -0.335848 0.737086 False
splunk_hec_route_s3 26.04KiB 0.31 8.24MiB 1.89MiB 0.552486 8.26MiB 2.41MiB 0.879765 -0.155123 0.876773 False
http_pipelines_blackhole 5.44KiB 0.6 908.06KiB 714.16KiB 0 913.5KiB 743.44KiB 2.82132 -0.0969128 0.922825 False
syslog_regex_logs2metric_ddmetrics -16.24KiB -0.2 7.81MiB 583.63KiB 0 7.8MiB 607.35KiB 0 0.367031 0.713704 False
http_pipelines_blackhole_acks -73.98KiB -8.3 890.92KiB 768.46KiB 2.82132 816.95KiB 687.67KiB 0.826446 1.31725 0.188225 False
http_pipelines_no_grok_blackhole -78.3KiB -3.61 2.12MiB 917.45KiB 0 2.04MiB 1016.28KiB 0.552486 1.08808 0.276927 False
syslog_humio_logs -326.75KiB -3.32 9.62MiB 155.64KiB 4.94505 9.3MiB 259.08KiB 0 20.1769 2.78508e-68 False
splunk_hec_to_splunk_hec_logs_noack -583.27KiB -2.91 19.56MiB 1.13MiB 1.37363 18.99MiB 1.17MiB 0 6.44207 2.27324e-10 False
datadog_agent_remap_blackhole_acks -1.06MiB -3.41 30.93MiB 798.02KiB 0 29.88MiB 724.07KiB 0.263158 18.8144 9.33245e-64 False
fluent_elasticsearch -1.74MiB -2.54 68.69MiB 660.09KiB 0 66.95MiB 1.18MiB 0.554017 23.9613 1.52214e-88 False
Fine details of each soak run.
(experiment, variant) total samples mean std min average p90 p95 p99 max skewness
('fluent_remap_aws_firehose', 'comparison') 362 72.22MiB 1.05MiB 67.0MiB 72.14MiB 73.68MiB 74.03MiB 74.29MiB 74.53MiB -0.294658
('fluent_remap_aws_firehose', 'baseline') 363 71.02MiB 2.15MiB 64.7MiB 71.81MiB 73.36MiB 73.56MiB 73.83MiB 74.1MiB -0.671064
('fluent_elasticsearch', 'baseline') 295 68.69MiB 660.09KiB 67.26MiB 68.64MiB 69.56MiB 69.84MiB 70.32MiB 70.45MiB 0.321977
('fluent_elasticsearch', 'comparison') 361 66.95MiB 1.18MiB 52.09MiB 66.94MiB 68.13MiB 68.39MiB 69.16MiB 69.93MiB -5.35933
('datadog_agent_remap_datadog_logs', 'comparison') 365 38.64MiB 1.22MiB 36.68MiB 38.25MiB 40.11MiB 40.24MiB 40.51MiB 40.67MiB 0.0180442
('datadog_agent_remap_datadog_logs_acks', 'comparison') 362 38.58MiB 1.09MiB 36.62MiB 38.48MiB 39.9MiB 40.04MiB 40.31MiB 40.45MiB 0.0117977
('datadog_agent_remap_datadog_logs', 'baseline') 364 38.09MiB 1.18MiB 36.21MiB 38.18MiB 39.51MiB 39.65MiB 39.8MiB 39.99MiB -0.00906523
('datadog_agent_remap_datadog_logs_acks', 'baseline') 364 37.53MiB 864.37KiB 35.98MiB 37.47MiB 38.6MiB 38.88MiB 39.17MiB 39.39MiB 0.122479
('datadog_agent_remap_blackhole_acks', 'baseline') 331 30.93MiB 798.02KiB 28.9MiB 30.77MiB 31.99MiB 32.11MiB 32.37MiB 32.51MiB 0.0824885
('datadog_agent_remap_blackhole_acks', 'comparison') 380 29.88MiB 724.07KiB 27.87MiB 29.9MiB 30.79MiB 30.99MiB 31.38MiB 31.82MiB -0.0586033
('datadog_agent_remap_blackhole', 'comparison') 382 28.81MiB 609.28KiB 27.28MiB 28.77MiB 29.55MiB 29.82MiB 30.22MiB 30.82MiB 0.189755
('datadog_agent_remap_blackhole', 'baseline') 381 28.75MiB 497.26KiB 27.23MiB 28.76MiB 29.37MiB 29.54MiB 29.79MiB 30.08MiB -0.118964
('splunk_hec_indexer_ack_blackhole', 'comparison') 363 23.89MiB 986.72KiB 21.63MiB 23.84MiB 25.08MiB 25.47MiB 26.08MiB 27.41MiB 0.14933
('http_to_http_noack', 'comparison') 350 23.06MiB 1.61MiB 18.02MiB 23.15MiB 25.05MiB 25.49MiB 26.44MiB 27.29MiB -0.232951
('splunk_hec_indexer_ack_blackhole', 'baseline') 309 22.69MiB 1.2MiB 19.22MiB 22.74MiB 24.29MiB 24.56MiB 25.0MiB 26.34MiB -0.014016
('http_to_http_noack', 'baseline') 364 22.43MiB 1.64MiB 17.57MiB 22.56MiB 24.71MiB 25.1MiB 26.07MiB 26.78MiB 0.0232871
('splunk_hec_to_splunk_hec_logs_noack', 'baseline') 364 19.56MiB 1.13MiB 16.63MiB 19.55MiB 20.98MiB 21.27MiB 22.44MiB 23.27MiB 0.0846319
('splunk_hec_to_splunk_hec_logs_noack', 'comparison') 318 18.99MiB 1.17MiB 15.73MiB 18.98MiB 20.49MiB 20.9MiB 21.72MiB 22.11MiB 0.0468524
('splunk_hec_to_splunk_hec_logs_acks', 'comparison') 364 18.77MiB 1.37MiB 13.71MiB 18.68MiB 20.5MiB 20.92MiB 21.46MiB 23.12MiB -0.350219
('splunk_hec_to_splunk_hec_logs_acks', 'baseline') 341 18.73MiB 1.34MiB 14.21MiB 18.8MiB 20.47MiB 20.77MiB 21.26MiB 21.87MiB -0.355605
('syslog_splunk_hec_logs', 'comparison') 366 9.84MiB 92.25KiB 9.6MiB 9.84MiB 9.96MiB 9.98MiB 10.01MiB 10.04MiB -0.107577
('syslog_splunk_hec_logs', 'baseline') 342 9.73MiB 93.14KiB 9.13MiB 9.73MiB 9.81MiB 9.83MiB 9.87MiB 9.9MiB -3.2738
('syslog_humio_logs', 'baseline') 364 9.62MiB 155.64KiB 8.92MiB 9.63MiB 9.79MiB 9.81MiB 9.85MiB 9.89MiB -1.68108
('syslog_humio_logs', 'comparison') 343 9.3MiB 259.08KiB 8.84MiB 9.4MiB 9.59MiB 9.64MiB 9.77MiB 9.83MiB 0.0163559
('syslog_log2metric_splunk_hec_metrics', 'comparison') 365 9.13MiB 138.13KiB 8.57MiB 9.16MiB 9.25MiB 9.27MiB 9.3MiB 9.31MiB -2.04149
('syslog_log2metric_splunk_hec_metrics', 'baseline') 364 9.08MiB 89.52KiB 8.88MiB 9.08MiB 9.19MiB 9.21MiB 9.27MiB 9.32MiB -0.0329936
('syslog_log2metric_humio_metrics', 'comparison') 362 8.66MiB 74.46KiB 8.26MiB 8.66MiB 8.73MiB 8.75MiB 8.82MiB 8.89MiB -0.984405
('syslog_log2metric_humio_metrics', 'baseline') 361 8.5MiB 74.65KiB 8.25MiB 8.5MiB 8.6MiB 8.61MiB 8.66MiB 8.69MiB 0.0736083
('splunk_hec_route_s3', 'comparison') 341 8.26MiB 2.41MiB 1.91MiB 8.22MiB 11.41MiB 12.1MiB 13.64MiB 15.64MiB 0.000139864
('splunk_hec_route_s3', 'baseline') 362 8.24MiB 1.89MiB 3.34MiB 8.16MiB 10.57MiB 11.68MiB 12.88MiB 14.48MiB 0.293412
('syslog_regex_logs2metric_ddmetrics', 'baseline') 363 7.81MiB 583.63KiB 6.47MiB 7.98MiB 8.46MiB 8.5MiB 8.54MiB 8.58MiB -0.537735
('syslog_regex_logs2metric_ddmetrics', 'comparison') 362 7.8MiB 607.35KiB 6.67MiB 8.16MiB 8.33MiB 8.36MiB 8.41MiB 8.48MiB -0.670957
('syslog_loki', 'comparison') 362 7.07MiB 185.72KiB 6.28MiB 7.12MiB 7.21MiB 7.22MiB 7.26MiB 7.29MiB -2.39163
('syslog_loki', 'baseline') 361 6.91MiB 83.51KiB 6.64MiB 6.92MiB 7.01MiB 7.04MiB 7.08MiB 7.11MiB -0.334367
('http_to_http_acks', 'comparison') 317 5.8MiB 2.38MiB 1.13MiB 5.84MiB 9.34MiB 9.57MiB 10.11MiB 11.65MiB 0.0778626
('http_to_http_acks', 'baseline') 325 5.07MiB 1.87MiB 486.96KiB 4.88MiB 7.27MiB 7.62MiB 9.41MiB 10.4MiB 0.119104
('splunk_transforms_splunk3', 'comparison') 319 4.67MiB 1.54MiB 1.58MiB 4.56MiB 6.64MiB 7.57MiB 8.46MiB 10.04MiB 0.521333
('splunk_transforms_splunk3', 'baseline') 361 4.46MiB 1.62MiB 945.62KiB 4.32MiB 6.58MiB 7.48MiB 8.96MiB 10.37MiB 0.539988
('http_datadog_filter_blackhole', 'comparison') 362 2.26MiB 1017.84KiB 0B 2.2MiB 3.67MiB 4.0MiB 4.5MiB 4.87MiB 0.249485
('http_datadog_filter_blackhole', 'baseline') 363 2.15MiB 1.18MiB 0B 2.03MiB 3.79MiB 4.29MiB 5.52MiB 5.85MiB 0.601762
('http_pipelines_no_grok_blackhole', 'baseline') 362 2.12MiB 917.45KiB 312.24KiB 2.04MiB 3.35MiB 3.56MiB 4.45MiB 4.49MiB 0.384948
('http_pipelines_no_grok_blackhole', 'comparison') 362 2.04MiB 1016.28KiB 95.94KiB 1.83MiB 3.41MiB 3.64MiB 4.29MiB 5.95MiB 0.511874
('http_pipelines_blackhole', 'comparison') 319 913.5KiB 743.44KiB 0B 860.75KiB 1.89MiB 2.21MiB 3.06MiB 3.34MiB 0.878692
('http_pipelines_blackhole', 'baseline') 360 908.06KiB 714.16KiB 0B 835.3KiB 1.92MiB 2.18MiB 2.61MiB 2.9MiB 0.646564
('http_pipelines_blackhole_acks', 'baseline') 319 890.92KiB 768.46KiB 0B 771.4KiB 1.91MiB 2.21MiB 3.38MiB 3.94MiB 1.1572
('http_pipelines_blackhole_acks', 'comparison') 363 816.95KiB 687.67KiB 0B 617.81KiB 1.61MiB 2.06MiB 2.53MiB 4.06MiB 1.22686

Copy link
Member Author

@jszwedko jszwedko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't approve since I'm the author, but didn't make any of these changes, so this comment indicates approval 😄

@jszwedko jszwedko merged commit 2d6dd1c into master Dec 28, 2021
@jszwedko jszwedko deleted the revert-10566-revert-10521-spencer/10467-update-k8s-e2e-to-new-helm branch December 28, 2021 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci-condition: k8s e2e all targets Run Kubernetes E2E test suite for all targets (instead of just the essential subset) ci-condition: k8s e2e tests enable Run Kubernetes E2E test suite for this PR domain: ci Anything related to Vector's CI environment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants