Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(ci): Revert update k8s-e2e to use new helm chart #10566

Merged

Conversation

jszwedko
Copy link
Member

Reverts #10521

Appears to be causing CI failures. Reverting so we can re-open as a PR to investigate.

https://github.com/vectordotdev/vector/actions/runs/1608811257

@netlify
Copy link

netlify bot commented Dec 22, 2021

✔️ Deploy Preview for vector-project ready!

🔨 Explore the source changes: 7391c64

🔍 Inspect the deploy log: https://app.netlify.com/sites/vector-project/deploys/61c3ab9efaeddf0007b5fe64

😎 Browse the preview: https://deploy-preview-10566--vector-project.netlify.app

@github-actions github-actions bot added the domain: ci Anything related to Vector's CI environment label Dec 22, 2021
Copy link
Contributor

@spencergilbert spencergilbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😢

@jszwedko jszwedko merged commit 738e6c2 into master Dec 22, 2021
@jszwedko jszwedko deleted the revert-10521-spencer/10467-update-k8s-e2e-to-new-helm branch December 22, 2021 22:53
jszwedko added a commit that referenced this pull request Dec 22, 2021
@github-actions
Copy link

Soak Test Results

Baseline: 633a27c
Comparison: 7391c64
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Test units below are bytes/second/CPU, except for "skewness". The further "skewness" is from 0.0 the more indication that vector lacks consistency in behavior, making predictions of fitness in the field challenging.

The abbreviated table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. The abbreviated table will be omitted if no statistically interesting changes are observed.

No statistically interesting changes with confidence 90.00%.

Fine details of change detection per experiment.
experiment Δ mean Δ mean % baseline mean baseline stdev baseline outlier percentage comparison mean comparison stdev comparison outlier percentage t-statistic p-value erratic
splunk_hec_route_s3 503.82KiB 5.95 8.27MiB 2.0MiB 0 8.76MiB 1.84MiB 0.828729 -3.38432 0.000754136 False
datadog_agent_remap_datadog_logs_acks 385.74KiB 1.03 36.67MiB 589.1KiB 0 37.05MiB 586.14KiB 0 -8.88524 4.91605e-18 False
http_to_http_acks 303.43KiB 5.81 5.1MiB 2.39MiB 0 5.4MiB 2.36MiB 0 -1.67807 0.0937659 True
datadog_agent_remap_blackhole_acks 249.39KiB 0.79 30.79MiB 1.5MiB 0 31.03MiB 650.24KiB 0 -2.91039 0.00376766 False
splunk_transforms_splunk3 232.08KiB 5.19 4.36MiB 1.55MiB 1.38122 4.59MiB 1.56MiB 1.92837 -1.9659 0.0496923 False
http_to_http_noack 112.79KiB 0.5 22.09MiB 1.61MiB 1.93906 22.2MiB 1.65MiB 0 -0.908098 0.364129 False
fluent_elasticsearch 111.9KiB 0.17 64.69MiB 643.52KiB 0.828729 64.8MiB 591.79KiB 1.9337 -2.4352 0.0151262 False
http_pipelines_blackhole_acks 75.99KiB 9.3 817.14KiB 729.86KiB 1.91781 893.13KiB 744.99KiB 2.1875 -1.34463 0.1792 False
http_pipelines_no_grok_blackhole 8.45KiB 0.4 2.06MiB 1022.01KiB 0.550964 2.07MiB 1.1MiB 0.555556 -0.105762 0.915801 False
http_pipelines_blackhole 6.28KiB 0.76 822.0KiB 682.49KiB 0.833333 828.28KiB 703.2KiB 0.831025 -0.121707 0.903165 False
syslog_splunk_hec_logs -7.06KiB -0.07 9.34MiB 510.43KiB 0 9.33MiB 266.68KiB 0 0.226146 0.821169 False
http_datadog_filter_blackhole -45.92KiB -2.04 2.2MiB 1.05MiB 0.554017 2.15MiB 1.01MiB 1.38122 0.585259 0.558557 False
syslog_loki -84.02KiB -1.13 7.27MiB 162.24KiB 0 7.18MiB 349.78KiB 14.5205 4.161 3.71365e-05 True
syslog_humio_logs -148.92KiB -1.52 9.58MiB 351.56KiB 0 9.43MiB 334.14KiB 0 5.81487 9.18056e-09 False
syslog_log2metric_splunk_hec_metrics -179.52KiB -1.86 9.41MiB 284.67KiB 0 9.23MiB 60.98KiB 0.549451 11.4186 4.45696e-26 False
syslog_log2metric_humio_metrics -270.74KiB -3.17 8.33MiB 113.8KiB 0 8.07MiB 227.31KiB 0 20.3411 2.08096e-68 False
syslog_regex_logs2metric_ddmetrics -300.79KiB -3.89 7.55MiB 595.81KiB 0 7.26MiB 498.24KiB 0 7.21221 1.46446e-12 False
splunk_hec_to_splunk_hec_logs_noack -341.82KiB -1.75 19.02MiB 1.34MiB 0.584795 18.69MiB 1.39MiB 0.290698 3.19574 0.00145895 False
splunk_hec_to_splunk_hec_logs_acks -560.83KiB -2.98 18.37MiB 1.53MiB 0.550964 17.82MiB 1.41MiB 0.294118 4.94345 9.61274e-07 False
datadog_agent_remap_datadog_logs -739.68KiB -1.84 39.36MiB 462.47KiB 0.879765 38.63MiB 1.08MiB 0 11.7235 3.6059e-28 False
splunk_hec_indexer_ack_blackhole -769.03KiB -3.14 23.92MiB 1.01MiB 1.10193 23.17MiB 1.27MiB 0 8.78899 1.19858e-17 False
datadog_agent_remap_blackhole -955.8KiB -3.09 30.23MiB 451.77KiB 0.520833 29.29MiB 751.73KiB 0 21.2737 7.98965e-76 False
fluent_remap_aws_firehose -1.1MiB -1.53 72.05MiB 1.17MiB 0 70.95MiB 2.17MiB 0 8.62129 6.39199e-17 False
Fine details of each soak run.
(experiment, variant) total samples mean std min average p90 p95 p99 max skewness
('fluent_remap_aws_firehose', 'baseline') 362 72.05MiB 1.17MiB 69.45MiB 72.2MiB 73.47MiB 73.7MiB 73.96MiB 74.17MiB -0.127982
('fluent_remap_aws_firehose', 'comparison') 374 70.95MiB 2.17MiB 67.35MiB 71.88MiB 73.52MiB 73.71MiB 74.16MiB 74.29MiB -0.0582723
('fluent_elasticsearch', 'comparison') 362 64.8MiB 591.79KiB 62.77MiB 64.81MiB 65.52MiB 65.72MiB 66.08MiB 66.36MiB -0.248747
('fluent_elasticsearch', 'baseline') 362 64.69MiB 643.52KiB 62.68MiB 64.7MiB 65.47MiB 65.71MiB 66.11MiB 66.35MiB -0.134279
('datadog_agent_remap_datadog_logs', 'baseline') 341 39.36MiB 462.47KiB 38.15MiB 39.34MiB 39.91MiB 40.09MiB 40.59MiB 40.76MiB 0.199883
('datadog_agent_remap_datadog_logs', 'comparison') 364 38.63MiB 1.08MiB 36.62MiB 38.79MiB 39.98MiB 40.12MiB 40.38MiB 40.56MiB -0.0147848
('datadog_agent_remap_datadog_logs_acks', 'comparison') 371 37.05MiB 586.14KiB 35.4MiB 37.04MiB 37.79MiB 37.98MiB 38.23MiB 38.46MiB -0.0372783
('datadog_agent_remap_datadog_logs_acks', 'baseline') 362 36.67MiB 589.1KiB 35.42MiB 36.67MiB 37.42MiB 37.57MiB 37.85MiB 37.96MiB 0.0252158
('datadog_agent_remap_blackhole_acks', 'comparison') 382 31.03MiB 650.24KiB 29.38MiB 31.06MiB 31.87MiB 31.97MiB 32.19MiB 32.52MiB -0.228074
('datadog_agent_remap_blackhole_acks', 'baseline') 380 30.79MiB 1.5MiB 27.92MiB 30.93MiB 32.49MiB 32.69MiB 33.13MiB 33.32MiB -0.0917497
('datadog_agent_remap_blackhole', 'baseline') 384 30.23MiB 451.77KiB 28.74MiB 30.26MiB 30.75MiB 30.95MiB 31.21MiB 31.36MiB -0.112247
('datadog_agent_remap_blackhole', 'comparison') 380 29.29MiB 751.73KiB 27.46MiB 29.29MiB 30.23MiB 30.42MiB 30.82MiB 31.0MiB -0.0404183
('splunk_hec_indexer_ack_blackhole', 'baseline') 363 23.92MiB 1.01MiB 21.04MiB 23.87MiB 25.23MiB 25.69MiB 26.35MiB 26.94MiB 0.169528
('splunk_hec_indexer_ack_blackhole', 'comparison') 362 23.17MiB 1.27MiB 19.7MiB 23.08MiB 24.76MiB 25.29MiB 26.23MiB 26.58MiB 0.0582674
('http_to_http_noack', 'comparison') 363 22.2MiB 1.65MiB 17.72MiB 22.13MiB 24.38MiB 24.87MiB 25.69MiB 26.51MiB 0.0529204
('http_to_http_noack', 'baseline') 361 22.09MiB 1.61MiB 17.57MiB 21.98MiB 24.13MiB 24.9MiB 26.39MiB 28.42MiB 0.449848
('splunk_hec_to_splunk_hec_logs_noack', 'baseline') 342 19.02MiB 1.34MiB 15.26MiB 18.98MiB 20.79MiB 21.33MiB 22.12MiB 22.68MiB 0.0270861
('splunk_hec_to_splunk_hec_logs_noack', 'comparison') 344 18.69MiB 1.39MiB 14.87MiB 18.77MiB 20.45MiB 20.97MiB 21.8MiB 22.32MiB -0.0913886
('splunk_hec_to_splunk_hec_logs_acks', 'baseline') 363 18.37MiB 1.53MiB 12.93MiB 18.44MiB 20.17MiB 20.71MiB 21.34MiB 21.87MiB -0.366735
('splunk_hec_to_splunk_hec_logs_acks', 'comparison') 340 17.82MiB 1.41MiB 12.87MiB 17.8MiB 19.61MiB 19.88MiB 20.5MiB 21.31MiB -0.30887
('syslog_humio_logs', 'baseline') 349 9.58MiB 351.56KiB 9.03MiB 9.82MiB 9.93MiB 9.96MiB 9.98MiB 10.04MiB -0.179723
('syslog_humio_logs', 'comparison') 370 9.43MiB 334.14KiB 8.98MiB 9.24MiB 9.81MiB 9.83MiB 9.89MiB 9.91MiB 0.0501696
('syslog_log2metric_splunk_hec_metrics', 'baseline') 342 9.41MiB 284.67KiB 8.99MiB 9.55MiB 9.72MiB 9.75MiB 9.79MiB 9.84MiB -0.0919355
('syslog_splunk_hec_logs', 'baseline') 362 9.34MiB 510.43KiB 8.43MiB 9.25MiB 9.89MiB 9.9MiB 9.96MiB 9.97MiB -0.0217921
('syslog_splunk_hec_logs', 'comparison') 278 9.33MiB 266.68KiB 8.97MiB 9.21MiB 9.79MiB 9.84MiB 9.91MiB 9.96MiB 0.715613
('syslog_log2metric_splunk_hec_metrics', 'comparison') 364 9.23MiB 60.98KiB 9.07MiB 9.23MiB 9.3MiB 9.32MiB 9.35MiB 9.41MiB -0.125031
('splunk_hec_route_s3', 'comparison') 362 8.76MiB 1.84MiB 4.35MiB 8.61MiB 11.2MiB 11.89MiB 13.04MiB 15.41MiB 0.399959
('syslog_log2metric_humio_metrics', 'baseline') 372 8.33MiB 113.8KiB 8.06MiB 8.34MiB 8.48MiB 8.5MiB 8.54MiB 8.58MiB -0.0365983
('splunk_hec_route_s3', 'baseline') 341 8.27MiB 2.0MiB 2.77MiB 8.25MiB 10.88MiB 11.98MiB 12.8MiB 13.38MiB 0.195577
('syslog_log2metric_humio_metrics', 'comparison') 363 8.07MiB 227.31KiB 7.4MiB 8.16MiB 8.3MiB 8.32MiB 8.36MiB 8.42MiB -0.723561
('syslog_regex_logs2metric_ddmetrics', 'baseline') 363 7.55MiB 595.81KiB 6.06MiB 7.63MiB 8.27MiB 8.32MiB 8.38MiB 8.41MiB -0.631996
('syslog_loki', 'baseline') 363 7.27MiB 162.24KiB 6.94MiB 7.24MiB 7.47MiB 7.49MiB 7.53MiB 7.56MiB -0.0622397
('syslog_regex_logs2metric_ddmetrics', 'comparison') 326 7.26MiB 498.24KiB 6.05MiB 7.47MiB 7.71MiB 7.73MiB 7.78MiB 7.81MiB -1.02782
('syslog_loki', 'comparison') 365 7.18MiB 349.78KiB 6.29MiB 7.29MiB 7.49MiB 7.5MiB 7.56MiB 7.62MiB -1.26595
('http_to_http_acks', 'comparison') 362 5.4MiB 2.36MiB 731.75KiB 5.36MiB 8.82MiB 9.5MiB 10.02MiB 10.46MiB 0.105938
('http_to_http_acks', 'baseline') 362 5.1MiB 2.39MiB 0B 5.0MiB 8.01MiB 9.29MiB 9.73MiB 9.95MiB 0.0896893
('splunk_transforms_splunk3', 'comparison') 363 4.59MiB 1.56MiB 1.16MiB 4.35MiB 6.62MiB 7.51MiB 9.12MiB 9.8MiB 0.680675
('splunk_transforms_splunk3', 'baseline') 362 4.36MiB 1.55MiB 670.89KiB 4.26MiB 6.38MiB 7.12MiB 8.54MiB 10.25MiB 0.485751
('http_datadog_filter_blackhole', 'baseline') 361 2.2MiB 1.05MiB 186.49KiB 2.13MiB 3.61MiB 4.06MiB 4.81MiB 5.68MiB 0.449688
('http_datadog_filter_blackhole', 'comparison') 362 2.15MiB 1.01MiB 0B 2.05MiB 3.52MiB 3.88MiB 4.97MiB 5.35MiB 0.518779
('http_pipelines_no_grok_blackhole', 'comparison') 360 2.07MiB 1.1MiB 0B 1.88MiB 3.55MiB 4.12MiB 4.66MiB 5.53MiB 0.488737
('http_pipelines_no_grok_blackhole', 'baseline') 363 2.06MiB 1022.01KiB 31.98KiB 2.01MiB 3.32MiB 3.8MiB 4.56MiB 5.44MiB 0.366794
('http_pipelines_blackhole_acks', 'comparison') 320 893.13KiB 744.99KiB 0B 676.5KiB 1.92MiB 2.36MiB 2.98MiB 3.28MiB 0.984654
('http_pipelines_blackhole', 'comparison') 361 828.28KiB 703.2KiB 0B 736.04KiB 1.91MiB 2.06MiB 2.63MiB 2.72MiB 0.759162
('http_pipelines_blackhole', 'baseline') 360 822.0KiB 682.49KiB 0B 661.6KiB 1.83MiB 2.03MiB 2.52MiB 3.1MiB 0.850093
('http_pipelines_blackhole_acks', 'baseline') 365 817.14KiB 729.86KiB 0B 616.77KiB 1.8MiB 2.03MiB 2.95MiB 3.82MiB 1.19164

jdrouet pushed a commit that referenced this pull request Dec 27, 2021
jszwedko added a commit that referenced this pull request Dec 28, 2021
* Revert "Revert "chore: Update k8s-e2e to use new helm chart (#10521)" (#10566)"

This reverts commit 738e6c2.

* chore: update minikube version used in ci

Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>

* add action-tmate to debug ci

Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>

* removing tmate from ci

Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>

* wait for all when starting minikube

Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>

* remove cri-o tests for the time being

Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>

* wait longer in simple_checkpoint test

Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>

Co-authored-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: ci Anything related to Vector's CI environment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants