Feature: upgrade & downgrade tests #2388

cardil · 2020-01-15T19:09:45Z

Proposed Changes

Adding upgrade & downgrade tests with a prober that can be used in e2e tests and in operator e2e tests
Tests can run with serving component, but it's not required

For more verbose description see: test/upgrade/README.md in changed files.

Release Note

NONE

Prober will be used to it's own package to enable running it from within e2e tests and upgrade tests.

Implemented receiver deployment and removal with getting service node port, to later be able to connect and fetch results.

* ksvc ready * triggers ready * pod ready * ksvc scales to zero

knative-prow-robot · 2020-01-15T19:10:01Z

Hi @cardil. Thanks for your PR.

I'm waiting for a knative member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

matzew · 2020-01-15T20:01:54Z

/ok-to-test

chizhg · 2020-01-15T20:36:20Z

/uncc @yt3liu

cardil · 2020-05-18T19:17:36Z

/retest

cardil · 2020-05-19T10:26:04Z

Hey @vaikas @n3wscott @chizhg @mgencur. This PR is ready for final review. I think I've been able to put all the findings from WG meeting on April, 23.

What is interesting is that after I rebase the code to use master tests starts to report that the events are propagated twice after the upgrade. 😲

n3wscott · 2020-05-19T16:37:17Z

test/test_images/wathola-receiver/pod.yaml

+spec:
+  containers:
+    - name: wathola-receiver
+      image: ko://knative.dev/eventing/test/test_images/wathola-receiver


n3wscott · 2020-05-19T16:38:05Z

test/upgrade/README.md

@@ -0,0 +1,83 @@
+# Upgrade Tests


omg thanks for this documentation.

n3wscott · 2020-05-19T16:40:39Z

Thank you so much for the hard work, I think this is great and we can start with this and as we ramp up the knative-wide testing repo, this test will play a key role.

/lgtm

Ville to approve

vaikas

Thanks for doing this!!!

vaikas · 2020-05-19T16:41:06Z

test/upgrade/prober/config.toml

@@ -0,0 +1,6 @@
+# logLevel = 5 # DEBUG(5)
+[sender]
+address = 'http://default-broker.{{- .Namespace -}}.svc.cluster.local'


This will only work for soon to be removed ChannelBasedBroker. Can you file an issue to actually fetch the URL from the Broker.Status instead of hardcoding it?

vaikas · 2020-05-19T16:43:40Z

test/upgrade/README.md

+          |        |       |         | |
+          +--------+       +---------+ |
+           (default)        +----------+
+```


This documentation is great, I'd like to see a section on what the output is, where to look for debug hints, look like you dump a bunch of logs, etc. For example you report that there are double events, my first knee jerk reaction is, are there for example double-subscriptions being created somehow. Not quite sure how I'd go about figuring it out. Please create an issue to follow up so we don't have to hold / rebase, etc.

In the e2e tests we dump the objects / k8s events on test failures so something like that would be nice. But again, fine as a followup.

Now, we are not failing even that we found errors. If we switch https://github.com/cardil/eventing/blob/feature/upgrade-tests/test/upgrade/probe_test.go#L55 then we will have default fail dump after this test. Maybe that is sufficient?

That line L55 gives me 404. I guess it's this: https://github.com/knative/eventing/pull/2388/files#diff-38e78729550b76fecdd458bc3146c1c3R55 ?

Yup. I've removed the source branch, sorry. But, now it's merged it's on master: https://github.com/knative/eventing/blob/master/test/upgrade/probe_test.go#L55

vaikas · 2020-05-19T16:47:00Z

/approve

knative-prow-robot · 2020-05-19T16:47:13Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cardil, vaikas

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [vaikas]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

vaikas · 2020-05-19T17:51:09Z

Node exporter image pulloff failure? Odd... Other two came up just fine...

Waiting until all pods in namespace knative-monitoring are up......................................................................................................................................................
ERROR: timeout waiting for pods to come up
elasticsearch-logging-0               1/1   Running            0     5m21s
elasticsearch-logging-1               1/1   Running            0     3m36s
fluentd-ds-2hvrw                      1/1   Running            0     5m21s
fluentd-ds-7m2pj                      1/1   Running            0     5m21s
fluentd-ds-jlf82                      1/1   Running            0     5m21s
grafana-79b645d9c7-zttnb              1/1   Running            0     5m20s
kibana-logging-b5d75f556-df6n6        1/1   Running            0     5m21s
kube-state-metrics-5cb5c6986b-fx5md   1/1   Running            0     5m21s
node-exporter-9k29p                   2/2   Running            0     5m20s
node-exporter-ps75k                   1/2   ImagePullBackOff   0     5m20s
node-exporter-x7tsj                   2/2   Running            0     5m20s
prometheus-system-0                   1/1   Running            0     5m19s
prometheus-system-1                   1/1   Running            0     5m19s

vaikas · 2020-05-19T17:51:18Z

/test pull-knative-eventing-integration-tests

vaikas · 2020-05-19T22:04:18Z

/test pull-knative-eventing-integration-tests

grantr · 2020-05-19T22:45:28Z

kube-state-metrics-b6bcff8f4-f2cxj 0/1 ImagePullBackOff 0 5m18s

Might be a quay outage issue?

cardil · 2020-05-20T09:53:25Z

Quay is now up. Retesting.

/test pull-knative-eventing-integration-tests

cardil added 16 commits January 7, 2020 20:21

Change scripts to enable upgrade tests

0df4543

Upgrade tests overall structure

485e839

Adding Istio & Serving to upgrade tests

816b3ce

Delete non required Istio files

e13bf3f

Move prober to separate package

acb8dc6

Prober will be used to it's own package to enable running it from within e2e tests and upgrade tests.

Working deploy & remove of configuration

b45ff25

Probe receiver deploy & remove

882119e

Implemented receiver deployment and removal with getting service node port, to later be able to connect and fetch results.

Probe forwarder deploy & removal

2d925ae

Switch to zap.SugarLogger for better messages visibility

e91afc7

Inteligent waiting for all conditions

2a66e45

* ksvc ready * triggers ready * pod ready * ksvc scales to zero

Fix interval configuration for Sender

614cd40

Deploy & removal of sender pod

dfbf3b6

Update ready message a bit

b046fd4

Fetch receiver report via HTTP Get

d9f02c3

Don't deploy IMC from HEAD before each test

a16e983

Describe an upgrade tests

99909a9

knative-prow-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. area/test-and-release Test infrastructure, tests or release needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 15, 2020

mattmoor reviewed Jan 15, 2020

View reviewed changes

knative-prow-robot requested review from chizhg, lionelvillard and yt3liu January 15, 2020 19:10

cardil changed the title ~~Feature/upgrade tests~~ Feature: upgrade & downgrade tests Jan 15, 2020

googlebot added the cla: yes Indicates the PR's author has signed the CLA. label Jan 15, 2020

knative-prow-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 15, 2020

cardil added 2 commits May 18, 2020 18:15

Migrate to cloudevents v2

fe54820

Move prober + wathola tool to upgrade directory

823bdf6

cardil force-pushed the feature/upgrade-tests branch from 23e6067 to 823bdf6 Compare May 18, 2020 16:56

Use of latest version of TOML parser

4739de5

cardil force-pushed the feature/upgrade-tests branch from 1a75336 to 4739de5 Compare May 18, 2020 21:54

n3wscott reviewed May 19, 2020

View reviewed changes

test/upgrade/README.md

@@ -0,0 +1,83 @@

# Upgrade Tests

Copy link

Contributor

n3wscott May 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

omg thanks for this documentation.

knative-prow-robot assigned n3wscott May 19, 2020

knative-prow-robot added the lgtm Indicates that a PR is ready to be merged. label May 19, 2020

vaikas reviewed May 19, 2020

View reviewed changes

knative-prow-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 19, 2020

knative-prow-robot merged commit f1a586b into knative:master May 20, 2020

cardil deleted the feature/upgrade-tests branch May 20, 2020 11:05

cardil mentioned this pull request Jun 3, 2020

Separate Knative Eventing upgrade tests from regular e2e tests knative/test-infra#2152

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: upgrade & downgrade tests #2388

Feature: upgrade & downgrade tests #2388

cardil commented Jan 15, 2020 •

edited

Loading

knative-prow-robot commented Jan 15, 2020

matzew commented Jan 15, 2020

chizhg commented Jan 15, 2020

cardil commented May 18, 2020

cardil commented May 19, 2020

n3wscott May 19, 2020

n3wscott May 19, 2020

n3wscott commented May 19, 2020

vaikas left a comment

vaikas May 19, 2020

cardil May 20, 2020

vaikas May 19, 2020

vaikas May 19, 2020

cardil May 20, 2020

mgencur May 20, 2020 •

edited

Loading

cardil May 20, 2020

vaikas commented May 19, 2020

knative-prow-robot commented May 19, 2020

vaikas commented May 19, 2020

vaikas commented May 19, 2020

vaikas commented May 19, 2020

grantr commented May 19, 2020

cardil commented May 20, 2020

Feature: upgrade & downgrade tests #2388

Feature: upgrade & downgrade tests #2388

Conversation

cardil commented Jan 15, 2020 • edited Loading

Proposed Changes

knative-prow-robot commented Jan 15, 2020

matzew commented Jan 15, 2020

chizhg commented Jan 15, 2020

cardil commented May 18, 2020

cardil commented May 19, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

n3wscott commented May 19, 2020

vaikas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mgencur May 20, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vaikas commented May 19, 2020

knative-prow-robot commented May 19, 2020

vaikas commented May 19, 2020

vaikas commented May 19, 2020

vaikas commented May 19, 2020

grantr commented May 19, 2020

cardil commented May 20, 2020

cardil commented Jan 15, 2020 •

edited

Loading

mgencur May 20, 2020 •

edited

Loading