e2e: only expect cluster's major version is > 3 in release upgrade test #11266

YoyinZyc · 2019-10-16T20:56:06Z

Fix e2e test TestReleaseUpgrade.
The test will skip only when the cluster version is <3.0.

=== RUN TestReleaseUpgrade
--- SKIP: TestReleaseUpgrade (9.81s)
etcd_release_upgrade_test.go:59: #0: v3 is not ready yet (read /dev/ptmx: input/output error (expected ""etcdcluster":"3.5", got ["{"etcdserver":"3.4.0","etcdcluster":"3.4.0"}"]))
etcd_release_upgrade_test.go:59: #1: v3 is not ready yet (read /dev/ptmx: input/output error (expected ""etcdcluster":"3.5", got ["{"etcdserver":"3.4.0","etcdcluster":"3.4.0"}"]))
etcd_release_upgrade_test.go:59: #2: v3 is not ready yet (read /dev/ptmx: input/output error (expected ""etcdcluster":"3.5", got ["{"etcdserver":"3.4.0","etcdcluster":"3.4.0"}"]))
etcd_release_upgrade_test.go:59: #3: v3 is not ready yet (read /dev/ptmx: input/output error (expected ""etcdcluster":"3.5", got ["{"etcdserver":"3.4.0","etcdcluster":"3.4.0"}"]))
etcd_release_upgrade_test.go:59: #4: v3 is not ready yet (read /dev/ptmx: input/output error (expected ""etcdcluster":"3.5", got ["{"etcdserver":"3.4.0","etcdcluster":"3.4.0"}"]))
etcd_release_upgrade_test.go:59: #5: v3 is not ready yet (read /dev/ptmx: input/output error (expected ""etcdcluster":"3.5", got ["{"etcdserver":"3.4.0","etcdcluster":"3.4.0"}"]))
etcd_release_upgrade_test.go:59: #6: v3 is not ready yet (read /dev/ptmx: input/output error (expected ""etcdcluster":"3.5", got ["{"etcdserver":"3.4.0","etcdcluster":"3.4.0"}"]))
etcd_release_upgrade_test.go:66: cannot pull version (read /dev/ptmx: input/output error (expected ""etcdcluster":"3.5", got ["{"etcdserver":"3.4.0","etcdcluster":"3.4.0"}"]))
Test ignored.

YoyinZyc · 2019-10-16T21:41:40Z

CI test failed due to #11264

YoyinZyc · 2019-10-16T21:43:08Z

cc @gyuho @wenjiaswe @jingyih

jingyih · 2019-10-16T22:55:28Z

@gyuho Could you help take a look?

gyuho · 2019-10-18T00:13:38Z

@YoyinZyc Can you fix

=== RUN   TestReleaseUpgrade
panic: test timed out after 30m0s

?

YoyinZyc · 2019-10-18T00:18:32Z

Yuchen Zhou Can you fix

=== RUN   TestReleaseUpgrade
panic: test timed out after 30m0s

?

This failure is due to the upgrade issue #11264.

jingyih · 2019-10-18T02:28:27Z

To make the test pass, we need to merge #11275, release 3.4.2, and update MANUAL_VER to 3.4.2 here:

etcd/tests/semaphore.test.bash

Line 11 in 9c48dfa

    
           sudo HOST_TMP_DIR=/tmp TEST_OPTS="PASSES='build release e2e' MANUAL_VER=v3.4.0" make docker-test

Is my understanding correct?

YoyinZyc · 2019-10-18T16:55:00Z

To make the test pass, we need to merge #11275, release 3.4.2, and update MANUAL_VER to 3.4.2 here:

etcd/tests/semaphore.test.bash

Line 11 in 9c48dfa

sudo HOST_TMP_DIR=/tmp TEST_OPTS="PASSES='build release e2e' MANUAL_VER=v3.4.0" make docker-test

Is my understanding correct?

Yep. The test will pass without setting MANUAL_VER to 3.4.2. But since we have already released 3.4.2, we should update it. It seems that there are couple of things we need to update manually when we want to release or upgrade to new version. Do we have a checklist or test for that?

gyuho · 2019-10-18T17:26:34Z

@YoyinZyc Then, let's finish up the fix in master branch. Then, I can release a new patch release, and fix the test script.

wenjiaswe · 2019-10-18T17:44:36Z

tests/e2e/etcd_release_upgrade_test.go

 	}
-	if err := cURLGet(cx.epc, cURLReq{endpoint: "/metrics", expected: fmt.Sprintf(`etcd_cluster_version{cluster_version="%s"} 1`, ver), metricsURLScheme: cx.cfg.metricsURLScheme}); err != nil {


Prior to your change, the test was using the /version endpoint to check the version before upgrade then use the /metrics endpoint to check the version afterward. I think it's better with your change that we check the version with the consistent way. But I am just curious, @gyuho is there specific reason that we previous check the /metrics endpoint afterward? Just want to make sure we are not missing testing anything.

@wenjiaswe I think either /version or /metrics is fine, since they are all set by the same constant (version.Version). Since this is just upgrade test, using /version makes more sense.

…sts.

YoyinZyc · 2019-11-25T21:38:13Z

Can we merge the test fix since we have already released 3.4.3 ?

codecov-io · 2019-11-25T21:42:47Z

Codecov Report

Merging #11266 into master will increase coverage by 0.66%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #11266      +/-   ##
==========================================
+ Coverage   64.11%   64.78%   +0.66%     
==========================================
  Files         403      403              
  Lines       37973    37973              
==========================================
+ Hits        24348    24599     +251     
+ Misses      11995    11742     -253     
- Partials     1630     1632       +2

Impacted Files	Coverage Δ
auth/jwt.go	`56.19% <0%> (-12.39%)`	⬇️
pkg/proxy/server.go	`60.2% <0%> (-1.02%)`	⬇️
etcdserver/v3_server.go	`73.29% <0%> (-0.43%)`	⬇️
etcdserver/server.go	`68.4% <0%> (+0.26%)`	⬆️
clientv3/watch.go	`91.57% <0%> (+0.63%)`	⬆️
etcdserver/cluster_util.go	`57.31% <0%> (+0.79%)`	⬆️
clientv3/retry_interceptor.go	`69.3% <0%> (+0.99%)`	⬆️
etcdserver/api/rafthttp/peer.go	`78.77% <0%> (+1.11%)`	⬆️
etcdserver/api/v3rpc/watch.go	`79.73% <0%> (+1.3%)`	⬆️
clientv3/leasing/kv.go	`90.36% <0%> (+1.32%)`	⬆️
... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ec52217...cab4cac. Read the comment docs.

jingyih

LGTM

YoyinZyc force-pushed the upgrade-downgrade-test branch from 5544c02 to a44eb1c Compare October 18, 2019 17:29

wenjiaswe reviewed Oct 18, 2019

View reviewed changes

YoyinZyc added 2 commits November 25, 2019 10:57

e2e: only expect cluster's major version is > 3 in release upgrade te…

5a11023

…sts.

tests: change MANUAL_VER to 3.4.3

cab4cac

YoyinZyc force-pushed the upgrade-downgrade-test branch from a44eb1c to cab4cac Compare November 25, 2019 21:08

jingyih approved these changes Nov 25, 2019

View reviewed changes

jingyih merged commit 44fc9e3 into etcd-io:master Nov 25, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

e2e: only expect cluster's major version is > 3 in release upgrade test #11266

e2e: only expect cluster's major version is > 3 in release upgrade test #11266

YoyinZyc commented Oct 16, 2019

YoyinZyc commented Oct 16, 2019

YoyinZyc commented Oct 16, 2019

jingyih commented Oct 16, 2019

gyuho commented Oct 18, 2019

YoyinZyc commented Oct 18, 2019

jingyih commented Oct 18, 2019

YoyinZyc commented Oct 18, 2019

gyuho commented Oct 18, 2019

wenjiaswe Oct 18, 2019

gyuho Oct 18, 2019

YoyinZyc commented Nov 25, 2019

codecov-io commented Nov 25, 2019 •

edited

Loading

jingyih left a comment

		}
		if err := cURLGet(cx.epc, cURLReq{endpoint: "/metrics", expected: fmt.Sprintf(`etcd_cluster_version{cluster_version="%s"} 1`, ver), metricsURLScheme: cx.cfg.metricsURLScheme}); err != nil {

e2e: only expect cluster's major version is > 3 in release upgrade test #11266

e2e: only expect cluster's major version is > 3 in release upgrade test #11266

Conversation

YoyinZyc commented Oct 16, 2019

YoyinZyc commented Oct 16, 2019

YoyinZyc commented Oct 16, 2019

jingyih commented Oct 16, 2019

gyuho commented Oct 18, 2019

YoyinZyc commented Oct 18, 2019

jingyih commented Oct 18, 2019

YoyinZyc commented Oct 18, 2019

gyuho commented Oct 18, 2019

wenjiaswe Oct 18, 2019

Choose a reason for hiding this comment

gyuho Oct 18, 2019

Choose a reason for hiding this comment

YoyinZyc commented Nov 25, 2019

codecov-io commented Nov 25, 2019 • edited Loading

Codecov Report

jingyih left a comment

Choose a reason for hiding this comment

codecov-io commented Nov 25, 2019 •

edited

Loading