[3.4] allow downgrade from 3.5 #17330

siyuanfoundation · 2024-01-27T00:03:10Z

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

Part of #15878 (comment)
Tested downgrade from 3.5 cluster locally:

Start 3.5 cluster

bin/etcd --name infra1 --listen-client-urls http://127.0.0.1:2379 --advertise-client-urls http://127.0.0.1:2379 --listen-peer-urls http://127.0.0.1:12380 --initial-advertise-peer-urls http://127.0.0.1:12380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --enable-pprof --logger=zap --log-outputs=stderr

bin/etcd --name infra2 --listen-client-urls http://127.0.0.1:22379 --advertise-client-urls http://127.0.0.1:22379 --listen-peer-urls http://127.0.0.1:22380 --initial-advertise-peer-urls http://127.0.0.1:22380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --enable-pprof --logger=zap --log-outputs=stderr

bin/etcd --name infra3 --listen-client-urls http://127.0.0.1:32379 --advertise-client-urls http://127.0.0.1:32379 --listen-peer-urls http://127.0.0.1:32380 --initial-advertise-peer-urls http://127.0.0.1:32380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --enable-pprof --logger=zap --log-outputs=stderr

start putting data

go run tools/benchmark/main.go put --total=10000000 --key-size=64 --val-size=512`

bin/etcdctl --endpoints=http://127.0.0.1:2379,http://127.0.0.1:22379,http://127.0.0.1:32379 endpoint status -w table
bin/etcdctl auth status

kill infra3
restart infra3 with 3.4 binary

../etcd-3.4/bin/etcd --name infra3 --listen-client-urls http://127.0.0.1:32379 --advertise-client-urls http://127.0.0.1:32379 --listen-peer-urls http://127.0.0.1:32380 --initial-advertise-peer-urls http://127.0.0.1:32380 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state existing --enable-pprof --logger=zap --log-outputs=stderr --data-dir infra3.etcd --accept-next-storage-version

bin/etcdctl auth status

kill and restart infra2 with 3.4 binary

../etcd-3.4/bin/etcd --name infra2 --listen-client-urls http://127.0.0.1:22379 --advertise-client-urls http://127.0.0.1:22379 --listen-peer-urls http://127.0.0.1:22380 --initial-advertise-peer-urls http://127.0.0.1:22380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state existing --enable-pprof --logger=zap --log-outputs=stderr --data-dir infra2.etcd --accept-next-storage-version

kill and restart infra1 with 3.4 binary

../etcd-3.4/bin/etcd --name infra1 --listen-client-urls http://127.0.0.1:2379 --advertise-client-urls http://127.0.0.1:2379 --listen-peer-urls http://127.0.0.1:12380 --initial-advertise-peer-urls http://127.0.0.1:12380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state existing --enable-pprof --logger=zap --log-outputs=stderr --data-dir infra1.etcd --accept-next-storage-version

siyuanfoundation · 2024-01-29T16:20:45Z

cc @serathius @ahrtr

serathius · 2024-02-02T13:26:29Z

Not sure how this PR refers to dev work described in https://docs.google.com/document/d/1siSo-tA_xLRo5li5wK8KqWsOiq34PIgtmuSDcyxxwsY/edit?usp=sharing

In document you described that downgrade will gated behind "--allow-downgrade" flag and will include implementation of AuthStatusRequest handling.

mvcc/kvstore.go

etcdserver/api/membership/cluster.go

siyuanfoundation · 2024-02-02T23:04:33Z

Not sure how this PR refers to dev work described in https://docs.google.com/document/d/1siSo-tA_xLRo5li5wK8KqWsOiq34PIgtmuSDcyxxwsY/edit?usp=sharing

In document you described that downgrade will gated behind "--allow-downgrade" flag and will include implementation of AuthStatusRequest handling.

With the etcdutil migrate and online schema check, "--allow-downgrade" flag is kind of redundant.
But I see your point about snapshot.

Added "--allow-downgrade" flag and implementation of AuthStatusRequest handling.

siyuanfoundation · 2024-02-02T23:22:01Z

/retest

siyuanfoundation · 2024-02-06T19:31:37Z

/retest

siyuanfoundation · 2024-02-06T21:45:34Z

cc @lavacat

ahrtr · 2024-02-14T09:51:33Z

mvcc/downgrade.go

+// UnsafeDetectSchemaVersion returns version of storage schema. Returned value depends on etcd version that created the backend. For
+// * v3.6 and newer will return storage version.
+// * v3.5 will return it's version if it includes any storage fields added in v3.5.
+// * v3.4 will return it's version if it doesn't include any storage fields added in v3.5.


For 3.3, it will also return v3_4. So it will not impact the case of upgrading 3.3 to 3.4. We need to clarify this at least in comment and verify it in test.

Added a comment. As for test, I could not find any difference in the data schemas of 3.3 and 3.4, not sure about what to test.

etcdserver/api/membership/cluster.go

ahrtr · 2024-02-14T19:12:19Z

Overall looks good. We need to add a couple of e2e test a well,

The normal upgrade from 3.3 to 3.4. We need to ensure the normal upgrading from 3.3 to 3.4 isn't impacted by this PR.
Downgrade from 3.5 to 3.4
- next-cluster-version-compatible = false
- next-cluster-version-compatible = true

It can be addressed in a followup, but please manually verify the cases.

siyuanfoundation · 2024-02-15T01:42:41Z

Tested 3.3 -> 3.4 -> 3.5 ->3.3 sequence manually with:

start 3.3 cluster

../etcd-3.3/etcd --name infra1 --listen-client-urls http://127.0.0.1:2379 --advertise-client-urls http://127.0.0.1:2379 --listen-peer-urls http://127.0.0.1:12380 --initial-advertise-peer-urls http://127.0.0.1:12380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new

../etcd-3.3/etcd --name infra2 --listen-client-urls http://127.0.0.1:22379 --advertise-client-urls http://127.0.0.1:22379 --listen-peer-urls http://127.0.0.1:22380 --initial-advertise-peer-urls http://127.0.0.1:22380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new

../etcd-3.3/etcd --name infra3 --listen-client-urls http://127.0.0.1:32379 --advertise-client-urls http://127.0.0.1:32379 --listen-peer-urls http://127.0.0.1:32380 --initial-advertise-peer-urls http://127.0.0.1:32380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new

verify status

go run tools/benchmark/main.go put --total=10000000 --key-size=64 --val-size=512`

bin/etcdctl --endpoints=http://127.0.0.1:2379,http://127.0.0.1:22379,http://127.0.0.1:32379 endpoint status -w table

upgrade server one by one to 3.4

../etcd-3.4/bin/etcd --name infra3 --listen-client-urls http://127.0.0.1:32379 --advertise-client-urls http://127.0.0.1:32379 --listen-peer-urls http://127.0.0.1:32380 --initial-advertise-peer-urls http://127.0.0.1:32380 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state existing --data-dir infra3.etcd 

../etcd-3.4/bin/etcd --name infra2 --listen-client-urls http://127.0.0.1:22379 --advertise-client-urls http://127.0.0.1:22379 --listen-peer-urls http://127.0.0.1:22380 --initial-advertise-peer-urls http://127.0.0.1:22380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state existing --data-dir infra2.etcd

../etcd-3.4/bin/etcd --name infra1 --listen-client-urls http://127.0.0.1:2379 --advertise-client-urls http://127.0.0.1:2379 --listen-peer-urls http://127.0.0.1:12380 --initial-advertise-peer-urls http://127.0.0.1:12380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state existing --data-dir infra1.etcd

verify status

bin/etcdctl --endpoints=http://127.0.0.1:2379,http://127.0.0.1:22379,http://127.0.0.1:32379 endpoint status -w table

upgrade server one by one to 3.5
verify status

bin/etcdctl --endpoints=http://127.0.0.1:2379,http://127.0.0.1:22379,http://127.0.0.1:32379 endpoint status -w table
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|        ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|  http://127.0.0.1:2379 | 8211f1d0f64f3269 |  3.5.12 |  377 MB |     false |      false |        17 |     454917 |             454917 |        |
| http://127.0.0.1:22379 | 91bc3c398fb3c146 |  3.5.12 |  377 MB |     false |      false |        17 |     454917 |             454917 |        |
| http://127.0.0.1:32379 | fd422379fda50e48 |  3.5.12 |  377 MB |      true |      false |        17 |     454917 |             454917 |        |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

bin/etcdctl auth status
>> Authentication Status: false
>> AuthRevision: 1

downgrade to 3.4 without the --next-cluster-version-compatible flag

../etcd-3.4/bin/etcd --name infra3 --listen-client-urls http://127.0.0.1:32379 --advertise-client-urls http://127.0.0.1:32379 --listen-peer-urls http://127.0.0.1:32380 --initial-advertise-peer-urls http://127.0.0.1:32380 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state existing --data-dir infra3.etcd

# fails with
>> etcdserver/membership: invalid downgrade; (current version: 3.4.29 is lower than determined cluster version: 3.5).

downgrade to 3.4 the --next-cluster-version-compatible flag

../etcd-3.4/bin/etcd --name infra3 --listen-client-urls http://127.0.0.1:32379 --advertise-client-urls http://127.0.0.1:32379 --listen-peer-urls http://127.0.0.1:32380 --initial-advertise-peer-urls http://127.0.0.1:32380 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state existing --data-dir infra3.etcd --next-cluster-version-compatible

verify status
finish downgrading the other 2 servers, with enough writes when some server is off to trigger snapshot restore.
verify status

bin/etcdctl --endpoints=http://127.0.0.1:2379,http://127.0.0.1:22379,http://127.0.0.1:32379 endpoint status -w table
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|        ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|  http://127.0.0.1:2379 | 8211f1d0f64f3269 |  3.4.29 |  527 MB |     false |      false |        20 |     635804 |             635804 |        |
| http://127.0.0.1:22379 | 91bc3c398fb3c146 |  3.4.29 |  527 MB |     false |      false |        20 |     635804 |             635804 |        |
| http://127.0.0.1:32379 | fd422379fda50e48 |  3.4.29 |  528 MB |      true |      false |        20 |     635804 |             635804 |        |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

mvcc/downgrade.go

ahrtr · 2024-02-15T09:14:37Z

cc @fuweid to take a look. This PR should be also important to AKS.

fuweid

The change looks good. But it seems that this patch needs go mod tidy.

ahrtr · 2024-02-19T10:01:02Z

But it seems that this patch needs go mod tidy.

This PR doesn't change anything for go.mod and go.sum files. I do not see any issue when running "go mod tidy" command. Please ensure you use exactly the same go version as https://github.com/etcd-io/etcd/blob/release-3.4/.go-version#L1

fuweid · 2024-02-19T10:14:50Z

This PR doesn't change anything for go.mod and go.sum files. I do not see any issue when running "go mod tidy" command.

If you pull this pull request in your local, you will see that. This pull request doesn't rebase so it shows diff.
It's fine. It's not blocking comment.

ahrtr · 2024-02-19T10:21:57Z

If you pull this pull request in your local, you will see that. This pull request doesn't rebase so it shows diff.

Yes, it's true. Curious why the workflow did not fail.

Anyway, please rebase this PR. @siyuanfoundation

Signed-off-by: Siyuan Zhang <sizhang@google.com>

siyuanfoundation · 2024-02-20T17:26:41Z

rebased. the workflow still passes.

siyuanfoundation force-pushed the 3.4-downgrade branch 2 times, most recently from f5c17ba to ebbda41 Compare January 27, 2024 00:23

serathius reviewed Feb 2, 2024

View reviewed changes

mvcc/kvstore.go Outdated Show resolved Hide resolved

serathius reviewed Feb 2, 2024

View reviewed changes

etcdserver/api/membership/cluster.go Outdated Show resolved Hide resolved

siyuanfoundation marked this pull request as draft February 2, 2024 20:06

k8s-ci-robot added the do-not-merge/work-in-progress label Feb 2, 2024

siyuanfoundation force-pushed the 3.4-downgrade branch 2 times, most recently from a562708 to 89201d9 Compare February 2, 2024 23:02

siyuanfoundation marked this pull request as ready for review February 3, 2024 04:35

k8s-ci-robot removed the do-not-merge/work-in-progress label Feb 3, 2024

siyuanfoundation requested a review from serathius February 3, 2024 04:35

siyuanfoundation force-pushed the 3.4-downgrade branch from 89201d9 to f8234be Compare February 5, 2024 17:31

siyuanfoundation marked this pull request as draft February 5, 2024 17:31

k8s-ci-robot added the do-not-merge/work-in-progress label Feb 5, 2024

siyuanfoundation force-pushed the 3.4-downgrade branch 4 times, most recently from 1312bf5 to ccff3d7 Compare February 6, 2024 18:31

siyuanfoundation force-pushed the 3.4-downgrade branch from ccff3d7 to ce46503 Compare February 6, 2024 21:21

siyuanfoundation marked this pull request as ready for review February 6, 2024 21:44

k8s-ci-robot removed the do-not-merge/work-in-progress label Feb 6, 2024

siyuanfoundation force-pushed the 3.4-downgrade branch 3 times, most recently from 04fa384 to 325ec09 Compare February 6, 2024 22:15

ahrtr reviewed Feb 14, 2024

View reviewed changes

siyuanfoundation force-pushed the 3.4-downgrade branch from a18a80c to ccf8e33 Compare February 14, 2024 17:21

siyuanfoundation force-pushed the 3.4-downgrade branch from ccf8e33 to 1b3688f Compare February 15, 2024 01:04

siyuanfoundation force-pushed the 3.4-downgrade branch from 1b3688f to 4b197f6 Compare February 15, 2024 03:15

siyuanfoundation requested a review from ahrtr February 15, 2024 03:18

serathius reviewed Feb 15, 2024

View reviewed changes

mvcc/downgrade.go Outdated Show resolved Hide resolved

serathius reviewed Feb 15, 2024

View reviewed changes

mvcc/downgrade.go Outdated Show resolved Hide resolved

serathius approved these changes Feb 15, 2024

View reviewed changes

siyuanfoundation force-pushed the 3.4-downgrade branch from b92975c to a6a77dd Compare February 15, 2024 17:27

fuweid self-requested a review February 16, 2024 14:29

fuweid reviewed Feb 17, 2024

View reviewed changes

Bogdan Kanivets and others added 4 commits February 20, 2024 08:55

add flag to allow downgrade from 3.5

e7da7eb

Signed-off-by: Siyuan Zhang <sizhang@google.com>

Add function to migrate 3.5 data online.

c65c1ea

Signed-off-by: Siyuan Zhang <sizhang@google.com>

Add handling of AuthStatusRequest.

83f97c1

Signed-off-by: Siyuan Zhang <sizhang@google.com>

Add schema verification when closing etcd.

2caf0f0

Signed-off-by: Siyuan Zhang <sizhang@google.com>

siyuanfoundation force-pushed the 3.4-downgrade branch from a6a77dd to 2caf0f0 Compare February 20, 2024 16:56

ahrtr approved these changes Feb 20, 2024

View reviewed changes

ahrtr merged commit c057f47 into etcd-io:release-3.4 Feb 20, 2024
12 checks passed

siyuanfoundation mentioned this pull request Feb 29, 2024

Downgrade support from 3.5 to 3.4 #15878

Open

This was referenced Mar 28, 2024

add 3.4 changelog of adding downgrade support. #17664

Merged

[3.4] Allow new server to join higher cluster version if NextClusterVersionCompatible is true #17665

Merged

siyuanfoundation mentioned this pull request Apr 8, 2024

Add e2e tests to verify snapshot data compatibility in a mixed version cluster #17752

Open

3 tasks

lhy1024 mentioned this pull request Aug 8, 2024

*: upgrade etcd to v3.5.15 tikv/pd#8441

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[3.4] allow downgrade from 3.5 #17330

[3.4] allow downgrade from 3.5 #17330

siyuanfoundation commented Jan 27, 2024 •

edited

Loading

siyuanfoundation commented Jan 29, 2024

serathius commented Feb 2, 2024

siyuanfoundation commented Feb 2, 2024

siyuanfoundation commented Feb 2, 2024

siyuanfoundation commented Feb 6, 2024

siyuanfoundation commented Feb 6, 2024

ahrtr Feb 14, 2024

siyuanfoundation Feb 14, 2024

ahrtr commented Feb 14, 2024

siyuanfoundation commented Feb 15, 2024 •

edited

Loading

ahrtr commented Feb 15, 2024

fuweid left a comment

ahrtr commented Feb 19, 2024

fuweid commented Feb 19, 2024

ahrtr commented Feb 19, 2024

siyuanfoundation commented Feb 20, 2024

[3.4] allow downgrade from 3.5 #17330

[3.4] allow downgrade from 3.5 #17330

Conversation

siyuanfoundation commented Jan 27, 2024 • edited Loading

siyuanfoundation commented Jan 29, 2024

serathius commented Feb 2, 2024

siyuanfoundation commented Feb 2, 2024

siyuanfoundation commented Feb 2, 2024

siyuanfoundation commented Feb 6, 2024

siyuanfoundation commented Feb 6, 2024

ahrtr Feb 14, 2024

Choose a reason for hiding this comment

siyuanfoundation Feb 14, 2024

Choose a reason for hiding this comment

ahrtr commented Feb 14, 2024

siyuanfoundation commented Feb 15, 2024 • edited Loading

ahrtr commented Feb 15, 2024

fuweid left a comment

Choose a reason for hiding this comment

ahrtr commented Feb 19, 2024

fuweid commented Feb 19, 2024

ahrtr commented Feb 19, 2024

siyuanfoundation commented Feb 20, 2024

siyuanfoundation commented Jan 27, 2024 •

edited

Loading

siyuanfoundation commented Feb 15, 2024 •

edited

Loading