v4.0.16 upgrade to master(1130) fail

[seiya-annie]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. k8s env, install v4.0.16 cluster
2. do br backup/restore
3. upgrade to master

2. What did you expect to see? (Required)

upgrade successfully

3. What did you see instead (Required)

tidb fail with:
tingli@tings-MacBook-Pro sqlfeature % kubectl -n brie-tidbmaster-comp-tps-1323534-1-906 logs upstream-tidb-0
start tidb-server ...
/tidb-server --store=tikv --advertise-address=upstream-tidb-0.upstream-tidb-peer.brie-tidbmaster-comp-tps-1323534-1-906.svc --host=0.0.0.0 --path=upstream-pd:2379 --config=/etc/tidb/tidb.toml

goroutine 1 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
/usr/local/go/src/runtime/debug/stack.go:16 +0x19
github.com/pingcap/tidb/session.mustExecute({0x4f96ea0, 0xc000b3a000}, {0x48bf967, 0x88}, {0xc000cf80c0, 0x4, 0x4})
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/bootstrap.go:2309 +0x12c
github.com/pingcap/tidb/session.insertBuiltinBindInfoRow(...)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/bootstrap.go:1574
github.com/pingcap/tidb/session.upgradeToVer57({0x4f96ea0, 0xc000b3a000}, 0x48726c8?)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/bootstrap.go:1565 +0xb2
github.com/pingcap/tidb/session.upgrade({0x4f96ea0?, 0xc000b3a000?})
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/bootstrap.go:890 +0x582
github.com/pingcap/tidb/session.runInBootstrapSession({0x4f4ad90, 0xc000dc2690}, 0x49135d0)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:3088 +0x20d
github.com/pingcap/tidb/session.BootstrapSession({0x4f4ad90, 0xc000dc2690})
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2943 +0x259
main.createStoreAndDomain()
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:310 +0x1c5
main.main()
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:211 +0x2ca
tingli@tings-MacBook-Pro sqlfeature %

4. What is your TiDB version? (Required)

master

[Defined2014]

seems caused by #39156, @fzzf678 PTAL

goroutine 1 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
        /usr/local/go/src/runtime/debug/stack.go:16 +0x19
github.com/pingcap/tidb/session.mustExecute({0x4fa8180, 0xc000d84500}, {0x48cfdfd, 0x88}, {0xc000e62380, 0x4, 0x4})
        /home/jason/tidb/session/bootstrap.go:2309 +0x12c
github.com/pingcap/tidb/session.insertBuiltinBindInfoRow(...)
        /home/jason/tidb/session/bootstrap.go:1574
github.com/pingcap/tidb/session.upgradeToVer57({0x4fa8180, 0xc000d84500}, 0x4882b5e?)
        /home/jason/tidb/session/bootstrap.go:1565 +0xb2
github.com/pingcap/tidb/session.upgrade({0x4fa8180?, 0xc000d84500?})
        /home/jason/tidb/session/bootstrap.go:890 +0x582
github.com/pingcap/tidb/session.runInBootstrapSession({0x4f5c070, 0xc000d191d0}, 0x4923a88)
        /home/jason/tidb/session/session.go:3088 +0x20d
github.com/pingcap/tidb/session.BootstrapSession({0x4f5c070, 0xc000d191d0})
        /home/jason/tidb/session/session.go:2943 +0x259
main.createStoreAndDomain()
        /home/jason/tidb/tidb-server/main.go:310 +0x1c5
main.main()
        /home/jason/tidb/tidb-server/main.go:211 +0x2ca
[2022/12/01 12:14:26.597 +08:00] [FATAL] [bootstrap.go:2310] ["mustExecute error"] [error="[planner:1136]Column count doesn't match value count at row 1"] [stack="github.com/pingcap/tidb/session.mustExecute\n\t/home/jason/tidb/session/bootstrap.go:2310\ngithub.com/pingcap/tidb/session.insertBuiltinBindInfoRow\n\t/home/jason/tidb/session/bootstrap.go:1574\ngithub.com/pingcap/tidb/session.upgradeToVer57\n\t/home/jason/tidb/session/bootstrap.go:1565\ngithub.com/pingcap/tidb/session.upgrade\n\t/home/jason/tidb/session/bootstrap.go:890\ngithub.com/pingcap/tidb/session.runInBootstrapSession\n\t/home/jason/tidb/session/session.go:3088\ngithub.com/pingcap/tidb/session.BootstrapSession\n\t/home/jason/tidb/session/session.go:2943\nmain.createStoreAndDomain\n\t/home/jason/tidb/tidb-server/main.go:310\nmain.main\n\t/home/jason/tidb/tidb-server/main.go:211\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]

[fzzf678]

How can i test this in my local machine，must in k8s？M1 chips seems not support 4.0.16

[Defined2014]

How can i test this in my local machine，must in k8s？M1 chips seems not support 4.0.16

Not must in k8s. Any v4.0.16 upgrade to master will cause this problem. Maybe you need a x86 machine to test it.

modify config.toml.example like

diff --git a/config/config.toml.example b/config/config.toml.example
index 588379f20..14e7936b7 100644
--- a/config/config.toml.example
+++ b/config/config.toml.example
@@ -10,10 +10,10 @@ advertise-address = ""
 port = 4000

 # Registered store name, [tikv, mocktikv, unistore]
-store = "unistore"
+store = "tikv"

 # TiDB storage path.
-path = "/tmp/tidb"
+path = "127.0.0.1:2379"

 # The socket file to use for connection.
 socket = "/tmp/tidb-{Port}.sock"

then execute like below, @fzzf678

tiup playground v6.4.0 --db 0 --tiflash 0 --without-monitor
./tidb-server-4-0-16 --config=config/config.toml.example # use 4.0.16 to create system table
ctrl + c
./tidb-server --config=config/config.toml.example # use master to update system table