TiUP cluster does not work with builtin ssh executor

[sticnarf]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do?

Using tiup cluster to manage a cluster deployed locally. For example:

tiup cluster start test # --ssh builtin

If I specify --ssh system, it works well.

2. What did you expect to see?

Start the cluster successfully.

3. What did you see instead?

2021-11-03T16:44:58.672+0800    INFO    Starting component pd
2021-11-03T16:44:58.672+0800    INFO            Starting instance 127.0.0.1:2379
2021-11-03T16:44:58.690+0800    ERROR   SSHCommand      {"host": "127.0.0.1", "port": "22", "cmd": "export LANG=C; PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin /usr/bin/sudo -H bash -c \"systemctl daemon-reload && systemctl start pd-2379.service\"", "error": "ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain", "stdout": "", "stderr": ""}
2021-11-03T16:44:58.690+0800    ERROR   CheckPoint      {"host": "127.0.0.1", "port": 22, "user": "tidb", "sudo": true, "cmd": "systemctl daemon-reload && systemctl start pd-2379.service", "stdout": "", "stderr": "", "error": "executor.ssh.execute_failed: Failed to execute command over SSH for 'tidb@127.0.0.1:22' {ssh_stderr: , ssh_stdout: , ssh_command: export LANG=C; PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin /usr/bin/sudo -H bash -c \"systemctl daemon-reload && systemctl start pd-2379.service\"}, cause: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain", "errorVerbose": "executor.ssh.execute_failed: Failed to execute command over SSH for 'tidb@127.0.0.1:22' {ssh_stderr: , ssh_stdout: , ssh_command: export LANG=C; PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin /usr/bin/sudo -H bash -c \"systemctl daemon-reload && systemctl start pd-2379.service\"}, cause: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain\n at github.com/pingcap/tiup/pkg/cluster/executor.(*EasySSHExecutor).Execute()\n\tgithub.com/pingcap/tiup/pkg/cluster/executor/ssh.go:174\n at github.com/pingcap/tiup/pkg/cluster/executor.(*CheckPointExecutor).Execute()\n\tgithub.com/pingcap/tiup/pkg/cluster/executor/checkpoint.go:85\n at github.com/pingcap/tiup/pkg/cluster/module.(*SystemdModule).Execute()\n\tgithub.com/pingcap/tiup/pkg/cluster/module/systemd.go:98\n at github.com/pingcap/tiup/pkg/cluster/operation.systemctl()\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:395\n at github.com/pingcap/tiup/pkg/cluster/operation.startInstance()\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:373\n at github.com/pingcap/tiup/pkg/cluster/operation.StartComponent.func1()\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:499\n at golang.org/x/sync/errgroup.(*Group).Go.func1()\n\tgolang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\n at runtime.goexit()\n\truntime/asm_amd64.s:1581", "__hash__": "1f6ca2b18e5c96c075efb4c08084d6dc8659050b", "__func__": "github.com/pingcap/tiup/pkg/cluster/executor.(*CheckPointExecutor).Execute", "hit": false}
2021-11-03T16:44:58.690+0800    DEBUG   TaskFinish      {"task": "StartCluster", "error": "failed to start pd: failed to start: 127.0.0.1 pd-2379.service, please check the instance's log(/data/deploy/pd-2379/log) for more detail.: executor.ssh.execute_failed: Failed to execute command over SSH for 'tidb@127.0.0.1:22' {ssh_stderr: , ssh_stdout: , ssh_command: export LANG=C; PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin /usr/bin/sudo -H bash -c \"systemctl daemon-reload && systemctl start pd-2379.service\"}, cause: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain", "errorVerbose": "executor.ssh.execute_failed: Failed to execute command over SSH for 'tidb@127.0.0.1:22' {ssh_stderr: , ssh_stdout: , ssh_command: export LANG=C; PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin /usr/bin/sudo -H bash -c \"systemctl daemon-reload && systemctl start pd-2379.service\"}, cause: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain\n at github.com/pingcap/tiup/pkg/cluster/executor.(*EasySSHExecutor).Execute()\n\tgithub.com/pingcap/tiup/pkg/cluster/executor/ssh.go:174\n at github.com/pingcap/tiup/pkg/cluster/executor.(*CheckPointExecutor).Execute()\n\tgithub.com/pingcap/tiup/pkg/cluster/executor/checkpoint.go:85\n at github.com/pingcap/tiup/pkg/cluster/module.(*SystemdModule).Execute()\n\tgithub.com/pingcap/tiup/pkg/cluster/module/systemd.go:98\n at github.com/pingcap/tiup/pkg/cluster/operation.systemctl()\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:395\n at github.com/pingcap/tiup/pkg/cluster/operation.startInstance()\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:373\n at github.com/pingcap/tiup/pkg/cluster/operation.StartComponent.func1()\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:499\n at golang.org/x/sync/errgroup.(*Group).Go.func1()\n\tgolang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\n at runtime.goexit()\n\truntime/asm_amd64.s:1581\nfailed to start: 127.0.0.1 pd-2379.service, please check the instance's log(/data/deploy/pd-2379/log) for more detail.\ngithub.com/pingcap/tiup/pkg/cluster/operation.toFailedActionError\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:607\ngithub.com/pingcap/tiup/pkg/cluster/operation.startInstance\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:374\ngithub.com/pingcap/tiup/pkg/cluster/operation.StartComponent.func1\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:499\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1581\nfailed to start pd"}
2021-11-03T16:44:58.690+0800    INFO    Execute command finished        {"code": 1, "error": "failed to start pd: failed to start: 127.0.0.1 pd-2379.service, please check the instance's log(/data/deploy/pd-2379/log) for more detail.: executor.ssh.execute_failed: Failed to execute command over SSH for 'tidb@127.0.0.1:22' {ssh_stderr: , ssh_stdout: , ssh_command: export LANG=C; PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin /usr/bin/sudo -H bash -c \"systemctl daemon-reload && systemctl start pd-2379.service\"}, cause: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain", "errorVerbose": "executor.ssh.execute_failed: Failed to execute command over SSH for 'tidb@127.0.0.1:22' {ssh_stderr: , ssh_stdout: , ssh_command: export LANG=C; PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin /usr/bin/sudo -H bash -c \"systemctl daemon-reload && systemctl start pd-2379.service\"}, cause: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain\n at github.com/pingcap/tiup/pkg/cluster/executor.(*EasySSHExecutor).Execute()\n\tgithub.com/pingcap/tiup/pkg/cluster/executor/ssh.go:174\n at github.com/pingcap/tiup/pkg/cluster/executor.(*CheckPointExecutor).Execute()\n\tgithub.com/pingcap/tiup/pkg/cluster/executor/checkpoint.go:85\n at github.com/pingcap/tiup/pkg/cluster/module.(*SystemdModule).Execute()\n\tgithub.com/pingcap/tiup/pkg/cluster/module/systemd.go:98\n at github.com/pingcap/tiup/pkg/cluster/operation.systemctl()\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:395\n at github.com/pingcap/tiup/pkg/cluster/operation.startInstance()\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:373\n at github.com/pingcap/tiup/pkg/cluster/operation.StartComponent.func1()\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:499\n at golang.org/x/sync/errgroup.(*Group).Go.func1()\n\tgolang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\n at runtime.goexit()\n\truntime/asm_amd64.s:1581\nfailed to start: 127.0.0.1 pd-2379.service, please check the instance's log(/data/deploy/pd-2379/log) for more detail.\ngithub.com/pingcap/tiup/pkg/cluster/operation.toFailedActionError\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:607\ngithub.com/pingcap/tiup/pkg/cluster/operation.startInstance\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:374\ngithub.com/pingcap/tiup/pkg/cluster/operation.StartComponent.func1\n\tgithub.com/pingcap/tiup/pkg/cluster/operation/action.go:499\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1581\nfailed to start pd"}

4. What version of TiUP are you using (tiup --version)?

1.6.1 tiup
Go Version: go1.17.2
Git Ref: v1.6.1
GitHash: 1f6ca2b18e5c96c075efb4c08084d6dc8659050b

[AstroProfundis]

I suspect the SSH server on your deployment machine (localhost in this case) is too new and share no common cipher with our builtin SSH implementation...

[sticnarf]

I suspect the SSH server on your deployment machine (localhost in this case) is too new and share no common cipher with our builtin SSH implementation...

Ah, I see. OpenSSH 8.8p1 disables RSA/SHA-1 authentication. SHA-256 signature must be used instead.

[sticnarf]

Blocked by golang/go#39885

[ahmedriza]

It will be good to add the workaround to the documentation, https://docs.pingcap.com/tidb/stable/quick-start-with-tidb#Linux. That is to use --ssh=system as I also ran into this and spent quite a bit of time before seeing the work around after a Google search.