Local strategy fails on port forwarding in our case of cross-cluster migration (error upgrading connection: Unauthorized)

[lgyurci]

Describe the bug
Generally, port forwarding fails while using local strategy.
Digging more into the problem, it seems like the Dialer object (in portforward.go, after calling forwarder.ForwardPorts()) throws back an error, stating error upgrading connection: Unauthorized. The strange thing is, that pv-migrate is easily able to make a connection to the cluster, because the pods are created properly. Port forwarding with kubectl also works without an issue, with the same kube config files (and yes, I have tried to explicitly give the context files to pv-migrate as well). Comparing the source code of pv-migrate with kubectl, the used RestConfig is a bit different, but the BearerToken is the same in both cases, so that shouldn't be the issue. Other than that, the port forwarding code is (at first glance) identical with the one used in kubectl.

To Reproduce
Steps to reproduce the behavior:

1. Run command pv-migrate migrate --log-level trace -s local -i -c cluster1 -C cluster2 -N testns -n testns vol-claim1 vol-claim2
2. See error

Expected behavior
Port forward should not fail.

Console output

🚀  Starting migration
💡  PVC vol-claim1 is mounted to node cluster1-node2, ignoring...
💡  PVC vol-claim2 is mounted to node cluster2-node1, ignoring...
💭  Will attempt 1 strategies: local
🚁  Attempting strategy: local
🔑  Generating SSH key pair
creating 4 resource(s)
beginning wait for 4 resources with timeout of 1m0s
Deployment is not ready: testns/pv-migrate-abebc-src-sshd. 0 out of 1 expected pods are ready
Deployment is not ready: testns/pv-migrate-abebc-src-sshd. 0 out of 1 expected pods are ready
creating 4 resource(s)
beginning wait for 4 resources with timeout of 1m0s
Deployment is not ready: testns/pv-migrate-abebc-dest-sshd. 0 out of 1 expected pods are ready
Deployment is not ready: testns/pv-migrate-abebc-dest-sshd. 0 out of 1 expected pods are ready
❌  Error on port-forward
🧹  Cleaning up
uninstall: Deleting pv-migrate-abebc-src
uninstall: given cascade value: , defaulting to delete propagation background
Starting delete for "pv-migrate-abebc-src-sshd" Service
Starting delete for "pv-migrate-abebc-src-sshd" Deployment
Starting delete for "pv-migrate-abebc-src-sshd" Secret
Starting delete for "pv-migrate-abebc-src-sshd" ServiceAccount
beginning wait for 4 resources to be deleted with timeout of 1m0s
purge requested for pv-migrate-abebc-src
uninstall: Deleting pv-migrate-abebc-dest
uninstall: given cascade value: , defaulting to delete propagation background
Starting delete for "pv-migrate-abebc-dest-sshd" Service
Starting delete for "pv-migrate-abebc-dest-sshd" Deployment
Starting delete for "pv-migrate-abebc-dest-sshd" Secret
Starting delete for "pv-migrate-abebc-dest-sshd" ServiceAccount
beginning wait for 4 resources to be deleted with timeout of 1m0s
purge requested for pv-migrate-abebc-dest
✨  Cleanup done
🔶  Migration failed with this strategy, will try with the remaining strategies
Error: all strategies failed

Version

• Source and destination Kubernetes versions: v1.23.12
• Source and destination container runtimes: docker://23.0.5
• pv-migrate version and architecture: v1.2.0 - linux_x86_64
• Installation method: binary download and also built from source
• Source and destination PVC type, size and accessModes: ReadWriteMany, 8G, rook ceph cephfs

Additional context
Rancher clusters. Probably not reproducible in every cluster, since I'm the first reporter of this issue.

[utkuozdemir]

Thanks for the detailed report.

Can you retry with the args --log-level=debug --log-format=json? It might shed some light on the underlying error.

[lgyurci]

Sure. I don't see any extra info in it unfortunately.

{"dest":"vol-claim","dest_ns":"testns","level":"info","msg":":bulb: PVC vol-claim is mounted to node cluster1-node2, ignoring...","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:15:55+02:00"}
{"dest":"vol-claim","dest_ns":"testns","level":"info","msg":":bulb: PVC vol-claim is mounted to node cluster2-node1, ignoring...","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:15:55+02:00"}
{"dest":"vol-claim","dest_ns":"testns","level":"info","msg":":thought_balloon: Will attempt 1 strategies: local","source":"vol-claim","source_ns":"testns","strategies":"local","time":"2023-08-21T09:15:55+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"info","msg":":helicopter: Attempting strategy: local","source":"vol-claim","source_ns":"testns","strategy":"local","time":"2023-08-21T09:15:55+02:00"}
{"dest":"vol-claim","dest_ns":"testns","level":"info","msg":":key: Generating SSH key pair","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:15:55+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"creating 4 resource(s)","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:15:57+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"beginning wait for 4 resources with timeout of 1m0s","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:15:57+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"Deployment is not ready: testns/pv-migrate-daeec-src-sshd. 0 out of 1 expected pods are ready","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:15:57+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"Deployment is not ready: testns/pv-migrate-daeec-src-sshd. 0 out of 1 expected pods are ready","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:15:59+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"Deployment is not ready: testns/pv-migrate-daeec-src-sshd. 0 out of 1 expected pods are ready","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:16:01+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"creating 4 resource(s)","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:16:05+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"beginning wait for 4 resources with timeout of 1m0s","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:16:05+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"Deployment is not ready: testns/pv-migrate-daeec-dest-sshd. 0 out of 1 expected pods are ready","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:16:05+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"Deployment is not ready: testns/pv-migrate-daeec-dest-sshd. 0 out of 1 expected pods are ready","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:16:07+02:00"}
{"dest":"vol-claim","dest_ns":"testns","error":"error upgrading connection: Unauthorized","level":"error","msg":":cross_mark: Error on port-forward","name":"pv-migrate-daeec-src-sshd-76b7f4556-2rm7w","ns":"testns","port":36935,"source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:17:09+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"info","msg":":broom: Cleaning up","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:17:39+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"uninstall: Deleting pv-migrate-daeec-src","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:17:39+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"uninstall: given cascade value: , defaulting to delete propagation background","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:17:40+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"Starting delete for \"pv-migrate-daeec-src-sshd\" Service","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:17:40+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"Starting delete for \"pv-migrate-daeec-src-sshd\" Deployment","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:17:40+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"Starting delete for \"pv-migrate-daeec-src-sshd\" Secret","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:17:40+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"Starting delete for \"pv-migrate-daeec-src-sshd\" ServiceAccount","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:17:40+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"beginning wait for 4 resources to be deleted with timeout of 1m0s","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:17:40+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"purge requested for pv-migrate-daeec-src","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:17:40+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"uninstall: Deleting pv-migrate-daeec-dest","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:17:41+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"uninstall: given cascade value: , defaulting to delete propagation background","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:17:42+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"Starting delete for \"pv-migrate-daeec-dest-sshd\" Service","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:17:42+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"Starting delete for \"pv-migrate-daeec-dest-sshd\" Deployment","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:17:42+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"Starting delete for \"pv-migrate-daeec-dest-sshd\" Secret","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:17:42+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"Starting delete for \"pv-migrate-daeec-dest-sshd\" ServiceAccount","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:17:42+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"beginning wait for 4 resources to be deleted with timeout of 1m0s","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:17:42+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"debug","msg":"purge requested for pv-migrate-daeec-dest","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:17:42+02:00"}
{"dest":"vol-claim","dest_ns":"testns","id":"daeec","level":"info","msg":":sparkles: Cleanup done","source":"vol-claim","source_ns":"testns","time":"2023-08-21T09:17:42+02:00"}
{"dest":"vol-claim","dest_ns":"testns","error":"timed out waiting for port-forward to be ready","id":"daeec","level":"warning","msg":":large_orange_diamond: Migration failed with this strategy, will try with the remaining strategies","source":"vol-claim","source_ns":"testns","strategy":"local","time":"2023-08-21T09:17:42+02:00"}```