integration test: jenkins running the wrong binary but showing the right commit id

[medyagh]

while doing this PR: #5530

I noticed a spooky behaviour that my integraiton tests would pass on local machine but fail horribly the jenkins

The logs showed that it had the right commit :

https://storage.googleapis.com/minikube-builds/logs/5530/KVM_Linux.txt

Started by upstream project "Build_Cross" build number 8647
originally caused by:
 GitHub pull request #5530 of commit 390992571fbad36aff04fd08fd59e6d5b8a36415, no merge conflicts.
Running as SYSTEM
[EnvInject] - Loading node environment variables.
[EnvInject] - Preparing an environment for the bui

but I suspected that it actually runs the right binary !
I added a logging message and I expected it to see in the logs:

exit.WithError("Failed to update cluster<--Temproary logging-->", err)

however I got this error that doesn't belong to that code !

X Failed to update cluster: starting kubelet command: "/bin/bash -c "sudo systemctl daemon-reload && sudo systemctl start kubelet"" output: "\n** stderr ** \nusage: sudo -h | -K | -k | -V\nusage: sudo -v [-AknS] [-g group] [-h host] [-p prompt] [-u user]\nusage: sudo -l [-AknS] [-g group] [-h host] [-p prompt] [-U user] [-u user]\n            [command]\nusage: sudo [-AbEHknPS] [-C num] [-g group] [-h host] [-p prompt] [-T timeout]\n            [-u user] [VAR=value] [-i|-s] []\nusage: sudo -e [-AknS] [-C num] [-g group] [-h host] [-p prompt] [-T timeout]\n            [-u user] file ...\n\n** /stderr **": Process exited with status 1

10:42:40         *

10:42:40         * Sorry that minikube crashed. If this was unexpected, we would love to hear from you:

Also : I see the following failed to delete in the beggining of the script

11:03:34 >> Cleaning up after previous test runs ...
11:03:34 * Cleaning stale test root: /home/jenkins/minikube-integration/linux-amd64-kvm2-5225-12310-98e906446c6a6a4dadd13b0a6537dc9f1b3d895a
11:03:34 rmdir: failed to remove '/home/jenkins/minikube-integration/linux-amd64-kvm2-5225-12310-98e906446c6a6a4dadd13b0a6537dc9f1b3d895a': Directory not empty
11:03:34 test.json
11:03:34 test.out
11:03:34 * Cleaning stale test root: /home/jenkins/minikube-integration/linux-amd64-kvm2-5225-29692-c3c62a451c0b089f352485fa9bf668b86bdb5ba4
11:03:34 rmdir: failed to remove '/home/jenkins/minikube-integration/linux-amd64-kvm2-5225-29692-c3c62a451c0b089f352485fa9bf668b86bdb5ba4': Directory not empty
11:03:34 tests.json
11:03:34 * Cleaning stale test root: /home/jenkins/minikube-integration/linux-amd64-kvm2-5225-603-9e58a9a0b93e0f3ba132b98f5ef2df806bf7898b
11:03:34 rmdir: failed to remove '/home/jenkins/minikube-integration/linux-amd64-kvm2-5225-603-9e58a9a0b93e0f3ba132b98f5ef2df806bf7898b': Directory not empty
11:03:34 test.json
11:03:34 * Cleaning stale test root: /home/jenkins/minikube-integration/linux-amd64-none-5225-21635-079c256655b4f84a00b505d8f53b396b52e22885
11:03:34 rmdir: failed to remove '/home/jenkins/minikube-integration/linux-amd64-none-5225-21635-079c256655b4f84a00b505d8f53b396b52e22885': Directory not empty

[medyagh]

I added the following to the common.sh integration script:

echo ">> Starting ${E2E_BIN} at $(date)"
echo ">> The minikube version & commit id:"
echo $(out/minikube-${OS_ARCH} version)

and I expected it to see in the logs but !!!! it did not show up, even though the logs point to the right commit ID the common.sh is not the new one !

11:22:21 >> Starting out/e2e-linux-amd64 at Sat Oct 12 18:22:21 UTC 2019
11:23:13 2019/10/12 18:23:13 [DEBUG] GET http://10.138.0.2:5000
11:25:21 2019/10/12 18:25:21 [DEBUG] GET http://127.0.0.1:40735/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/
11:30:40 --- FAIL: TestStartStop (168.17s)
11:30:40     --- FAIL: TestStartStop/group (168.

no sign of the version I added !!!!

[tstromberg]

@sharifelgamal mentioned there were some observed gsutil behavioral changes with a recent update, which got me thinking: this has to be related.

I took a look at the repository, and did an experiment, and indeed, updated scripts/binaries are being placed into a subfolder rather than at root. Here is what I believe is the errant command:

minikube/hack/jenkins/minikube_cross_build_and_upload.sh

Line 66 in 73bf396

gsutil -m cp -r out/ "gs://${bucket}/${ghprbPullId}/"

The first time this command is run, it creates a new folder: gs://minikube-builds/5530

However, the second time this command is run, a new folder is created by gsutil v4.43 (what jenkins master has): gs://minikube-builds/5530/out/

I believe this behavioral change is what the gsutil changelog states coyly as: "Fixed cp/mv errors that occur when destination directory is a substring of another directory. Behavior now more closely matches OS cp/mv commands.".

I'll send out a PR to fix the command line.

[medyagh]

Ahh !!!

[medyagh]

@tstromberg that was some great detective work !

[tstromberg]

PR's affected by this bug, based on output of gsutil ls 'gs://minikube-builds/*/out'

#4780 #4999 #5243 #5400 #5425 #5471 #5530 #5543 #5548 #5550 #5554 #5555 #5559 #5565 #5566 #5572 #5574 #5582 #5584 #5593 #5601 #5607