perma-red Windows fanned

[Trott]

Dozens of failures in a row in the git-rebase step....

Example:

https://ci.nodejs.org/job/git-rebase/59837/

19:08:51 Started by upstream project "node-test-commit-arm-fanned" build number 12166
19:08:51 originally caused by:
19:08:51  Started by upstream project "node-test-commit" build number 34405
19:08:51  originally caused by:
19:08:51   Started by upstream project "node-test-pull-request" build number 28040
19:08:51   originally caused by:
19:08:51    Started by user Gabriel Schulhof
19:08:51 Running as SYSTEM
19:08:51 [EnvInject] - Loading node environment variables.
19:08:51 Building remotely on test-packetnet-ubuntu1604-x64-1 (Ubuntu Ubuntu-16.04 amd64-Ubuntu jenkins-workspace amd64-Ubuntu-16.04 16.04 amd64) in workspace /home/iojs/build/workspace/git-rebase
19:08:51 [ssh-agent] Looking for ssh-agent implementation...
19:08:51 [ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
19:08:51 $ ssh-agent
19:08:51 SSH_AUTH_SOCK=/tmp/ssh-scrMeWVDmQom/agent.15278
19:08:51 SSH_AGENT_PID=15280
19:08:51 [ssh-agent] Started.
19:08:51 Running ssh-add (command line suppressed)
19:08:52 Identity added: /home/iojs/build/workspace/git-rebase@tmp/private_key_6689631567002377301.key (/home/iojs/build/workspace/git-rebase@tmp/private_key_6689631567002377301.key)
19:08:52 [ssh-agent] Using credentials iojs (ci@iojs.org GitHub)
19:08:52 using credential 96d5f81c-e9ad-45f7-ba5d-bc8107c0ae2c
19:08:52  > git rev-parse --is-inside-work-tree # timeout=10
19:08:52 Fetching changes from the remote Git repository
19:08:52  > git config remote.origin.url git@github.com:nodejs/node.git # timeout=10
19:08:52 Fetching upstream changes from git@github.com:nodejs/node.git
19:08:52  > git --version # timeout=10
19:08:52 using GIT_SSH to set credentials 
19:08:52  > git fetch --tags --progress git@github.com:nodejs/node.git +refs/heads/*:refs/remotes/origin/* +refs/pull/31063/head:refs/remotes/origin/_jenkins_local_branch # timeout=30
19:08:54  > git rev-parse refs/remotes/origin/_jenkins_local_branch^{commit} # timeout=10
19:08:54  > git rev-parse refs/remotes/origin/origin/_jenkins_local_branch^{commit} # timeout=10
19:08:54 Checking out Revision c48f9d1f1ade8d85d6692add9b690a7e8810b52c (refs/remotes/origin/_jenkins_local_branch)
19:08:54  > git config core.sparsecheckout # timeout=10
19:08:54  > git checkout -f c48f9d1f1ade8d85d6692add9b690a7e8810b52c # timeout=10
19:08:54 Commit message: "build: warn upon --use-largepages config option"
19:08:54 Using 'Changelog to branch' strategy.
19:08:54 Cleaning workspace
19:08:54  > git rev-parse --verify HEAD # timeout=10
19:08:54 Resetting working tree
19:08:54  > git reset --hard # timeout=10
19:08:55  > git clean -fdx # timeout=10
19:08:55 Run condition [Always] enabling prebuild for step [[Execute shell]]
19:08:55 [git-rebase] $ /bin/sh -xe /tmp/jenkins3602013457042107307.sh
19:08:55 + [ -n b08601bf79cddbd6fce904d10c70b6dc1051f1be ]
19:08:55 + git config --replace-all user.name Jenkins Rebaser
19:08:55 + git config --replace-all user.email dummy@dummy.com
19:08:55 + git config user.name
19:08:55 Jenkins Rebaser
19:08:55 + git config user.email
19:08:55 dummy@dummy.com
19:08:55 + echo GIT_COMMITTER_NAME:
19:08:55 GIT_COMMITTER_NAME:
19:08:55 + echo GIT_COMMITTER_EMAIL:
19:08:55 GIT_COMMITTER_EMAIL:
19:08:55 + echo GIT_COMMITTER_DATE:
19:08:55 GIT_COMMITTER_DATE:
19:08:55 + echo GIT_AUTHOR_NAME:
19:08:55 GIT_AUTHOR_NAME:
19:08:55 + git rebase --abort
19:08:55 No rebase in progress?
19:08:55 + true
19:08:55 + git checkout -f refs/remotes/origin/_jenkins_local_branch
19:08:56 HEAD is now at c48f9d1... build: warn upon --use-largepages config option
19:08:56 + git config user.name
19:08:56 Jenkins Rebaser
19:08:56 + git config user.email
19:08:56 dummy@dummy.com
19:08:56 + echo GIT_COMMITTER_NAME:
19:08:56 GIT_COMMITTER_NAME:
19:08:56 + echo GIT_COMMITTER_EMAIL:
19:08:56 GIT_COMMITTER_EMAIL:
19:08:56 + echo GIT_COMMITTER_DATE:
19:08:56 GIT_COMMITTER_DATE:
19:08:56 + echo GIT_AUTHOR_NAME:
19:08:56 GIT_AUTHOR_NAME:
19:08:56 + git status
19:08:56 HEAD detached at c48f9d1
19:08:56 nothing to commit, working directory clean
19:08:56 + git rev-parse HEAD
19:08:56 c48f9d1f1ade8d85d6692add9b690a7e8810b52c
19:08:56 + git rev-parse b08601bf79cddbd6fce904d10c70b6dc1051f1be
19:08:56 b08601bf79cddbd6fce904d10c70b6dc1051f1be
19:08:56 + git rebase b08601bf79cddbd6fce904d10c70b6dc1051f1be
19:08:56 Current branch HEAD is up to date.
19:08:56 + git rev-parse HEAD
19:08:56 c48f9d1f1ade8d85d6692add9b690a7e8810b52c
19:08:56 [git-rebase] $ /bin/bash -ex /tmp/jenkins5125851323422266215.sh
19:08:56 + git checkout -B jenkins-node-test-commit-arm-fanned-c48f9d1f1ade8d85d6692add9b690a7e8810b52c
19:08:57 Switched to and reset branch 'jenkins-node-test-commit-arm-fanned-c48f9d1f1ade8d85d6692add9b690a7e8810b52c'
19:08:57 + grep -q '^169.60.150.88' /home/iojs/.ssh/known_hosts
19:08:57 + git push binary_tmp@169.60.150.88:binary_tmp.git +jenkins-node-test-commit-arm-fanned-c48f9d1f1ade8d85d6692add9b690a7e8810b52c
19:08:59 error: refs/heads/jenkins-node-test-commit-arm-fanned-7a736d9b15afec058da10c88df6d6c11a8d39463-binary-pi1p/cc-armv7 does not point to a valid object!
19:08:59 remote: error: refs/heads/jenkins-node-test-commit-arm-fanned-7a736d9b15afec058da10c88df6d6c11a8d39463-binary-pi1p/cc-armv7 does not point to a valid object!        
19:08:59 remote: fatal: bad object .alternate        
19:09:00 error: refs/heads/jenkins-node-test-commit-arm-fanned-7a736d9b15afec058da10c88df6d6c11a8d39463-binary-pi1p/cc-armv7 does not point to a valid object!
19:09:00 fatal: bad object .alternate
19:09:00 To binary_tmp@169.60.150.88:binary_tmp.git
19:09:00  ! [remote rejected] jenkins-node-test-commit-arm-fanned-c48f9d1f1ade8d85d6692add9b690a7e8810b52c -> jenkins-node-test-commit-arm-fanned-c48f9d1f1ade8d85d6692add9b690a7e8810b52c (missing necessary objects)
19:09:00 error: failed to push some refs to 'binary_tmp@169.60.150.88:binary_tmp.git'
19:09:00 Build step 'Execute shell' marked build as failure
19:09:00 $ ssh-agent -k
19:09:00 unset SSH_AUTH_SOCK;
19:09:00 unset SSH_AGENT_PID;
19:09:00 echo Agent pid 15280 killed;
19:09:00 [ssh-agent] Stopped.
19:09:00 Run condition [Always] enabling perform for step [[Execute shell]]
19:09:00 [git-rebase] $ /bin/sh -xe /tmp/jenkins8276278637949751558.sh
19:09:00 + [ -e .git/index.lock ]
19:09:00 + du -s .git
19:09:00 + cut -f1
19:09:00 + [ 435576 -gt 5000000 ]
19:09:00 Collecting metadata...
19:09:00 Metadata collection done.
19:09:00 Notifying upstream projects of job completion
19:09:00 Finished: FAILURE

@joaocgreis (sorry)

[rvagg]

Same error as #2115, borked binary_tmp git repo that we use to share binaries between build machines. It's not the first time this has happened either. I'm not sure what the proper way to fix such git problems is (git fsck doesn't do it, git gc borks) so I just manually remove the file in question. In this case it's removal of /home/binary_tmp/binary_tmp.git/refs/heads/jenkins-node-test-commit-arm-fanned-7a736d9b15afec058da10c88df6d6c11a8d39463-binary-pi1p/cc-armv7 on the machine.
nodejs_build_test people have access to do this too btw, it's on the server indicated by that IP address, ssh root@169.60.150.88 -i ~/.ssh/nodejs_build_test (or just ssh test-softlayer-ubuntu1604-x64-1 if your ~/.ssh/config is setup properly). Just make sure the directory is entirely owned by binary_tmp when you're done if you modify anything in there.

[rvagg]

running test job https://ci.nodejs.org/job/node-test-commit/34429/

[rvagg]

https://ci.nodejs.org/job/node-test-commit/34429/ good except for what looks like some flakes