arm-fanned perma-build-failing

[Trott]

arm-fanned task git-nodesource-update-reference is red eight times in a row.

remote: Compressing objects: 100% (91555/91555), done.        
21:26:48 Receiving objects:   0% (1/394400)   
fatal: write error: Bad file descriptor
21:26:48 fatal: index-pack failed
21:26:49 debug1: client_input_channel_req: channel 0 rtype exit-signal reply 0
21:26:50 Build step 'Execute shell' marked build as failure
21:26:50 Collecting metadata...
21:26:50 Metadata collection done.
21:26:50 Notifying upstream projects of job completion
21:26:50 Finished: FAILURE




arm-fanned combined with "Bad file descriptor" makes me automatically suspect NFS trouble, but that may be a guess from anecdote and ignorance. @rvagg

[rvagg]

taken arm-fanned offline, time for some whole-cluster maintenance I think, it's been a couple of months.

[rvagg]

@refack: there's 2 pi's marked offline by you back in April: test-requireio_securogroup-debian9-armv6l_pi1p-2 "Keep offline to save space" and test-requireio_davglass-debian9-armv6l_pi1p-1 "Keep offline to safe space" - can you explain what this means?

test-requireio_davglass-debian9-armv6l_pi1p-1 has been dodgy in the past, I wouldn't be surprised if it's a perma-flake at the hardware level, but I'm not sure if I can interpret these messages as "Rod needs to check the hardware" or some other thing that can just be re-enabled?

@Trott they're enabled again after an update and some reboots. Seems to be a permafail on test.parallel/test-worker-debug on master though. Is this a known issue?

[sam-github]

@nodejs/workers See above, test-worker-debug permafailing on arm, shall we mark it as flaky?

[Trott]

@rvagg @sam-github test-worker-debug is a known-flaky on Windows, but I don't recall it being much of an issue on ARM. nodejs/node#28106

[refack]

there's 2 pi's marked offline by you back in April "Keep offline to save space"

Since we don't test master and v12 on PI1, I wanted to see if having two machines offline will stave off filling up the shared storage, and/or cause any significant backlog.
I think it's a no on both - no significant backlog on PI1s, but we had hit storage overflow twice since then.

tl;dr AFAICT we can permanently retire 2 PI1 machines, if we want.

[rvagg]

@refack thanks for the context

tl;dr AFAICT we can permanently retire 2 PI1 machines, if we want.
This is good news because I know a couple of them are hardware-level flaky. Can't be 100% certain about it but I seem to be fixing the same ones over and over. I might put them all back into rotation for now and knock out dodgy ones as they pop up their heads.

[BridgeAR]

Is this still up to date or could this be closed?

[rvagg]

still up to date, still offline, still haven't budgeted time to shave dem yaks

[rvagg]

bringing these back online, first casualty of them being offline seems to be nodejs/node#30786

will write up what I've done to get these back online in a PR for the changes once I'm all done

[rvagg]

moving to #2077