FreeBSD build timed out, failing

[Trott]

https://ci.nodejs.org/job/node-test-commit-freebsd/27478/nodes=freebsd11-x64/console

00:10:22 Build timed out (after 6 minutes). Marking the build as failed.
00:10:22 Build was aborted

Same thing at https://ci.nodejs.org/job/node-test-commit-freebsd/27409/nodes=freebsd11-x64/console.

Two build failures in a week is not too bad, really, but would love to know the cause and if there's potential mitigation. (Does we just need to increase the timeout and everything will be fine?)

/ping @nodejs/platform-freebsd in case someone wants to dig on on this.

[nomadlogic]

It looks like both of the referenced builds failed at two separate phases of the build process.

unfortunately its not clear to me at least as to what the build bot was doing during those 6mins. is it possible to get the system logs themselves to see if there is another error happening that is causing the build to hang? Perhaps an out-of-mem condition?

[rvagg]

@nomadlogic what's the best way to see system logs on freebsd?

Using dmesg from a Linux user instinct I see a whole bunch of swap_pager_getswapspace(9): failed. I'm not sure the best way to see how much memory or swap this thing has but top is showing me:

Mem: 56M Active, 70M Inact, 358M Wired, 41K Buf, 1474M Free
ARC: 90M Total, 21M MFU, 50M MRU, 800K Anon, 1081K Header, 17M Other
     48M Compressed, 68M Uncompressed, 1.42:1 Ratio
Swap: 2048M Total, 396M Used, 1651M Free, 19% Inuse

Other than that message, there is a ton of this type of thing: pid 48547 (node), uid 1002: exited on signal 4 (4's, 6's, some 11's and some of them have (core dumped) too). I don't see any timestamps associated with these things and there's only those two types of errors in there, the swap one has 2,582, followed by 119 of the exit one.

/var/log/messages and /var/log/messages.*.bz2 are pretty much all those same error messages repeated over and over. I see nothing else.

[rvagg]

btw these machines are pretty much as they come from DigitalOcean's FreeBSD droplets plus some basic stuff to get the jenkins client running on there, I don't see anything particularly notable in ansible/ for freebsd other than the usual update/upgrade, install compiler & java and start a jenkins service.

[nomadlogic]

by the looks of it these systems are running out of memory during the compilation process. if i'm reading this correct it looks like there is 4GB of ram allocated to these instances, which ZFS will try to make use of as well. so the systems are starting to page to disk during compilation and failing.

if possible it may be worth seeing if memory can be increased, alternatively additional swap can be added as per this document (specifically by using a swapfile):
https://www.freebsd.org/doc/handbook/adding-swap-space.html

[emaste]

by the looks of it these systems are running out of memory during the compilation process

Exactly this; if using ZFS we can try tuning it to be more conservative with memory use, or try using UFS instead. I assume the same DO config is used on Linux? (i.e., 4GB droplets)?

[github-actions]

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.

[Trott]

I haven't seen this in a while, so I'll close. Feel free to re-open if I'm wrong and this is still A Thing.