Disk space too low on the Jenkins CI server

[joaocgreis]

Note: this is NOT related to #3287 - different hosts.

This morning I noticed https://ci.nodejs.org/manage/computer/(built-in)/ was offline because disk space was too low. I confirmed job logs were taking up all the available space.

I suspected the backup server was not deleting old jobs as expected, so I logged in and run https://github.com/nodejs/build/blob/main/backup/backup_scripts/remove_old.sh manually. That freed 59G on the ci server.

I also noticed the backup server is configured to run the script weekly and not daily as per the instructions in https://github.com/nodejs/build/tree/main/backup. I don't know why that change was made and the immediate issue is fixed, so I'll leave it as I found it.

The dates on the old logs seem to confirm everything is running as it should, early March makes sense if we consider the last run was last week and it removed everything older than 3 weeks at the time. Also, the oldest node-compile-windows is 10 days old - it is configured to be cleaned by Jenkins on the job configuration because it takes up more space than other jobs.

Either this was a one-off because of some surge in job runs, or 21 days became too much for the load we have now. We have to re-evaluate if this happens again.

[richardlau]

Either this was a one-off because of some surge in job runs, or 21 days became too much for the load we have now. We have to re-evaluate if this happens again.

I rather suspect that people have been resuming builds a lot more because of test flakes, which appear to have become more frequent after the recent V8 update (several tests now randomly time out).

[richardlau]

I also noticed the backup server is configured to run the script weekly and not daily as per the instructions in https://github.com/nodejs/build/tree/main/backup. I don't know why that change was made and the immediate issue is fixed, so I'll leave it as I found it.

As for this discrepancy I have no idea. #2362 (comment) (from 2020) refers to the pruning being done weekly.

[joaocgreis]

The free space has been stable on the CI server, so I'll close this and assume it was a one-off.