No space left on test-macstadium-macos10.11-x64-2

[targos]

While running citgm-smoker: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/2075/nodes=osx1011/

[sam-github]

There doesn't seem to be any obvious things to cleanup when I looked.

% ssh test-macstadium-macos10.11-x64-2 df /Users/iojs                                                                            
Filesystem   512-blocks     Used Available Capacity  iused   ifree %iused  Mounted on
/dev/disk0s2   49Gi   43Gi  5.7Gi    89% 11391507 1505789   88%   /

Much of the space usage is due to the nodejs build workspace. You could try deleting that and redoing a citgm run, and checking disk space while you do it, that would at least give you an idea how space hungry the citgm is.

Assuming that it is in fact / which ran out of space, the libtool error was less than informative.

[sam-github]

FYI:

ssh test-macstadium-macos10.11-x64-2 du -hs /Users/iojs/build/workspace/\*                                                                     
283M    /Users/iojs/build/workspace/citgm-continuous-integration                                                                                                             
1.3G    /Users/iojs/build/workspace/citgm-smoker                                                                                                                             
130M    /Users/iojs/build/workspace/citgm-smoker-nobuild                                                                                                                     
 38M    /Users/iojs/build/workspace/libuv-test-commit-osx                                                                                                                    
7.8G    /Users/iojs/build/workspace/node-test-commit-osx                                                                                                                     
161M    /Users/iojs/build/workspace/node-test-node-addon-api-new           

[sam-github]

And (to save anyone else the time to wait for this):

796M    /Applications
4.9G    /Library
  0B    /Network
4.6G    /System
 15G    /Users
4.0K    /Volumes
2.5M    /bin
  0B    /cores
4.0K    /dev
4.0K    /etc
1.0K    /home
4.0K    /installer.failurerequests
1.0K    /net
3.0G    /private
1.0M    /sbin
4.0K    /tmp
1.3G    /usr
4.0K    /var

[richardlau]

There's also some discussion over in #1908 (comment).

First step though is to work out what disk libtool is attempting to use.

[sam-github]

% ssh test-macstadium-macos10.11-x64-2 df                                                                             
Filesystem    512-blocks     Used Available Capacity  iused   ifree %iused  Mounted on
/dev/disk0s2   103178384 90633560  12032824    89% 11393193 1504103   88%   /
devfs                348      348         0   100%      602       0  100%   /dev
map -hosts             0        0         0   100%        0       0  100%   /net
map auto_home          0        0         0   100%        0       0  100%   /home

My comment about "assuming it is /" was not helpful, sorry, there is only /, the other 3 aren't real filesystems.

There doesn't seem to be the same cruft, from the other comment, running the same du command:

test-macstadium-macos10:~ iojs$ du -sh .babel.json .cache/ .config/ .node-clinic-rc  .node-gyp/ .node-spawn-wrap-/ .npm* .standard-* .yarn/ .yo-rc-global.json
6.3M    .babel.json
du: .cache/: No such file or directory
8.0K    .config/
4.0K    .node-clinic-rc
du: .node-gyp/: No such file or directory
  0B    .node-spawn-wrap-/
1.3M    .npm
4.0K    .npmrc
112K    .standard-cache
384K    .standard-v12-cache
 40K    .standard-v13-cache
168K    .standard-v14-cache
  0B    .yarn/
4.0K    .yo-rc-global.json

[richardlau]

There doesn't seem to be the same cruft, from the other comment, running the same du command:

On one hand that's good news as Rod did a clear out and CITGM has since been updated to redirect the home directory for the module being tested so the cruft shouldn't occur in home anymore.

In any case the CITGM job on macOS isn't getting as far as running CITGM as the fatal error: /Library/Developer/CommandLineTools/usr/bin/libtool: can't write to output file (No space left on device) error is occurring while building Node.js.

[sam-github]

The citgm job does ANOTHER src build of nodejs? If so, that alone could be the problem. If you look through above, you'll see that there is only about 12 G available on /, but that a nodejs build (at least node-test-commit-osx) takes about 8G... citgm could just be pushing this machine over the edge for space.

[richardlau]

The citgm job does ANOTHER src build of nodejs? If so, that alone could be the problem. If you look through above, you'll see that there is only about 12 G available on /, but that a nodejs build (at least node-test-commit-osx) takes about 8G... citgm could just be pushing this machine over the edge for space.

@sam-github Yes, citgm-smoker builds Node.js and then runs CITGM against it (it's used to test Node.js PR's). There's a separate citgm-smoker-nobuild job that downloads released/nightly binaries and runs CITGM against that (mainly used to test changes to CITGM's lookup).

[sam-github]

That makes sense.

Until I'm into macstadium, I'm not sure what options we have.

It might be worth searching for a test machine with more disk (they might vary), and pinning the build to that machine. It might also be possible to delete the other jobs from workspace before running a citgm run.

[rvagg]

don't forget Caches:

$ du -sh ~iojs/Library/Caches/
899M	/Users/iojs/Library/Caches/

That's annoying, but 7.8G for workspace/node-test-commit-osx is a much bigger problem on these Macs where we have resource constraints.

Node's build size is getting kind of out of control:

$ du -sh /Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1011/out/Release
6.4G	/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1011/out/Release

The biggest offenders:

1.5G	libv8_base_without_compiler.a
1.5G	obj.target/v8_base_without_compiler

Due to space constraints I'm experiencing in rv-test-commit-linux-docker I've added a make distclean at the end in the same script that performs tap2junit, debug builds being a big problem are now taken care of. In the past we've adopted a "keep it around so we can debug" approach, but these crazy sizes might suggest we adopt an alternative approach.

[rvagg]

I've gone ahead and put make distclean || true in the post-build task script for node-test-commit-osx. That should go a fair way toward alleviating this particular problem but we may also want to do the same for citgm jobs.

[richardlau]

I've gone ahead and put make distclean || true in the post-build task script for node-test-commit-osx. That should go a fair way toward alleviating this particular problem but we may also want to do the same for citgm jobs.

Turns out the citgm-smoker job already had a post build task that removes the directory used to build Node.js.

[rvagg]

well that explains why the citgm workspace directory was so trim. I think this should become standard practice on arch/os's that we are regularly having to clean up for space.