VSO / Universal image nearing too large to build

[Chuxel]

The VSO image build has now exceeded the size of the hosted CI servers and is failing to build because it is running out of space even with its own dedicated build. The maximum space is 14GB and adding helm seems to have pushed it over the edge.

Part of the problem is the sheer number of layers in the upstream Oryx image. I'd propose we hold off adding Helm for now (and other things) and look at trying to get an image that is a big more manageable. It's over 8GB even after it's all built.

The only alternative is to get a dedicated VM host with a build agent on it specifically for VSO.

Currently it is blocking CI entirely.

//cc: @lostintangent @nikmd23

[Chuxel]

There may be an upstream bug that caused this: actions/runner-images#709

Storage seems to have dropped at least 2GB from what I am reading.

[Chuxel]

I found a near term workaround by running apt-get autoremove / clean in the CI server prior to building. However, this tells us we're on the edge of this image not building anymore.

We'll either need a dedicated VM for CI here or start thinking about reducing the size of the image in some way.

//cc: @srivatsn @jkeech

[jkeech]

@Chuxel, I’m pretty sure we use the hosted agents in Azure Pipelines and haven’t run into disk space issues so far. Are the hosted GitHub Actions runners using a smaller hardware config? Maybe we could wire up a public Azure Pipelines job if GH runners aren’t sufficient. Maintaining your own self-hosted agent (patching, compliance) is something I’d like to avoid.

[Chuxel]

@jkeech They're the same. They quote 10GB.

There was an issue here where the hosted agents were below that, but by running apt-get autoremove / clean, I got it down again. So that's a bug on their end which is why it just suddenly started.

However, it shows how close we are to not fitting.

[Chuxel]

One update here - it's not actually the build output storage that is running out, it's the host OS. That's why an autoclean works. /dev/sda1 only has 8.6GB free if you don't. With the cleanup, I can see 8GB being consumed on /dev/sda1 after the build for the VSO image.

Turns out removing /usr/share/dotnet frees up 15GB and its not needed for the build as well. Others like Apache have gone so far as to remove packages: https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh

So, there's a bit more headroom here with some hacks.

[Chuxel]

I'm going to close this for now since it looks like this turned out to be an upstream issue that has a workaround for now. We'll need to continue to watch it as we add more to this kitchen sink image.